Commit Graph

15305 Commits

Author SHA1 Message Date
c98b80ae6a [Feature](functions) support ignore and nullable functions (#27848)
support ignore and nullable functions
2023-12-05 14:09:32 +08:00
79f6f85cf1 [FIX](serde)fix datetimev2 serde parse from string with scale (#27965) 2023-12-05 13:58:32 +08:00
aa75f81af5 [fix](case) add "nonConcurrent" to prevent effect other case (#27941)
same as #27940
2023-12-05 13:04:25 +08:00
54fe1a166b [Refactor](scan) refactor scan scheduler to improve performance (#27948)
* [Refactor](scan) refactor scan scheduler to improve performance

* fix pipeline x core
2023-12-05 13:03:16 +08:00
da87fcb477 [bug](function) fix compound expr coredump problem (#27988) 2023-12-05 13:00:14 +08:00
fa0b495b33 [fix](cache)Fix partition cache support DATEV2 (#27978) 2023-12-05 12:59:47 +08:00
17016b9797 [improvement](decimal) use new way for decimal arithmetic precision promotion (#27787)
* [DNM](decimal) use new way for decimal arithmetic precision promotion

* [improvement](decimal) [DNM](decimal) use new way for decimal arithmetic precision promotion
1. [DNM](decimal) use new way for decimal arithmetic precision promotion
2. throw exception if it overflows for decimal arithmetics
3. throw exception if it overflows when casting among number types

* fix compile error of gcc

* improvement

---------

Co-authored-by: morrySnow <morrysnow@126.com>
2023-12-05 12:54:40 +08:00
ca6949ee3e [Bug](partition) fix auto list partition erros of incorrect partition name (#27974)
the partition name need limit it's length and can't have negative "-"
2023-12-05 12:54:06 +08:00
2f63999066 [fix](Nereids): Preserve "" in single quote strings and '' in double quote strings. (#27959) 2023-12-05 12:30:03 +08:00
358d73a0ae [FIX](complextype) fix empty quote with complex type (#27942) 2023-12-05 12:25:26 +08:00
fd2e60a2db [fix](move-memtable) exclude memtable insert memory in query tracker (#27953) 2023-12-05 12:04:15 +08:00
a13227cf4b [fix](move-memtable) fix sink v2 profile (#27982) 2023-12-05 11:53:18 +08:00
da40e1c767 [feature](nereids) Matiarilzed view query rewrite util implementation (#27568)
The basic util implementatation which is used by materialized view rewrite
2023-12-05 11:48:04 +08:00
26d642d5e9 [enhancement](Nereids) format some code in functional deps (#27797) 2023-12-05 11:45:03 +08:00
4afe07e12c [feature](Nereids): support drop constraint on table (#27944) 2023-12-05 11:41:25 +08:00
1ed99c4d8a [Improvement](inverted index) improve inverted index bkd performance in high concurrent scenario (#27820)
Improve BKD performance by enable bkd reader cache and improvement of fast compare and visit in compressed values in BKD tree.
2023-12-05 11:39:53 +08:00
75d0beb8cc [fix](move-memtable) only report load stream profile in the end (#27983) 2023-12-05 11:30:54 +08:00
bd9db7423b [fix](move-memtable) free resources before storage engine stop (#27980) 2023-12-05 11:15:05 +08:00
d69cdf8635 [improve](heartbeat) show more info when receive invalid cluster id (#27975) 2023-12-05 11:10:22 +08:00
a06ac930a0 [refactor](memtable) remove unused stream output (#27889)
Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>
2023-12-05 11:10:10 +08:00
3412a022f4 [fix](restore) fix Restore from __keep_on_local__ throws null pointer… (#26943)
Co-authored-by: walter <patricknicholas@foxmail.com>
Co-authored-by: hugoluo <hugoluo@tencent.com>
Co-authored-by: walter <patricknicholas@foxmail.com>
2023-12-05 10:55:28 +08:00
3c97e69f3c [fix](Nereids) should not push down project to the nullable side of outer join (#27912) 2023-12-05 10:43:33 +08:00
50ad40a7a8 [test](Nereids): add infer-predicates regression test (#27850) 2023-12-05 10:16:01 +08:00
20d4d7eb2b [fix](Hudi-catalog) fix hudi catalog code (#27963)
In the original logic, `allfields.addall` will modify the objects in `hmsTable`.
2023-12-04 22:28:19 +08:00
2b4c4bb442 [Fix][Opt](parquet-reader) Fix filter push down with decimal types in parquet reader. (#27897)
Fix filter push down with decimal types in parquet reader introduced by #22842
2023-12-04 22:25:39 +08:00
1afdbfe723 [enhance](BE) Refactor TaskWorkerPool (#27555) 2023-12-04 21:46:10 +08:00
4934f7ed8d [enhancement](Nereids) add test for some push down filter rule (#27757) 2023-12-04 20:57:57 +08:00
8e2961858e [enhancement](Nereids): extract group plan in struct info node (#27939) 2023-12-04 19:46:40 +08:00
4c9bf98dcd [config](p0) Pipeline p0 open arrow_flight_sql_port #27945 2023-12-04 19:24:15 +08:00
b096062680 [feature-wip](arrow-flight)(step6) Support regression test (#27847)
Design Documentation Linked to #25514

Regression test add a new group: arrow_flight_sql,

./run-regression-test.sh -g arrow_flight_sql to run regression-test, can use jdbc:arrow-flight-sql to run all Suites whose group contains arrow_flight_sql.
./run-regression-test.sh -g p0,arrow_flight_sql to run regression-test, can use jdbc:arrow-flight-sql to run all Suites whose group contains arrow_flight_sql, and use jdbc:mysql to run other Suites whose group contains p0 but does not contain arrow_flight_sql.
Requires attention, the formats of jdbc:arrow-flight-sql and jdbc:mysql and mysql client query results are different, for example:

Datatime field type: jdbc:mysql returns 2010-01-02T05:09:06, mysql client returns 2010-01-02 05:09:06, jdbc:arrow-flight-sql also returns 2010-01-02 05:09 :06.
Array and Map field types: jdbc:mysql returns ["ab", "efg", null], {"f1": 1, "f2": "a"}, jdbc:arrow-flight-sql returns ["ab ","efg",null], {"f1":1,"f2":"a"}, which is missing spaces.
Float field type: jdbc:mysql and mysql client returns 6.333, jdbc:arrow-flight-sql returns 6.333000183105469, in query_p0/subquery/test_subquery.groovy.
If the query result is empty, jdbc:arrow-flight-sql returns empty and jdbc:mysql returns \N.
use database; and query should be divided into two SQL executions as much as possible. otherwise the results may not be as expected. For example: USE information_schema; select cast ("0.0101031417" as datetime) The result is 2000-01-01 03:14:1 (constant fold), select cast ("0.0101031417" as datetime) The result is null (no constant fold),
In addition, doris jdbc:arrow-flight-sql still has unfinished parts, such as:

Unsupported data type: Decimal256. INVALID_ARGUMENT: [INTERNAL_ERROR]Fail to convert block data to arrow data, error: [E3] write_column_to_arrow with type Decimal256
Unsupported null value of map key. INVALID_ARGUMENT: [INTERNAL_ERROR]Fail to convert block data to arrow data, error: [E33] Can not write null value of map key to arrow.
Unsupported data type: ARRAY<MAP<TEXT,TEXT>>
jdbc:arrow-flight-sql not support connecting to specify DB name, such asjdbc:arrow-flight-sql://127.0.0.1:9090/{db_name}", In order to be compatible with regression-test, use db_nameis added before all SQLs whenjdbc:arrow-flight-sql` runs regression test.
select timediff("2010-01-01 01:00:00", "2010-01-02 01:00:00");, error java.lang.NumberFormatException: For input string: "-24:00:00"
2023-12-04 19:23:56 +08:00
e427ac9338 [typo](doc) update default value of compaction_promotion_min_size_mbytes (#25839)
Co-authored-by: gnehil <adamlee489@gamil.com>
2023-12-04 17:58:46 +08:00
283e1ea0b7 test operate txn 2pc exception handling (#27924)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-12-04 17:30:05 +08:00
a7d1e92fc2 [Fix](variant) handle StorageReadOptions to avoid crash in new_column_iterator_with_path (#27936)
In partial update, read variant without `opt` will lead to crash
2023-12-04 17:02:35 +08:00
2022a8ab32 [fix](invert index) fix reader does not close fd (#27918) 2023-12-04 16:44:50 +08:00
c80807a5c7 [fix](test) disable join reorder in bucket shuffle join test (#27930) 2023-12-04 16:15:00 +08:00
86c2b93e5b [improvement](fixreplica) move to healthy replica when fix replica bad (#27934)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-04 16:04:27 +08:00
4d1aa131ee [Feature](datatype) add be ut codes for IPv4/v6 (#26534)
Add unit test codes for IP types
2023-12-04 15:25:02 +08:00
2eb8e0d66a [minor](testcase) Add auto partition test cases (#27921)
Add auto partition test cases
2023-12-04 15:10:42 +08:00
a6a6892f90 [chore](status code) avoid print stack for DATA_QUALITY_ERROR (#27935)
issue introduced by #27065
2023-12-04 15:04:27 +08:00
e80526ee3a [opt](nereids)remove partition & histogram from col stats to reduce memory usage #27885 2023-12-04 14:52:05 +08:00
48935c14e2 [Improvement](variant) limit the column size on tablet schema (#27399) (#27785)
1. limit the column count to default 2048
2. fix get_inverted_index return nullptr when variant's unique id is -1, using it's parent unique id instead
3. avoid add same path subcolumn duplicately in tablet schema
4. make extracted column unique id -1
2023-12-04 14:47:36 +08:00
27e1b2ac2d [docs](fix) fix document error (#27612) 2023-12-04 14:30:41 +08:00
Pxl
e196a4fd8f [Chore](case) log out show alter table info on no_await (#27926)
log out show alter table info on no_await
2023-12-04 14:30:16 +08:00
Pxl
2b715924c5 [Chore](function) set normal function use_default_implementation_for_constants to default (#27891)
set normal function use_default_implementation_for_constants to default
2023-12-04 14:19:25 +08:00
e19af1b2ed [regression](Nereids) add rule test for push down limit + sort test (#26642) 2023-12-04 14:18:55 +08:00
Pxl
45a49ac059 Bug](column) support insert default for ColumnFixedLengthObject #27927 2023-12-04 12:52:50 +08:00
e62d19d90d [improve](partition) support auto list partition with more columns (#27817)
before the partition by column only have one column.
now remove those limit, could have more columns.
2023-12-04 11:33:18 +08:00
80f528bf26 [enhancement](backup-restore) add config for upload/download task num per be (#27772)
set upload/download task num per be, and improve the overall speed of upload/download, enhance the performance of backup and recovery.

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-12-04 11:19:45 +08:00
Pxl
e3d2425d47 [Improvement](join) remove insert_indices_from_join and special judge for -1 (#27779)
remove insert_indices_from_join and special judge for -1
2023-12-04 11:03:22 +08:00
f8bdbf67b4 [fix](deploy) K8s deploy manager cannot get group host info by endpoint (#27813)
K8s deploy manager cannot get group host info by endpoint. If we get group host info by endpoint, there is no need to init statefulset.
2023-12-04 10:50:43 +08:00