Commit Graph

15320 Commits

Author SHA1 Message Date
aaa44fa3dc [deps](odbc) undefined the BOOL in include/sqltypes.h (#28014)
When `ENABLE_PCH = false`, this define will be conflict with BOOL in `include/arrow/type_fwd.h`.
The ODBC table will be deprecated in 2.1, so I just simply undefined the BOOL in include/sqltypes.h
to make compile OK
2023-12-06 00:16:23 +08:00
e431c2b980 [Improvement](multi-catalog)make location easier to modified, decoupling all storage with single location class (#27874)
decoupling all storage with single location class
2023-12-06 00:13:54 +08:00
fd1db4da3d [agg](profile) fix incorrent profile (#28004) 2023-12-05 20:48:10 +08:00
7f1b558011 [fix](stats) truncate min/max if too long (#27955)
For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
2023-12-05 20:40:38 +08:00
05adbfdb3d [feature](inverted index) match_phrase_prefix feature added (#27404)
select count() from test_index_match_phrase_prefix where request match_phrase_prefix 'xxx';
2023-12-05 20:15:13 +08:00
ffa4ea66d5 [enhancement](main) donot coredump when be can not start (#27928) 2023-12-05 20:11:24 +08:00
ea275e687a [pipelineX](minor) remove unused code (#28016) 2023-12-05 19:41:40 +08:00
e79422addc [refactor](Nereids) compatible with all ability legacy planner (#27947)
refactor:
1. split InsertIntoTableCommand into three sub command
- InsertIntoTableCommand
- InsertOverwriteTableCommand
- BatchInsertIntoTableCommand

feature:
1. support DEFAULT keywords in values list
2. support empty values list
3. support temporary partition
4. support insert into values in txn model

fix:
1. should start transaction before release read lock on target table
2023-12-05 19:10:55 +08:00
8e161ad0f2 [debug](timeout) add some log to debug timeout== 0 (#28011)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-05 18:35:36 +08:00
59c0db4080 Fix workload group unstable (#28003) 2023-12-05 17:55:43 +08:00
8dc67d3be6 [case](regression-test) add backup recovery for inverted index and bl… (#27616) 2023-12-05 17:11:02 +08:00
6074cddcf8 [feature](mtmv)add Job and task tvf (#27967)
add:
select * from jobs("type"="mv");
select * from tasks("type"="mv");
select * from jobs("type"="insert");
select * from tasks("type"="insert");

add check priv for mv_infos("database"="xxx");

change JobType MTMV==>MV
2023-12-05 15:12:36 +08:00
02512cd0e2 [fix](stats)Drop stats or update updated rows after truncate table (#27931)
1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.
2023-12-05 14:53:35 +08:00
Pxl
8a761dff84 [Bug](materialized-view) fix create mv failed on unique table (#27971)
fix create mv failed on unique table
2023-12-05 14:53:09 +08:00
3595f21405 [improvement](executor)clear unused cgroup path (#27798)
* clear unused cgroup path

* use C++ api

* add gcc header
2023-12-05 14:18:23 +08:00
c98b80ae6a [Feature](functions) support ignore and nullable functions (#27848)
support ignore and nullable functions
2023-12-05 14:09:32 +08:00
79f6f85cf1 [FIX](serde)fix datetimev2 serde parse from string with scale (#27965) 2023-12-05 13:58:32 +08:00
aa75f81af5 [fix](case) add "nonConcurrent" to prevent effect other case (#27941)
same as #27940
2023-12-05 13:04:25 +08:00
54fe1a166b [Refactor](scan) refactor scan scheduler to improve performance (#27948)
* [Refactor](scan) refactor scan scheduler to improve performance

* fix pipeline x core
2023-12-05 13:03:16 +08:00
da87fcb477 [bug](function) fix compound expr coredump problem (#27988) 2023-12-05 13:00:14 +08:00
fa0b495b33 [fix](cache)Fix partition cache support DATEV2 (#27978) 2023-12-05 12:59:47 +08:00
17016b9797 [improvement](decimal) use new way for decimal arithmetic precision promotion (#27787)
* [DNM](decimal) use new way for decimal arithmetic precision promotion

* [improvement](decimal) [DNM](decimal) use new way for decimal arithmetic precision promotion
1. [DNM](decimal) use new way for decimal arithmetic precision promotion
2. throw exception if it overflows for decimal arithmetics
3. throw exception if it overflows when casting among number types

* fix compile error of gcc

* improvement

---------

Co-authored-by: morrySnow <morrysnow@126.com>
2023-12-05 12:54:40 +08:00
ca6949ee3e [Bug](partition) fix auto list partition erros of incorrect partition name (#27974)
the partition name need limit it's length and can't have negative "-"
2023-12-05 12:54:06 +08:00
2f63999066 [fix](Nereids): Preserve "" in single quote strings and '' in double quote strings. (#27959) 2023-12-05 12:30:03 +08:00
358d73a0ae [FIX](complextype) fix empty quote with complex type (#27942) 2023-12-05 12:25:26 +08:00
fd2e60a2db [fix](move-memtable) exclude memtable insert memory in query tracker (#27953) 2023-12-05 12:04:15 +08:00
a13227cf4b [fix](move-memtable) fix sink v2 profile (#27982) 2023-12-05 11:53:18 +08:00
da40e1c767 [feature](nereids) Matiarilzed view query rewrite util implementation (#27568)
The basic util implementatation which is used by materialized view rewrite
2023-12-05 11:48:04 +08:00
26d642d5e9 [enhancement](Nereids) format some code in functional deps (#27797) 2023-12-05 11:45:03 +08:00
4afe07e12c [feature](Nereids): support drop constraint on table (#27944) 2023-12-05 11:41:25 +08:00
1ed99c4d8a [Improvement](inverted index) improve inverted index bkd performance in high concurrent scenario (#27820)
Improve BKD performance by enable bkd reader cache and improvement of fast compare and visit in compressed values in BKD tree.
2023-12-05 11:39:53 +08:00
75d0beb8cc [fix](move-memtable) only report load stream profile in the end (#27983) 2023-12-05 11:30:54 +08:00
bd9db7423b [fix](move-memtable) free resources before storage engine stop (#27980) 2023-12-05 11:15:05 +08:00
d69cdf8635 [improve](heartbeat) show more info when receive invalid cluster id (#27975) 2023-12-05 11:10:22 +08:00
a06ac930a0 [refactor](memtable) remove unused stream output (#27889)
Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>
2023-12-05 11:10:10 +08:00
3412a022f4 [fix](restore) fix Restore from __keep_on_local__ throws null pointer… (#26943)
Co-authored-by: walter <patricknicholas@foxmail.com>
Co-authored-by: hugoluo <hugoluo@tencent.com>
Co-authored-by: walter <patricknicholas@foxmail.com>
2023-12-05 10:55:28 +08:00
3c97e69f3c [fix](Nereids) should not push down project to the nullable side of outer join (#27912) 2023-12-05 10:43:33 +08:00
50ad40a7a8 [test](Nereids): add infer-predicates regression test (#27850) 2023-12-05 10:16:01 +08:00
20d4d7eb2b [fix](Hudi-catalog) fix hudi catalog code (#27963)
In the original logic, `allfields.addall` will modify the objects in `hmsTable`.
2023-12-04 22:28:19 +08:00
2b4c4bb442 [Fix][Opt](parquet-reader) Fix filter push down with decimal types in parquet reader. (#27897)
Fix filter push down with decimal types in parquet reader introduced by #22842
2023-12-04 22:25:39 +08:00
1afdbfe723 [enhance](BE) Refactor TaskWorkerPool (#27555) 2023-12-04 21:46:10 +08:00
4934f7ed8d [enhancement](Nereids) add test for some push down filter rule (#27757) 2023-12-04 20:57:57 +08:00
8e2961858e [enhancement](Nereids): extract group plan in struct info node (#27939) 2023-12-04 19:46:40 +08:00
4c9bf98dcd [config](p0) Pipeline p0 open arrow_flight_sql_port #27945 2023-12-04 19:24:15 +08:00
b096062680 [feature-wip](arrow-flight)(step6) Support regression test (#27847)
Design Documentation Linked to #25514

Regression test add a new group: arrow_flight_sql,

./run-regression-test.sh -g arrow_flight_sql to run regression-test, can use jdbc:arrow-flight-sql to run all Suites whose group contains arrow_flight_sql.
./run-regression-test.sh -g p0,arrow_flight_sql to run regression-test, can use jdbc:arrow-flight-sql to run all Suites whose group contains arrow_flight_sql, and use jdbc:mysql to run other Suites whose group contains p0 but does not contain arrow_flight_sql.
Requires attention, the formats of jdbc:arrow-flight-sql and jdbc:mysql and mysql client query results are different, for example:

Datatime field type: jdbc:mysql returns 2010-01-02T05:09:06, mysql client returns 2010-01-02 05:09:06, jdbc:arrow-flight-sql also returns 2010-01-02 05:09 :06.
Array and Map field types: jdbc:mysql returns ["ab", "efg", null], {"f1": 1, "f2": "a"}, jdbc:arrow-flight-sql returns ["ab ","efg",null], {"f1":1,"f2":"a"}, which is missing spaces.
Float field type: jdbc:mysql and mysql client returns 6.333, jdbc:arrow-flight-sql returns 6.333000183105469, in query_p0/subquery/test_subquery.groovy.
If the query result is empty, jdbc:arrow-flight-sql returns empty and jdbc:mysql returns \N.
use database; and query should be divided into two SQL executions as much as possible. otherwise the results may not be as expected. For example: USE information_schema; select cast ("0.0101031417" as datetime) The result is 2000-01-01 03:14:1 (constant fold), select cast ("0.0101031417" as datetime) The result is null (no constant fold),
In addition, doris jdbc:arrow-flight-sql still has unfinished parts, such as:

Unsupported data type: Decimal256. INVALID_ARGUMENT: [INTERNAL_ERROR]Fail to convert block data to arrow data, error: [E3] write_column_to_arrow with type Decimal256
Unsupported null value of map key. INVALID_ARGUMENT: [INTERNAL_ERROR]Fail to convert block data to arrow data, error: [E33] Can not write null value of map key to arrow.
Unsupported data type: ARRAY<MAP<TEXT,TEXT>>
jdbc:arrow-flight-sql not support connecting to specify DB name, such asjdbc:arrow-flight-sql://127.0.0.1:9090/{db_name}", In order to be compatible with regression-test, use db_nameis added before all SQLs whenjdbc:arrow-flight-sql` runs regression test.
select timediff("2010-01-01 01:00:00", "2010-01-02 01:00:00");, error java.lang.NumberFormatException: For input string: "-24:00:00"
2023-12-04 19:23:56 +08:00
e427ac9338 [typo](doc) update default value of compaction_promotion_min_size_mbytes (#25839)
Co-authored-by: gnehil <adamlee489@gamil.com>
2023-12-04 17:58:46 +08:00
283e1ea0b7 test operate txn 2pc exception handling (#27924)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-12-04 17:30:05 +08:00
a7d1e92fc2 [Fix](variant) handle StorageReadOptions to avoid crash in new_column_iterator_with_path (#27936)
In partial update, read variant without `opt` will lead to crash
2023-12-04 17:02:35 +08:00
2022a8ab32 [fix](invert index) fix reader does not close fd (#27918) 2023-12-04 16:44:50 +08:00
c80807a5c7 [fix](test) disable join reorder in bucket shuffle join test (#27930) 2023-12-04 16:15:00 +08:00