Commit Graph

15847 Commits

Author SHA1 Message Date
d672e321cc [feature](Nereids) support values inline table in query (#28972) 2023-12-26 15:51:08 +08:00
3ea9ff7d4a [opt](Nereids) let inverted index work with top opt (#29000) 2023-12-26 15:50:56 +08:00
31db633624 [improve](load) add profile for WaitFlushLimitTime (#29013) 2023-12-26 15:41:54 +08:00
f371528ada [fix](Nereids) generating function should not folding to NullLiteral (#29003)
should not fold table generating function to null when do constant folding.
we should remove Generate node and replaced it by project later.
2023-12-26 15:25:19 +08:00
983f851fb8 [fix](Nereids) runtime filter push down failed (#28997)
project child not always NamedExpression

failed msg
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:623) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:478) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:457) ~[classes/:?]
    at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:245) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[classes/:?]
    at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ClassCastException: class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.nereids.trees.plans.physical.PhysicalSetOperation.pushDownRuntimeFilter(PhysicalSetOperation.java:178) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.pushDownRuntimeFilter(PhysicalHashJoin.java:229) ~[classes/:?]
    at org.apache.doris.nereids.processor.post.RuntimeFilterGenerator.pushDownRuntimeFilterCommon(RuntimeFilterGenerator.java:386) ~[classes/:?]
```
2023-12-26 14:13:43 +08:00
10623ad671 [improvement](jdbc catalog) Optimize connection pool caching logic (#28859)
In the old caching logic, we only used jdbcurl, user, and password as cache keys. This may cause the old link to be still used when replacing the jar package, so we should concatenate all the parameters required for the connection pool as the key.
2023-12-26 14:12:37 +08:00
a8688509c9 [opt](Nereids) support cast bewteen numeric and boolean in FE (#29006) 2023-12-26 14:11:34 +08:00
00bb099fdd [hotfix](jdbc catalog) fix load table and column names npe (#28865)
This fix is for the npe situation that occurs when FE of non-Master nodes initializes Jdbc Catalog metadata.
2023-12-26 14:11:14 +08:00
52eeee347f [opt](compound) Optimize by deleting the compound expr after obtaining the final result (#28934) 2023-12-26 14:10:53 +08:00
f7a624de38 [fix](case) Add sync to test case (#29034) 2023-12-26 14:08:41 +08:00
c8ed14f11c [enhance](tablet) Reduce log in tablet meta (#28719) 2023-12-26 13:37:30 +08:00
509cfea99a [feature](Load)(step2)support nereids load job schedule (#26356)
We will  Integrate new load job manager into  new job scheduling framework so that the insert into task can be scheduled after the broker load sql  is converted to insert into TVF(table value function) sql.

issue: https://github.com/apache/doris/issues/24221

Now support:
1. load data by tvf insert into sql, but just for simple load(columns need to be defined in the table)
2. show load stmt
- job id, label name, job state, time info
- simple progress
3. cancel load from db
4. support that enable new load through Config.enable_nereids_load
5. can replay job after restarting doris

TODO:
- support partition insert job
- support show statistics from BE
- support multiple task and collect task statistic
- support transactional task
- need add ut case
2023-12-26 12:29:05 +08:00
b8fd55b0cf [feature](Nereids) support table valued function http_stream (#29004) 2023-12-26 11:47:04 +08:00
065eb9a72b [feature](nereids)support partition property in nereids (#28982) 2023-12-26 11:19:04 +08:00
92660bb1b2 [chore](config) modify variant_ratio_of_defaults_as_sparse_column from 0.95 to 1 (#28984)
since sparse column is not stable at present
2023-12-26 10:24:43 +08:00
f30e50676e [opt](scanner) optimize the number of threads of scanners (#28640)
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
2023-12-26 10:24:12 +08:00
75a45484b6 [chore](config) modify tablet_schema_cache_recycle_interval from 24h to 1h (#28980)
To prevent from too many tablet schema cache in memory and lead to performance issue when hold lock
to erase item
2023-12-26 00:34:58 +08:00
7434de9ed8 [improvement](nereids) Get partition related table disable nullable field and complete agg matched pattern mv rules. (#28973)
* [improvement] (nereids) Get partition related table disable nullable field and modify regression test, complete agg mv rules.

* make filed not null to create partition mv
2023-12-26 00:29:42 +08:00
2aea47c0a9 [fix](mtmv)add log for resolve pending task (#28999)
* add lock for resolve pending task

* add lock for resolve pending task
2023-12-26 00:29:28 +08:00
cefae3dc90 [bug](storage) Fix gc rowset bug (#28979) 2023-12-26 00:29:03 +08:00
17f3ca7349 [fix](planner)should save original select list item before analyze (#28187)
* [fix](planner)should save original select list item before analyze

* fix test case

* fix failed case
2023-12-25 23:06:45 +08:00
2411dbe0e6 [doc](multi-catalog)add krb and some ertificates FAQs (#28858)
add some security docs
2023-12-25 23:05:44 +08:00
137f785698 [fix](parquet_reader) misused bool pointer (#28986)
Signed-off-by: pengyu <pengyu@selectdb.com>
2023-12-25 22:58:08 +08:00
c2c5df9341 [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (#28935)
* NEED

* Update pipeline x

* fix pipelinex compile
2023-12-25 22:47:23 +08:00
0af9371a96 [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (#28991)
Introduced by #28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.

e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
2023-12-25 22:19:01 +08:00
8921b313b4 [fix](doc) typo fix in dynamic-partition page (#28511) 2023-12-25 21:02:37 +08:00
7081139bdc [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#27943) 2023-12-25 20:35:22 +08:00
91e5b47439 [fix](hdfs) Fix HdfsFileSystem::exists_impl crash (#28952)
Calling hdfsGetLastExceptionRootCause without initializing ThreadLocalState
will crash. This PR modifies the condition for determining the existence of
a hdfs file, because hdfsExists will set errno to ENOENT when the file does
not exist, we can use this condition to check whether a file existence rather
than check the existence of the root cause.
2023-12-25 19:18:01 +08:00
2dc7d82aff [Chore](Job)print log before task execute (#28962) 2023-12-25 16:55:59 +08:00
17917a0adb [feature](load) enable memtable on sink node by default (#28963) 2023-12-25 16:27:16 +08:00
8e47fd28e2 [fix](doc) typo fix in auto-partition page (#28512) 2023-12-25 15:49:18 +08:00
29d3d5e433 [Test](Job)Add test case (#28481) 2023-12-25 15:12:14 +08:00
f2cdf15078 (enhance)(regression) Support force_olap_table_replication_num=3 run test_insert_random_distribution_table case (#28903) 2023-12-25 14:06:48 +08:00
c2eabbd441 [fix](load) fix nullptr when getting memtable flush running count (#28942)
* [fix](load) fix nullptr when getting memtable flush running count

* style
2023-12-25 13:49:18 +08:00
1d984e0ebb return residual expr of join (#28760) 2023-12-25 12:53:14 +08:00
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318)
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
6de797844d [doc](insert) Add group commit docs (#25949) 2023-12-25 11:34:11 +08:00
48f8f8f2ad [fix](nereids)group by expr may be bound twice in bind agg slot (#28771) 2023-12-25 11:23:44 +08:00
c53611dcb3 [feature](nereids)support decimalv2 (#28726) 2023-12-25 11:23:27 +08:00
99755920e2 [fix](nereids)subquery unnest need handle subquery in Not expr correnctly (#28713) 2023-12-25 11:23:00 +08:00
6085a61f1e [chore](test) correct create table statement (#28863) 2023-12-25 10:55:35 +08:00
f374beaa4e [fix](log) regularise some BE error type and fix a load task check #28729 2023-12-25 10:45:19 +08:00
3273e0e635 [refactor](pipelineX)do not override dependency() function in pipelineX (#28848) 2023-12-25 10:36:31 +08:00
24b1b4d96b [fix](pipelineX) fix use global rf when there no shared_scans (#28869) 2023-12-25 10:35:22 +08:00
e326ebb63e [feature](pipelineX) control exchange sink by memory usage (#28814) 2023-12-25 10:31:50 +08:00
d42fd68d6b [opt](invert index) Empty strings are not written to the index in the case of TOKENIZED (#28822) 2023-12-25 10:23:07 +08:00
b49671b31e [feature](api) add profile text api (#28697) 2023-12-25 10:17:09 +08:00
ff365ca130 [docs] (DebugPoints) Update docs about Debug Points (#28347)
---------

Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-12-25 09:33:47 +08:00
b7ae7a07c7 [fix](join) incorrect result of left semi/anti join with empty build side (#28898) 2023-12-25 09:07:38 +08:00
d8cb4da73d [chore](prompt) Optimize tablet and replica prompt by pointing out what the numbers mean (#28925) 2023-12-24 22:29:24 +08:00