Commit Graph

7054 Commits

Author SHA1 Message Date
d672e321cc [feature](Nereids) support values inline table in query (#28972) 2023-12-26 15:51:08 +08:00
3ea9ff7d4a [opt](Nereids) let inverted index work with top opt (#29000) 2023-12-26 15:50:56 +08:00
f371528ada [fix](Nereids) generating function should not folding to NullLiteral (#29003)
should not fold table generating function to null when do constant folding.
we should remove Generate node and replaced it by project later.
2023-12-26 15:25:19 +08:00
983f851fb8 [fix](Nereids) runtime filter push down failed (#28997)
project child not always NamedExpression

failed msg
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:623) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:478) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:457) ~[classes/:?]
    at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:245) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[classes/:?]
    at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ClassCastException: class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.nereids.trees.plans.physical.PhysicalSetOperation.pushDownRuntimeFilter(PhysicalSetOperation.java:178) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.pushDownRuntimeFilter(PhysicalHashJoin.java:229) ~[classes/:?]
    at org.apache.doris.nereids.processor.post.RuntimeFilterGenerator.pushDownRuntimeFilterCommon(RuntimeFilterGenerator.java:386) ~[classes/:?]
```
2023-12-26 14:13:43 +08:00
10623ad671 [improvement](jdbc catalog) Optimize connection pool caching logic (#28859)
In the old caching logic, we only used jdbcurl, user, and password as cache keys. This may cause the old link to be still used when replacing the jar package, so we should concatenate all the parameters required for the connection pool as the key.
2023-12-26 14:12:37 +08:00
a8688509c9 [opt](Nereids) support cast bewteen numeric and boolean in FE (#29006) 2023-12-26 14:11:34 +08:00
00bb099fdd [hotfix](jdbc catalog) fix load table and column names npe (#28865)
This fix is for the npe situation that occurs when FE of non-Master nodes initializes Jdbc Catalog metadata.
2023-12-26 14:11:14 +08:00
509cfea99a [feature](Load)(step2)support nereids load job schedule (#26356)
We will  Integrate new load job manager into  new job scheduling framework so that the insert into task can be scheduled after the broker load sql  is converted to insert into TVF(table value function) sql.

issue: https://github.com/apache/doris/issues/24221

Now support:
1. load data by tvf insert into sql, but just for simple load(columns need to be defined in the table)
2. show load stmt
- job id, label name, job state, time info
- simple progress
3. cancel load from db
4. support that enable new load through Config.enable_nereids_load
5. can replay job after restarting doris

TODO:
- support partition insert job
- support show statistics from BE
- support multiple task and collect task statistic
- support transactional task
- need add ut case
2023-12-26 12:29:05 +08:00
b8fd55b0cf [feature](Nereids) support table valued function http_stream (#29004) 2023-12-26 11:47:04 +08:00
065eb9a72b [feature](nereids)support partition property in nereids (#28982) 2023-12-26 11:19:04 +08:00
f30e50676e [opt](scanner) optimize the number of threads of scanners (#28640)
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
2023-12-26 10:24:12 +08:00
7434de9ed8 [improvement](nereids) Get partition related table disable nullable field and complete agg matched pattern mv rules. (#28973)
* [improvement] (nereids) Get partition related table disable nullable field and modify regression test, complete agg mv rules.

* make filed not null to create partition mv
2023-12-26 00:29:42 +08:00
2aea47c0a9 [fix](mtmv)add log for resolve pending task (#28999)
* add lock for resolve pending task

* add lock for resolve pending task
2023-12-26 00:29:28 +08:00
17f3ca7349 [fix](planner)should save original select list item before analyze (#28187)
* [fix](planner)should save original select list item before analyze

* fix test case

* fix failed case
2023-12-25 23:06:45 +08:00
c2c5df9341 [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (#28935)
* NEED

* Update pipeline x

* fix pipelinex compile
2023-12-25 22:47:23 +08:00
2dc7d82aff [Chore](Job)print log before task execute (#28962) 2023-12-25 16:55:59 +08:00
17917a0adb [feature](load) enable memtable on sink node by default (#28963) 2023-12-25 16:27:16 +08:00
29d3d5e433 [Test](Job)Add test case (#28481) 2023-12-25 15:12:14 +08:00
1d984e0ebb return residual expr of join (#28760) 2023-12-25 12:53:14 +08:00
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318)
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
48f8f8f2ad [fix](nereids)group by expr may be bound twice in bind agg slot (#28771) 2023-12-25 11:23:44 +08:00
c53611dcb3 [feature](nereids)support decimalv2 (#28726) 2023-12-25 11:23:27 +08:00
99755920e2 [fix](nereids)subquery unnest need handle subquery in Not expr correnctly (#28713) 2023-12-25 11:23:00 +08:00
b49671b31e [feature](api) add profile text api (#28697) 2023-12-25 10:17:09 +08:00
d8cb4da73d [chore](prompt) Optimize tablet and replica prompt by pointing out what the numbers mean (#28925) 2023-12-24 22:29:24 +08:00
2f7deb3dea [enhancement](bulk-load) cancel loading tasks directly without retrying when timeout exceeded (#28666) 2023-12-24 21:57:31 +08:00
1e44a4b145 [opt](nereids) convert or to inpredicate and optimize inpredicate partition prune (#28316) 2023-12-24 21:42:03 +08:00
145683ccdb [improvement](group commit) make get column function more reliable when replaying wal (#28900) 2023-12-24 21:17:39 +08:00
b8de5cfc36 [fix] (nereids) Catch exception when mv fail and fix the npe (#28932) 2023-12-24 20:41:24 +08:00
6ea6ff5dee [fix](nereids) Fix query mv rewrite fail when mv cache build quickly (#28876) 2023-12-24 09:00:49 +08:00
5505fa3755 [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze (#28916)
Fix partition name NPE and sample for all table during auto analyze.
Sample for all tables because getData may have latency, which may cause full analyze a huge table and use too much resource. Sample for all tables to avoid this. Will improve the strategy later.
2023-12-24 01:49:49 +08:00
13a3550d8e [nereids] fix join fd computing bug (#28849) 2023-12-24 01:41:43 +08:00
37777dcdb5 [fix](mtmv) Related partition exclude null generate column when increment build materialized view (#28855)
Infer partition column by materialized view partition column, exclude null generate column in join when increment build materialized view
2023-12-24 01:37:55 +08:00
1d8822b2b7 [fix](paimon)fix like predicate (#28803)
fix like predict
2023-12-23 22:25:55 +08:00
66b14f4db1 [fix](mtmv)fix can not create mtmv all use default value (#28922) 2023-12-23 21:27:01 +08:00
0a1d9f4cbc [feature](mtmv)add more test case1 (#28910) 2023-12-23 14:39:44 +08:00
623257d02b [feature](mtmv)MTMV pause and resume (#28887)
- PAUSE MATERIALIZED VIEW JOB ON mv1
- RESUME MATERIALIZED VIEW JOB ON mv1
- fix when drop db,not drop job
- add lock for one materialized view can only run one task at a time
2023-12-23 14:30:54 +08:00
3d2b4ae244 [fix](mtmv) fix failed to specify the number of buckets when bucket auto (#28854)
Issue Number: close #xxx

- fix failed to specify the number of buckets when bucket auto
- delete unused SessionVariable
- if mtmv used external table ,check `isMaterializedViewRewriteEnableContainForeignTable`
2023-12-23 09:26:16 +08:00
ae64afd112 [fix](pipelineX) fix cannot runtime obtain profile on pipelineX #28795 2023-12-23 08:24:04 +08:00
e0bb7fa935 [exec](load) change default parallel num from 1 to 8 in no pipeline exec engine (#28864) 2023-12-23 00:05:44 +08:00
966766f3b7 [enhancement](broker-load) fix-move-memtable-session-var-for-s3 (#28894) 2023-12-22 23:25:06 +08:00
726a9b96c2 [enhancement](udf) add prepare function for java-udf (#28750) 2023-12-22 22:15:59 +08:00
3eec62e1d1 [fix](multi-catalog)filter impala generated path (#28786)
file impala generated dir _imapala_insert_staging
2023-12-22 21:58:50 +08:00
fa0ad56817 [exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior (#28818) 2023-12-22 19:50:57 +08:00
b2b209e938 [feature](Nereids) Pull up join from union all (#28682) 2023-12-22 19:35:20 +08:00
3ed82bcee2 [Feature](inverted index) add lowercase option for inverted index analyzer (#28704) 2023-12-22 18:22:44 +08:00
cb61a0727f [bug](sharedscan) Fix shared scan bug (#28841) 2023-12-22 15:13:14 +08:00
7f310cec9c [opt](sessionVar)show changed sessoin var first #28840
“show variables” command list changed vars before not changed vars,
2023-12-22 14:45:33 +08:00
007f498f3b (enhance)(InternalQuery) Support to collect profile for intenal query (#28762) 2023-12-22 14:03:48 +08:00
f38e11ec4e [fix](paimon)fix type convert for paimon (#28774)
fix type convert for paimon
2023-12-22 13:18:25 +08:00