Commit Graph

13681 Commits

Author SHA1 Message Date
923402df71 [test](fix) fix case bug for lateral_view.groovy (#24779) 2023-09-22 21:21:34 +08:00
f3062e3d74 [fix](insert) Make insert group commit regression stable (#24584) 2023-09-22 20:18:14 +08:00
b86f09418f [chore](build) Fix the FE build on CentOS 6 (#24798)
Using grpc-java whose version is newer than 1.34.0 will break the build on CentOS 6 due to the obsolete GLIBC.
2023-09-22 19:58:12 +08:00
3bf95b2cfe [fix](merge-on-write) fix duplicate key in schema change (#24782) 2023-09-22 19:50:31 +08:00
f4b1e2b343 [Fix](core) Fix segment cache core when output rowset is nullptr (#24778) 2023-09-22 19:48:42 +08:00
ac55d45f79 [Fix](topn opt) fix heap use after free when shrink in fetch phase (#24774) 2023-09-22 19:48:05 +08:00
35c0697a60 [fix](planner)SelectStmt's constructor should initialize originSelectList member (#24755)
resetSelectList method will use originSelectList to recover the origin select list. If the originSelectList is lost in constructor, the resetSelectList will fail to recover and make the analyze process fail.
2023-09-22 19:31:29 +08:00
eb9d334b55 [opt](nereids) enable runtime filter prune by default (#24717)
add stats back to physical plan in post processors. stats are used by rf-prune
set enable_runtime_filter_prune true by default
2023-09-22 19:11:10 +08:00
4007622ad9 [Improve](inverted index) improve match performance without index (#24751) 2023-09-22 18:45:11 +08:00
c943a05065 [fix](stats) Fix data size calculation of auto sample (#24672)
1. Fix data size calculation of auto sample, before this pr, the data size is include all the replicas
2. Move some auto analyze related options to global session variable
3. Add some logs
2023-09-22 18:12:39 +08:00
b34a4779e5 [feat](optimizer) calculate stats health based on real updated rows count (#24599)
Use actual load rows since last analyze rather than delta of total row count.
2023-09-22 18:12:13 +08:00
c346f4d638 [Improvement](Storage) Lazy init mini_download dir (#24649) 2023-09-22 16:23:26 +08:00
263506f8ab [refactor](pipelineX) add MultiCast operator (#24656) 2023-09-22 15:41:14 +08:00
85a1fbd5d3 [Improve](stats)Use Log4j class library instead of Quartz (#24732)
Quartz new version not support java 8
2023-09-22 15:23:58 +08:00
74bba4bdaf [enhancement](regression-test) Add routine load case (#24536) 2023-09-22 14:55:01 +08:00
4edba083c0 [feat](Nereids) Support tablesample syntax (#23717)
Add table sample support like such SQL:

```sql
select * from test_table_sample_tbl tablesample(4 rows);
select * from test_table_sample_tbl t tablesample(20 percent);
select * from test_table_sample_tbl t tablesample(20 percent) repeatable 2;
```

This function has already been implemented in legacy planner, more detailed description could be found here: https://github.com/apache/doris/pull/10170
2023-09-22 14:24:07 +08:00
3c99743bf2 [enhancement](csv_reader)Optimize the reading efficiency of nullable (string) columns. (#24698)
Optimize the performance of stream load tsv by reducing virtual function calls .
(Optimize read performance of nullable (string) columns by reducing virtual function calls.)
before : 600+ s
after : 560+ s
2023-09-22 13:44:37 +08:00
320fc1481a [fix](Nereids) some expression not cast in in predicate (#24680)
1. should use castIfNotSameType in InPredicate and CaseWhen
2. StringLikeLiteral should override equals to ignore type
2023-09-22 12:58:33 +08:00
034582bb64 [pipelineX](fix) Fix broadcast dependency hanging (#24740) 2023-09-22 12:24:32 +08:00
d82b603b53 [bug](jsonb) fix non simd int128 overflow #24750 2023-09-22 12:23:43 +08:00
22616d125d [function](bitmap) add function alias bitmap_andnot and bitmap_andnot_count (#24771) 2023-09-22 12:18:31 +08:00
c8655d1dae [enhancement](compaction) print correct error messsage & avoid unnecessary stacktrace (#24758) 2023-09-22 11:58:11 +08:00
b5f6ace204 [fix](planner) ctas should not change any meta of column in source table (#24767)
if previous PR #22770. we try to fix incorrect nullable in target table.
However we changed nullable info of column in source table unexpectly
2023-09-22 11:49:38 +08:00
8a85a75b8b [chore](scanner) check columns' nullable with schema (#24724)
Add a validation to prevent potential schema inconsistency issues.
2023-09-22 11:34:53 +08:00
6f961ba0e9 [Enhance](external)add prepare hive data in case (#24703) 2023-09-22 11:19:46 +08:00
37a7532a49 [fix](pipelineX) get sink local state in distinctstreamingagg (#24752) 2023-09-22 11:16:46 +08:00
181a218280 Revert "[Chore](ci)Temporarily cancel the mandatory restrictions of shellCheck (#24765)" (#24770)
This reverts commit 016dd2a45ff278bd4776018f6d608a6ce5e5bb21.
2023-09-22 11:14:46 +08:00
f5c07ad6da [enhance](external)add tools for generate orc and parquet file (#24713) 2023-09-22 11:13:58 +08:00
79fe9d58b0 [bug](expr) forget write date_liter expr code info (#24687)
[bug](expr) forget write date_liter expr code info
2023-09-22 11:08:41 +08:00
e9ef6c7da7 [chore](workflow) Fix security issues in Code Checks (#24761)
The workflow `Code Checks` needs write permissions granted by the event `pull_request_target` to comment on pull requests. However, if the workflow ran users' code, the malicious code would do some dangerous actions on our repository.

The following changes are made in this PR:
1. Instead of applying patches, we use `sed` to modify the `entrypoint.sh` in action-sh-checker explicitly in the workflow.
2. Revoke the write permissions when generating `compile_commands.json` which is produced by executing the build script `build.sh`.
2023-09-22 10:39:39 +08:00
016dd2a45f [Chore](ci)Temporarily cancel the mandatory restrictions of shellCheck (#24765) 2023-09-22 10:20:13 +08:00
a718f1f6cd [fix](lock): do not use instance lock to protect static data (#24611) 2023-09-22 10:16:54 +08:00
c832e018d0 [Dependence](Fe)Upgrade Fe dependencies (#24606)
* be scanner
- Upgrade avro to 1.11.2
fe
- Upgrade quartz to 2.5.0-rc1
- Upgrade maxcompute to 0.45-2-publish
- Binding  avro-ipc  to 1.11.2

* Binding hbase  version to 2.5.5
binding nimbusds version to 9.35
2023-09-22 10:14:42 +08:00
cec3fcdf78 [typo](doc) modifying old parameters (#24565)
* modifying old parameters

* Supplementary demo
2023-09-22 09:09:06 +08:00
269b9eaf8b [typo](docs)add FAQ for broker load error: Cannot locate default realm (#24693) 2023-09-22 09:06:36 +08:00
090be20ca4 [cases](regresstests) add negative case for agg table and fix agg table support replace typ… #24715
add negative case for agg table
fix agg table support replace agg type for complex type , and Now We only support complex type with agg state for replace only
fix test output
2023-09-22 09:05:20 +08:00
e4c55cc5aa [fix](web): field in Controller is singleton, which would introduce thread safe problem, so remove it (#24667)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-22 08:58:01 +08:00
a9576df74a [typo](comments) Change the Chinese comments to English #24659 2023-09-22 08:56:56 +08:00
09e03247ec [chore](readability) Better readability of ExecNode.cpp #24733 2023-09-22 08:54:57 +08:00
665e878811 [fix](Compile) fix BE complie error #24756 2023-09-22 08:53:30 +08:00
6685875e72 [Improvement](statistics)Mark unfinished analysis job failed after master reboot
Before, the Pending/Running analysis tasks/jobs will stay in the unfinished status for ever after FE reboot, which is misleading.
In this pr, jobs/tasks are only logged to editlog when they finish. So the unfinished tasks/jobs are abandoned after reboot.
Also return without retry when analyze table cancelled by user in with sync mode.
2023-09-21 23:11:50 +08:00
48f3614ce8 [fix](nereids) a bug in estimation of intersect ndv (#24664) 2023-09-21 23:06:31 +08:00
a17034b7ff [fix](planner) do not support UDF without paramter (#24730)
for example:
CREATE ALIAS FUNCTION f() WITH PARAMETERS() AS now();
2023-09-21 22:23:21 +08:00
c9b2f4cb92 [workload](pipeline) Add cgroup cpu controller (#24052) 2023-09-21 21:49:33 +08:00
94907ca0f2 [improvement](brpc) change parking slot to 32 (#24567) 2023-09-21 21:35:46 +08:00
b8d8cfadfe [enhancement](delete) avoid null txn state after delete job committed (#24741) 2023-09-21 21:21:06 +08:00
58ab25ccaa Revert "[Feature](merge-on-write)Support ignore mode for merge-on-write unique table (#21773)" (#24731)
This reverts commit 3ee89aea35726197cb7e94bb4f2c36bc9d50da84.
2023-09-21 21:01:28 +08:00
ce14183c3a [fix](compile) fix GHA mac ut compile failure (#24738) 2023-09-21 21:00:23 +08:00
3bd341a1be [enhancement](schema) Add schema consistency check when add partition (#24707) 2023-09-21 20:35:45 +08:00
232226e75a [Improve](count on index) improve performance for count on index other than match (#24678) 2023-09-21 20:30:27 +08:00