Commit Graph

15989 Commits

Author SHA1 Message Date
03ece437f0 Fix topicPublisher thread may NPE when no topic exists (#29306) 2023-12-29 23:04:20 +08:00
59fdd5e42b [fix](compaction) time series compaction policy -> compact consecutive empty rowsets (#27299)
Sometimes we need to merge a large number of empty versions to reduce meta-information.
2023-12-29 22:20:41 +08:00
fcb90dc808 [opt](Nereids) use enable_nereids_dml to control ctas and create table (#29284) 2023-12-29 20:47:22 +08:00
07bd65abe0 [pipelineX](local shuffle) remove unused code in local shuffle to improve performance #29292 2023-12-29 20:30:28 +08:00
7c5fda1169 [thirdparty](lib) Add streamvbyte thirdparty lib (#29308) 2023-12-29 19:22:07 +08:00
f0629da10a [pipelineX](fix) Fix illegal memory access (#29283) 2023-12-29 18:35:41 +08:00
4266432d11 [Fix](dialect) Fix trino dialect converter when sql does not end with delimiter. (#29106)
Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
2023-12-29 18:02:57 +08:00
7604401b06 [Enhance](regression)Do path creation ahead of time for case test_export_external_table (#28616)
Do path creation ahead of time for case test_export_external_table
2023-12-29 17:59:20 +08:00
7e0616ba77 [fix](regression test) fix test_publish_one_succ (#28993) 2023-12-29 17:31:55 +08:00
9c57483aa3 [ci](trigger) only master run performance check (#29290)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-12-29 17:01:31 +08:00
ada5cf4b2e [fix](chore) fix session var memtable on sink for bulk load (#29102) 2023-12-29 17:00:02 +08:00
1a2ec2fbd9 [fix](meta)Fix error code for unknown table #27751 (#27976)
This fix will improve the error code for unknown table
exception cases.
2023-12-29 16:28:04 +08:00
236276e9c1 [typo](docs) fix lakehouse doc 404 (#29299) 2023-12-29 16:27:48 +08:00
82635d4b59 [opt](memory) All LRU Cache inherit from LRUCachePolicy (#28940)
After all LRU Cache inherits from LRUCachePolicy, this will allow prune stale entry, eviction when memory exceeds limit, and define common properties. LRUCache constructor change to private, only allow LRUCachePolicy to construct it.

Impl DummyLRUCache, when LRU Cache capacity is 0, will no longer be meaningless insert and evict.
2023-12-29 16:15:56 +08:00
866fd5d32a [feature](fe)support last column or index definition end with comma in create table statement (#29167)
allow the column list end with COMMA in create table statement. This is a issue rooted from history. So nereids has to keep the same behavior as old planner.
CREATE TABLE t
(
k1 int,
)
2023-12-29 16:12:32 +08:00
9925f7be8e [fix](nereids)FillUpMissingSlots rule didn't process standalone having clause correctly (#29143) 2023-12-29 16:12:05 +08:00
c3c34e10bb [feature](executor) Add some check when create workload group/workload schedule policy (#29236) 2023-12-29 15:41:16 +08:00
d6dcf962a9 [Enhancement](page cache) insert into setting to disable page cache (#28913) 2023-12-29 15:11:41 +08:00
a7c4abde9f [fix](regression test) fix test_schema_change_concurrent_with_txn (#28994) 2023-12-29 15:07:11 +08:00
24cdc00b60 [improvement](doris compose) avoid docker network conflict (#28957) 2023-12-29 15:05:28 +08:00
660f12c717 [dependency](fe)Upgrade dependency library (#29049) 2023-12-29 14:50:46 +08:00
411f819306 [Opt](join) opt the join copy column from build side (#29007) 2023-12-29 14:39:33 +08:00
2308881e9f [improvement](statistics) Analyze partition columns when new partition loaded data for the first time. (#29154)
The first time load data to a partition, we need to analyze the partition columns even when the health rate is high. Because if not, the min max value of the column may not include the new partition values, which may cause bad plan.
2023-12-29 14:36:48 +08:00
02b27c1673 [ci](trigger) support "run performance" (#29259)
* [ci](trigger) support "run performance"
Co-authored-by: stephen <hello-stephen@qq.com>
2023-12-29 14:21:23 +08:00
xy
4fc5860a7c [Fix](doc)Fix word spelling errors (#29268)
Co-authored-by: xingying01 <xingying01@corp.netease.com>
2023-12-29 14:15:37 +08:00
36b45b570b [fix](mysql) fix mysql channel infinite blocking (#28808)
Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.
2023-12-29 13:57:22 +08:00
738a1644ff [refactor](operator) We dont need OperatorBuilder to instantiate Operator (#29133) 2023-12-29 13:54:54 +08:00
f34b46a366 [fix](glue) support amazonaws.com.cn endpoint (#29128) 2023-12-29 13:50:30 +08:00
9fc613de9c [fix](nereids) Fix query rewrite by mv fail when self join (#29227)
Fix query rewrite by mv fail when self join, after fix query like following can be rewrited

def materialized view = """
    select 
    a.o_orderkey,
    count(distinct a.o_orderstatus) num1,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate = '2023-12-08' AND b.o_orderdate = '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num2,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate >= '2023-12-01' AND a.o_orderdate <= '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num3,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority in (1,2) AND a.o_orderdate >= '2023-12-08' AND b.o_orderdate <= '2023-12-09' THEN a.o_shippriority-b.o_custkey ELSE 0 END) num4,
    AVG(a.o_totalprice) num5,
    MAX(b.o_totalprice) num6,
    MIN(a.o_totalprice) num7
    from
    orders a
    left outer join orders b
    on a.o_orderkey = b.o_orderkey
    and a.o_custkey = b.o_custkey
    group by a.o_orderkey;
"""

def query = """
    select 
    a.o_orderkey,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate = '2023-12-08' AND b.o_orderdate = '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num2,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate >= '2023-12-01' AND a.o_orderdate <= '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num3,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority in (1,2) AND a.o_orderdate >= '2023-12-08' AND b.o_orderdate <= '2023-12-09' THEN a.o_shippriority-b.o_custkey ELSE 0 END) num4,
    AVG(a.o_totalprice) num5,
    MAX(b.o_totalprice) num6,
    MIN(a.o_totalprice) num7
    from
    orders a
    left outer join orders b
    on a.o_orderkey = b.o_orderkey
    and a.o_custkey = b.o_custkey
    group by a.o_orderkey;
"""
2023-12-29 13:45:33 +08:00
2794427e7f [enhancement](Nereids): refactor eliminating inner join by foreign key (#28816) 2023-12-29 13:41:54 +08:00
b9572f9de0 [pipelineX](fix) Fix pip scanner context bug (#29229) 2023-12-29 13:24:39 +08:00
70f5a26f44 [pipelineX](fix) Fix heap-use-after-free for AggSource dependency (#29272) 2023-12-29 12:50:41 +08:00
a57ad36c4e [fix](docs) Update Link for 404 page of dev docs (#29213) 2023-12-29 11:37:50 +08:00
3cddb597c6 [fix](regression) fix pipeline load regression test error (#29216)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-12-29 11:06:20 +08:00
48d41a8c8b [feature](Nereids): support comparing mv with inferred predicate (#29132) 2023-12-29 10:38:53 +08:00
253846a99d [improve](compaction) enable compaction priority scheduling (#29261) 2023-12-29 10:13:08 +08:00
a525d5c5a3 [refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion (#29265)
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
2023-12-29 10:11:44 +08:00
d2dc12bed5 [fix](nereids)exists subquery should handle top level scarlar agg correctly (#29135) 2023-12-29 09:45:20 +08:00
61677d1d4b [ci](perf) 1. add perf check of tpcds, 2. adjust clickbench and tpch check (#28431) 2023-12-29 09:26:15 +08:00
269c1b189d [improve](vtablet_writer) check runtime state is cancel when back pressure (#29260) 2023-12-29 09:11:24 +08:00
2f29dda5aa [Fix](core) Fix file system scan deleted file (#29266) 2023-12-29 09:07:59 +08:00
c3679a2750 [opt](Nereids) derive physical properties of Project and Filter (#29171) 2023-12-29 07:08:12 +08:00
7d44c5a1f1 [FIX](map)fix element_at in old planner make fe exception and regress cases from ck #29241 2023-12-29 01:00:47 +08:00
ce13a1d951 [fix](nereids) make runtime filter order stable #29203 2023-12-29 01:00:27 +08:00
69c90b1640 [fix](group commit)fix group commit regresstion test (#29079) 2023-12-29 00:50:22 +08:00
825732a395 [fix](regression) fix unstable test case (#29264) 2023-12-29 00:45:34 +08:00
fb0ed8c253 [fix](move-memtable) check missing tablets before commit (#29223) 2023-12-29 00:33:58 +08:00
efea006f3a [ut](move-memtable) add CLOSE_LOAD before EOS ut case (#29253)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-12-29 00:33:34 +08:00
99a1e066b5 [fix](group_commit) group_commit is not support on table with property light_schema_change=false (#29244) 2023-12-29 00:26:38 +08:00
9be0f04506 (improv)[group commit] refactor some group commit code (#29180) 2023-12-29 00:26:10 +08:00