Commit Graph

18429 Commits

Author SHA1 Message Date
ada5cf4b2e [fix](chore) fix session var memtable on sink for bulk load (#29102) 2023-12-29 17:00:02 +08:00
1a2ec2fbd9 [fix](meta)Fix error code for unknown table #27751 (#27976)
This fix will improve the error code for unknown table
exception cases.
2023-12-29 16:28:04 +08:00
236276e9c1 [typo](docs) fix lakehouse doc 404 (#29299) 2023-12-29 16:27:48 +08:00
82635d4b59 [opt](memory) All LRU Cache inherit from LRUCachePolicy (#28940)
After all LRU Cache inherits from LRUCachePolicy, this will allow prune stale entry, eviction when memory exceeds limit, and define common properties. LRUCache constructor change to private, only allow LRUCachePolicy to construct it.

Impl DummyLRUCache, when LRU Cache capacity is 0, will no longer be meaningless insert and evict.
2023-12-29 16:15:56 +08:00
866fd5d32a [feature](fe)support last column or index definition end with comma in create table statement (#29167)
allow the column list end with COMMA in create table statement. This is a issue rooted from history. So nereids has to keep the same behavior as old planner.
CREATE TABLE t
(
k1 int,
)
2023-12-29 16:12:32 +08:00
9925f7be8e [fix](nereids)FillUpMissingSlots rule didn't process standalone having clause correctly (#29143) 2023-12-29 16:12:05 +08:00
c3c34e10bb [feature](executor) Add some check when create workload group/workload schedule policy (#29236) 2023-12-29 15:41:16 +08:00
d6dcf962a9 [Enhancement](page cache) insert into setting to disable page cache (#28913) 2023-12-29 15:11:41 +08:00
a7c4abde9f [fix](regression test) fix test_schema_change_concurrent_with_txn (#28994) 2023-12-29 15:07:11 +08:00
24cdc00b60 [improvement](doris compose) avoid docker network conflict (#28957) 2023-12-29 15:05:28 +08:00
660f12c717 [dependency](fe)Upgrade dependency library (#29049) 2023-12-29 14:50:46 +08:00
411f819306 [Opt](join) opt the join copy column from build side (#29007) 2023-12-29 14:39:33 +08:00
2308881e9f [improvement](statistics) Analyze partition columns when new partition loaded data for the first time. (#29154)
The first time load data to a partition, we need to analyze the partition columns even when the health rate is high. Because if not, the min max value of the column may not include the new partition values, which may cause bad plan.
2023-12-29 14:36:48 +08:00
02b27c1673 [ci](trigger) support "run performance" (#29259)
* [ci](trigger) support "run performance"
Co-authored-by: stephen <hello-stephen@qq.com>
2023-12-29 14:21:23 +08:00
xy
4fc5860a7c [Fix](doc)Fix word spelling errors (#29268)
Co-authored-by: xingying01 <xingying01@corp.netease.com>
2023-12-29 14:15:37 +08:00
36b45b570b [fix](mysql) fix mysql channel infinite blocking (#28808)
Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.
2023-12-29 13:57:22 +08:00
738a1644ff [refactor](operator) We dont need OperatorBuilder to instantiate Operator (#29133) 2023-12-29 13:54:54 +08:00
f34b46a366 [fix](glue) support amazonaws.com.cn endpoint (#29128) 2023-12-29 13:50:30 +08:00
9fc613de9c [fix](nereids) Fix query rewrite by mv fail when self join (#29227)
Fix query rewrite by mv fail when self join, after fix query like following can be rewrited

def materialized view = """
    select 
    a.o_orderkey,
    count(distinct a.o_orderstatus) num1,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate = '2023-12-08' AND b.o_orderdate = '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num2,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate >= '2023-12-01' AND a.o_orderdate <= '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num3,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority in (1,2) AND a.o_orderdate >= '2023-12-08' AND b.o_orderdate <= '2023-12-09' THEN a.o_shippriority-b.o_custkey ELSE 0 END) num4,
    AVG(a.o_totalprice) num5,
    MAX(b.o_totalprice) num6,
    MIN(a.o_totalprice) num7
    from
    orders a
    left outer join orders b
    on a.o_orderkey = b.o_orderkey
    and a.o_custkey = b.o_custkey
    group by a.o_orderkey;
"""

def query = """
    select 
    a.o_orderkey,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate = '2023-12-08' AND b.o_orderdate = '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num2,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate >= '2023-12-01' AND a.o_orderdate <= '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num3,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority in (1,2) AND a.o_orderdate >= '2023-12-08' AND b.o_orderdate <= '2023-12-09' THEN a.o_shippriority-b.o_custkey ELSE 0 END) num4,
    AVG(a.o_totalprice) num5,
    MAX(b.o_totalprice) num6,
    MIN(a.o_totalprice) num7
    from
    orders a
    left outer join orders b
    on a.o_orderkey = b.o_orderkey
    and a.o_custkey = b.o_custkey
    group by a.o_orderkey;
"""
2023-12-29 13:45:33 +08:00
2794427e7f [enhancement](Nereids): refactor eliminating inner join by foreign key (#28816) 2023-12-29 13:41:54 +08:00
b9572f9de0 [pipelineX](fix) Fix pip scanner context bug (#29229) 2023-12-29 13:24:39 +08:00
70f5a26f44 [pipelineX](fix) Fix heap-use-after-free for AggSource dependency (#29272) 2023-12-29 12:50:41 +08:00
a57ad36c4e [fix](docs) Update Link for 404 page of dev docs (#29213) 2023-12-29 11:37:50 +08:00
3cddb597c6 [fix](regression) fix pipeline load regression test error (#29216)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-12-29 11:06:20 +08:00
48d41a8c8b [feature](Nereids): support comparing mv with inferred predicate (#29132) 2023-12-29 10:38:53 +08:00
253846a99d [improve](compaction) enable compaction priority scheduling (#29261) 2023-12-29 10:13:08 +08:00
a525d5c5a3 [refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion (#29265)
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
2023-12-29 10:11:44 +08:00
d2dc12bed5 [fix](nereids)exists subquery should handle top level scarlar agg correctly (#29135) 2023-12-29 09:45:20 +08:00
61677d1d4b [ci](perf) 1. add perf check of tpcds, 2. adjust clickbench and tpch check (#28431) 2023-12-29 09:26:15 +08:00
269c1b189d [improve](vtablet_writer) check runtime state is cancel when back pressure (#29260) 2023-12-29 09:11:24 +08:00
2f29dda5aa [Fix](core) Fix file system scan deleted file (#29266) 2023-12-29 09:07:59 +08:00
c3679a2750 [opt](Nereids) derive physical properties of Project and Filter (#29171) 2023-12-29 07:08:12 +08:00
7d44c5a1f1 [FIX](map)fix element_at in old planner make fe exception and regress cases from ck #29241 2023-12-29 01:00:47 +08:00
ce13a1d951 [fix](nereids) make runtime filter order stable #29203 2023-12-29 01:00:27 +08:00
69c90b1640 [fix](group commit)fix group commit regresstion test (#29079) 2023-12-29 00:50:22 +08:00
825732a395 [fix](regression) fix unstable test case (#29264) 2023-12-29 00:45:34 +08:00
fb0ed8c253 [fix](move-memtable) check missing tablets before commit (#29223) 2023-12-29 00:33:58 +08:00
efea006f3a [ut](move-memtable) add CLOSE_LOAD before EOS ut case (#29253)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-12-29 00:33:34 +08:00
99a1e066b5 [fix](group_commit) group_commit is not support on table with property light_schema_change=false (#29244) 2023-12-29 00:26:38 +08:00
9be0f04506 (improv)[group commit] refactor some group commit code (#29180) 2023-12-29 00:26:10 +08:00
9a277a6f11 [fix](move-memtable) don't abort in replica write layer unless all replica fails (#29257) 2023-12-29 00:03:28 +08:00
feebe3e6fb [FIX](literal) fix expression literal error #29157 2023-12-28 23:08:01 +08:00
a90304c208 [fix](parquet) complex type in parquet is case sensitive (#29245)
Change name of complex type in parquet to case-insensitive. Otherwise, uppercase column names of complex types will return null.
2023-12-28 22:43:11 +08:00
8a491e7b1d Fix workload scheduler start too early may cause npe (#29258) 2023-12-28 22:41:42 +08:00
e64c5687f2 [fix](index compaction)support compact multi segments in one index (#28889) 2023-12-28 21:33:21 +08:00
ffd178f5ff [feat](pipelinex) support parallel scan on pipeline x engine (#29070)
* [feat](pipelinex) support parallel scan on pipeline x engine

* make parallel scan be independent of shared scan
2023-12-28 21:29:07 +08:00
0912b137e6 [Improvement](pipelineX) optimize local exchange sink (#29250) 2023-12-28 21:22:29 +08:00
b093097bc3 [improvement](statistic)Improve auto analyze visibility. (#29046)
Show auto analyze can show the running jobs, not only the finished/failed jobs.
Show analyze task status could show auto tasks as well.
Remove some useless code.
Auto analyze execute catalog/db/table in the order of id, small id first.
2023-12-28 21:21:17 +08:00
5129ab5738 [fix](decimalv2) fix decimalv2 agg errors (#29246) 2023-12-28 21:17:16 +08:00
c8a0d3e03c [fix](invert index) fix error handling for match_regexp resulting in an empty match. (#29233) 2023-12-28 19:58:41 +08:00