Commit Graph

7142 Commits

Author SHA1 Message Date
eac96005b3 [Fix](Job)Concurrency may result in event loss (#29385) 2024-01-02 14:06:16 +08:00
90b2ee90b2 [nereids] consider numNulls in filter estimation (#29184)
consider numNulls in filter estimation
2024-01-02 13:51:11 +08:00
4cbbd25d8c [fix](ctx) manager the lifecycle of connection context (#29346)
In FrontendService, we may create some connection context and set it as a thread local varaible.
These context should be removed from thread local after call.
Otherwise, it may be reused by other thread incorrectly.
2024-01-01 23:32:28 +08:00
3d59f24403 [fix](compatibility) add some keywords (#29251)
1. Add `BINARY` keywords
    `BINARY` is used for tell MySQL to treat a String(or String column) as case sensitive.
     eg: `"abc" = "ABC"` is true, but `BINARY "abc" = "ABC"` is false.

    But in Doris, `"abc" = "ABC"` is false by default.

     I add this `BINARY` keyword just for compatibility, it will take no effect.

2. Add `PARTITIONS` and `AUTO_INCREMENT` as reserved words.

    `PARTITIONS` is the table name in `information_schema` database.
    `AUTO_INCREMENT` is a column name of a table in `information_schema`
2024-01-01 17:20:45 +08:00
01a0f0915f [fix](mtmv)fix insert overwrite will generate garbage temporary partition when restarting FE (#29075) 2024-01-01 08:12:26 +08:00
5985d216f3 [feature](mtmv)support cancel mtmv task command (#29252)
- `CANCEL MATERIALIZED VIEW TASK taskId on mvName`
- CANCEL MATERIALIZED VIEW TASK, tasks("type"="mv") and jobs("type"="mv") support check auth use priv of mv
- tasks and jobs add column mvName and mvDbName,you can use `select * from tasks("type"="mv") where MvName="xxx"` get all tasks of mv
- fix `desc mv all` error
- fix p0 The task sequence is incorrect
2023-12-31 23:10:30 +08:00
3c6c652997 [Fix](schema change) disable convert light schema change (#28205) (#29300) 2023-12-31 17:02:15 +08:00
6c74313f60 [Fix](ut)(show) added UT cases for show backup stmt (#28604)
* Added UT for show backup
 * fixed some toSql() space and missing CLAUSE
 * Updated Doc with relevant syntax
2023-12-31 15:49:57 +08:00
f92c0acba0 [Bug] When using the hive catalog the hive table name is uppercase, the Doris query is reported wrong #29058 (#29059) 2023-12-31 13:48:08 +08:00
ac636a2aba [fix](nereids) partition prunner evaluates "not x=const" on single node range #29164 2023-12-31 08:50:16 +08:00
6aaa829870 [fix](Nereids) connection_id() should return bigint type (#29287) 2023-12-30 22:25:27 +08:00
56c3487a10 [test](ut) added UT cases for show backend stmt (#28603) 2023-12-30 20:55:28 +08:00
f5e2ea1699 [fix][fe]Fixed show proc tablet inaccuracies (#29186)
Both tables simply need to change the tablet id to get the same result from the show proc statement.

before:
image
Just change the tablet id to get the result, 10139 does not belong to the tablet in the table id of 10127

later:
image
2023-12-30 20:49:01 +08:00
13f3d30652 [fix](hive-udf) Fixed the calculation result error in Hive bitmap UDF (#29073) 2023-12-30 17:25:33 +08:00
80fb496523 [improve](proc) Show journal size in bdbje journal dir (#29314) 2023-12-30 16:43:06 +08:00
03901b9a7a [enhancement](group_commit): refector relay wal code (#29183) 2023-12-30 12:59:46 +08:00
2c4e52e44e [fix](es catalog) only es_query function can push down to ES (#29320)
Issue Number: close #29318 
1. Only push down `es_query` function to ES
2. Add null check where ES query result not have `_source` or `fields` fields.
2023-12-30 09:33:26 +08:00
8407490053 [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql (#28531) 2023-12-30 00:10:35 +08:00
445f72b395 [ut](stats) Added tests for HMS analysis tasks (#28583) 2023-12-30 00:09:50 +08:00
0bf0f945a2 [test](ut) added UT cases for show broker stmt (#28629) 2023-12-30 00:04:01 +08:00
7f105facc4 [fix](pipelineX) fix unable to merge profiles in multi be (#29204) 2023-12-29 23:48:46 +08:00
2518ed64ea [enhancement](bulk-load) strict second level granularity for timeout between (0,1) (#29028) 2023-12-29 23:44:43 +08:00
989d20e0ac [opt](Nereids) remove decimalv2 signature from min, max, sum, nvl and case when (#29282) 2023-12-29 23:22:32 +08:00
03ece437f0 Fix topicPublisher thread may NPE when no topic exists (#29306) 2023-12-29 23:04:20 +08:00
59fdd5e42b [fix](compaction) time series compaction policy -> compact consecutive empty rowsets (#27299)
Sometimes we need to merge a large number of empty versions to reduce meta-information.
2023-12-29 22:20:41 +08:00
fcb90dc808 [opt](Nereids) use enable_nereids_dml to control ctas and create table (#29284) 2023-12-29 20:47:22 +08:00
4266432d11 [Fix](dialect) Fix trino dialect converter when sql does not end with delimiter. (#29106)
Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
2023-12-29 18:02:57 +08:00
ada5cf4b2e [fix](chore) fix session var memtable on sink for bulk load (#29102) 2023-12-29 17:00:02 +08:00
1a2ec2fbd9 [fix](meta)Fix error code for unknown table #27751 (#27976)
This fix will improve the error code for unknown table
exception cases.
2023-12-29 16:28:04 +08:00
866fd5d32a [feature](fe)support last column or index definition end with comma in create table statement (#29167)
allow the column list end with COMMA in create table statement. This is a issue rooted from history. So nereids has to keep the same behavior as old planner.
CREATE TABLE t
(
k1 int,
)
2023-12-29 16:12:32 +08:00
9925f7be8e [fix](nereids)FillUpMissingSlots rule didn't process standalone having clause correctly (#29143) 2023-12-29 16:12:05 +08:00
c3c34e10bb [feature](executor) Add some check when create workload group/workload schedule policy (#29236) 2023-12-29 15:41:16 +08:00
d6dcf962a9 [Enhancement](page cache) insert into setting to disable page cache (#28913) 2023-12-29 15:11:41 +08:00
660f12c717 [dependency](fe)Upgrade dependency library (#29049) 2023-12-29 14:50:46 +08:00
2308881e9f [improvement](statistics) Analyze partition columns when new partition loaded data for the first time. (#29154)
The first time load data to a partition, we need to analyze the partition columns even when the health rate is high. Because if not, the min max value of the column may not include the new partition values, which may cause bad plan.
2023-12-29 14:36:48 +08:00
36b45b570b [fix](mysql) fix mysql channel infinite blocking (#28808)
Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.
2023-12-29 13:57:22 +08:00
f34b46a366 [fix](glue) support amazonaws.com.cn endpoint (#29128) 2023-12-29 13:50:30 +08:00
9fc613de9c [fix](nereids) Fix query rewrite by mv fail when self join (#29227)
Fix query rewrite by mv fail when self join, after fix query like following can be rewrited

def materialized view = """
    select 
    a.o_orderkey,
    count(distinct a.o_orderstatus) num1,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate = '2023-12-08' AND b.o_orderdate = '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num2,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate >= '2023-12-01' AND a.o_orderdate <= '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num3,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority in (1,2) AND a.o_orderdate >= '2023-12-08' AND b.o_orderdate <= '2023-12-09' THEN a.o_shippriority-b.o_custkey ELSE 0 END) num4,
    AVG(a.o_totalprice) num5,
    MAX(b.o_totalprice) num6,
    MIN(a.o_totalprice) num7
    from
    orders a
    left outer join orders b
    on a.o_orderkey = b.o_orderkey
    and a.o_custkey = b.o_custkey
    group by a.o_orderkey;
"""

def query = """
    select 
    a.o_orderkey,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate = '2023-12-08' AND b.o_orderdate = '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num2,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority = 1 AND a.o_orderdate >= '2023-12-01' AND a.o_orderdate <= '2023-12-09' THEN a.o_shippriority+b.o_custkey ELSE 0 END) num3,
    SUM(CASE WHEN a.o_orderstatus = 'o' AND a.o_shippriority in (1,2) AND a.o_orderdate >= '2023-12-08' AND b.o_orderdate <= '2023-12-09' THEN a.o_shippriority-b.o_custkey ELSE 0 END) num4,
    AVG(a.o_totalprice) num5,
    MAX(b.o_totalprice) num6,
    MIN(a.o_totalprice) num7
    from
    orders a
    left outer join orders b
    on a.o_orderkey = b.o_orderkey
    and a.o_custkey = b.o_custkey
    group by a.o_orderkey;
"""
2023-12-29 13:45:33 +08:00
2794427e7f [enhancement](Nereids): refactor eliminating inner join by foreign key (#28816) 2023-12-29 13:41:54 +08:00
48d41a8c8b [feature](Nereids): support comparing mv with inferred predicate (#29132) 2023-12-29 10:38:53 +08:00
d2dc12bed5 [fix](nereids)exists subquery should handle top level scarlar agg correctly (#29135) 2023-12-29 09:45:20 +08:00
c3679a2750 [opt](Nereids) derive physical properties of Project and Filter (#29171) 2023-12-29 07:08:12 +08:00
7d44c5a1f1 [FIX](map)fix element_at in old planner make fe exception and regress cases from ck #29241 2023-12-29 01:00:47 +08:00
ce13a1d951 [fix](nereids) make runtime filter order stable #29203 2023-12-29 01:00:27 +08:00
69c90b1640 [fix](group commit)fix group commit regresstion test (#29079) 2023-12-29 00:50:22 +08:00
99a1e066b5 [fix](group_commit) group_commit is not support on table with property light_schema_change=false (#29244) 2023-12-29 00:26:38 +08:00
9be0f04506 (improv)[group commit] refactor some group commit code (#29180) 2023-12-29 00:26:10 +08:00
feebe3e6fb [FIX](literal) fix expression literal error #29157 2023-12-28 23:08:01 +08:00
ffd178f5ff [feat](pipelinex) support parallel scan on pipeline x engine (#29070)
* [feat](pipelinex) support parallel scan on pipeline x engine

* make parallel scan be independent of shared scan
2023-12-28 21:29:07 +08:00
b093097bc3 [improvement](statistic)Improve auto analyze visibility. (#29046)
Show auto analyze can show the running jobs, not only the finished/failed jobs.
Show analyze task status could show auto tasks as well.
Remove some useless code.
Auto analyze execute catalog/db/table in the order of id, small id first.
2023-12-28 21:21:17 +08:00