Commit Graph

18429 Commits

Author SHA1 Message Date
e214eb1ea7 [chore](ci) fix ci check (#32992)
Co-authored-by: stephen <hello-stephen@qq.com>
2024-04-10 11:34:30 +08:00
3ee14a80ab [chore](ci) adjust ckb expect result (#32856)
Co-authored-by: stephen <hello-stephen@qq.com>
2024-04-10 11:34:30 +08:00
7e802c9127 [fix](variant)group name optimization (#32598)
1. Change the group name stars to repo_name
2024-04-10 11:34:30 +08:00
a6fc2ae176 [fix](test) replace 'null' to null for date/datetime column (#32972) 2024-04-10 11:34:30 +08:00
528a889077 [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than hive_sink_max_file_size. (#32959) 2024-04-10 11:34:29 +08:00
97a2977f2a [improvement](executor)Add tag property for workload group #32874 2024-04-10 11:34:29 +08:00
f1ee7f5767 [fix](merge-iterator) fix NOT_IMPLEMENTED_ERROR when read next block view (#32961) 2024-04-10 11:34:29 +08:00
dcddd88e01 Limit the max string length to 1024 while collecting column stats to control BE memory usage. (#32470) 2024-04-10 11:34:29 +08:00
0499d4013e Support identical column name in different index. (#32792) 2024-04-10 11:34:29 +08:00
121a0901c8 [chore](test) let suite compatible with Nereids (#32964) 2024-04-10 11:34:29 +08:00
407f8642da [Enhancement](data skew) extends show data skew (#32732) 2024-04-10 11:34:29 +08:00
3a6c37c6d5 [exec](column) change some complex column move to noexcept (#32954) 2024-04-10 11:34:29 +08:00
ed0949f6c5 [fix](compile) fe cannot compile in idea (#32955) 2024-04-10 11:34:29 +08:00
e980cd3e7f [feature](Nereids): add ColumnPruningPostProcessor. (#32800) 2024-04-10 11:34:29 +08:00
26e86d53a4 [enhance](mtmv)support olap table partition column is null (#32698) 2024-04-10 11:34:29 +08:00
22a7fc3c55 [improvement](mtmv) Support to get tables in materialized view when collecting table in plan (#32797)
Support to get tables in materialized view when collecting table in plan

table scehma as fllowing:

create materialized view mv1
BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 1 
PROPERTIES ('replication_num' = '1')
 as 
select 
  t1.c1, 
  t3.c2 
from 
  table1 t1 
  inner join table3 t3 on t1.c1 = t3.c2

if get table from the plan as follwoing, we can get [table1, table3, table2], the mv1 is expanded to get base tables;

SELECT 
  mv1.*, 
  uuid() 
FROM 
  mv1 LEFT SEMI 
  JOIN table2 ON mv1.c1 = table2.c1 
WHERE 
  mv1.c1 IN (
    SELECT 
      c1 
    FROM 
      table2
  ) 
  OR mv1.c1 < 10
2024-04-10 11:34:29 +08:00
bb8bc75af4 [feature](agg) add aggregate function sum0 (#32541) 2024-04-10 11:34:29 +08:00
e3bd2311b1 [Performance](exec) replace SipHash in function by XXHash (#32919) 2024-04-10 11:34:29 +08:00
28e2d89ce3 [Improve](inverted_index) update clucene and improve array inverted index writer (#32436) 2024-04-10 11:34:29 +08:00
8e6ed80692 [improvement](spill) Disable DistinctStreamingAgg when spill is enabled (#32932) 2024-04-10 11:34:29 +08:00
830d4c14c1 [fix](spill) SpillStream's writer maybe may not have been finalized (#32931) 2024-04-10 11:34:29 +08:00
80cdc74908 [fix](arrow-flight) Fix reach limit of connections error (#32911)
Fix Reach limit of connections error
in fe.conf , arrow_flight_token_cache_size is mandatory less than qe_max_connection/2. arrow flight sql is a stateless protocol, connection is usually not actively disconnected, bearer token is evict from the cache will unregister ConnectContext.

Fix ConnectContext.command not be reset to COM_SLEEP in time, this will result in frequent kill connection after query timeout.

Fix bearer token evict log and exception.

TODO: use arrow flight session: https://mail.google.com/mail/u/0/#inbox/FMfcgzGxRdxBLQLTcvvtRpqsvmhrHpdH
2024-04-10 11:34:29 +08:00
d959291c98 [improvement](decommission be) decommission check replica num (#32748) 2024-04-10 11:34:28 +08:00
06e5c6c966 [fix](grace-exit) Stop incorrectly of reportwork cause heap use after free #32929 2024-04-10 11:34:28 +08:00
f23a72b937 [chore](log) print query id before logging profile in be.INFO (#32922) 2024-04-10 11:34:28 +08:00
87f99271e1 [fix](spill) Avoid releasing resources while spill tasks are executing (#32783) 2024-04-10 11:34:28 +08:00
f5340039fc [fix](multicatalog) fix no data error when read hive table on cosn (#32815)
Currently, when reading a hive on cosn table, doris return empty result, but the table has data.
iceberg on cosn is ok.
The reason is misuse of cosn's file sytem. according to cosn's doc, its fs.cosn.impl should be org.apache.hadoop.fs.CosFileSystem
2024-04-10 11:34:28 +08:00
66536c2976 [fix](Nereids) NPE when create table with implicit index type (#32893) 2024-04-10 11:34:28 +08:00
dcfdbf0629 [chore](show) support statement to show views from table (#32358)
MySQL [test]> show views;
+----------------+
| Tables_in_test |
+----------------+
| t1_view        |
| t2_view        |
+----------------+
2 rows in set (0.00 sec)

MySQL [test]> show views like '%t1%';
+----------------+
| Tables_in_test |
+----------------+
| t1_view        |
+----------------+
1 row in set (0.01 sec)

MySQL [test]> show views where create_time > '2024-03-18';
+----------------+
| Tables_in_test |
+----------------+
| t2_view        |
+----------------+
1 row in set (0.02 sec)
2024-04-10 11:34:28 +08:00
96b995504c [enhancement](statistics) excluded delta rows num for rollup&mv tablets (#32568)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Co-authored-by: tsy <tangsiyang2001@foxmail.com>
2024-04-10 11:34:28 +08:00
e8d67e79fd [fix](test) fix duplicated catalog name in regression cases (#32442)
Each suite should use different catalog name,
other it will effect each other when running cases concurrently.
2024-04-10 11:34:28 +08:00
217514e5dd [minor](test) Add Iceberg hadoop catalog FE unit test (#32449)
For easy testing the behavior of Iceberg's HadoopCatalog.listNamespaces()
2024-04-10 11:34:28 +08:00
b130df2488 2.1.2-rc04 2024-04-09 16:28:27 +08:00
c35b2becdd [fix][docker] fix kafka test scritps (#33417)
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>
2024-04-09 16:11:09 +08:00
005f7af21f [bugfix](deadlock) should not use query cancelled in fragment mgr 2024-04-09 16:09:01 +08:00
e574b35833 [Enhancement](partition) Refine some auto partition behaviours (#32737) (#33412)
fix legacy planner grammer
fix nereids planner parsing
fix cases
forbid auto range partition with null column
fix CreateTableStmt with auto partition and some partition items.
1 and 2 are about #31585
doc pr: apache/doris-website#488
2024-04-09 15:51:02 +08:00
97850cf2bb [fix](cooldown) Fix hdfs path (#33315) 2024-04-09 12:55:53 +08:00
a1f80eaa7a 2.1.2-rc03 2024-04-09 12:49:05 +08:00
2a0644f442 [Fix](function) Fix unix_timestamp core for string input (#32871) 2024-04-09 12:48:35 +08:00
b5b0181a79 2.1.2-rc02 2024-04-09 12:37:31 +08:00
3c4ccb3981 Revert "[opt](scan) read scan ranges in the order of partitions (#31630)"
This reverts commit 5d99dffe6f1a3fcb107ce56181aeff96ef222def.
2024-04-09 12:37:31 +08:00
bfc9260507 [bugfix](deadlock) avoid deadlock in memtracker cancel query (#33400)
get_query_ctx(hold query ctx map lock) ---> QueryCtx ---> runtime statistics mgr --->

runtime statistics mgr ---> allocate block memory ---> cancel query

memtracker will try to cancel query when memory is not available during allocator.
BUT the allocator is a foundermental API, if it call the upper API it may deadlock.
Should not call any API during allocator.
2024-04-09 12:20:54 +08:00
0c8d3d007d [fix](jni) don't delete global ref if scanner is not openned (#33398) 2024-04-09 09:06:16 +08:00
0e1a15960c 2.1.2-rc01 2024-04-08 23:17:15 +08:00
7892e7300f [fix](external catalog) Reset external table creation status on log replay (#33393) 2024-04-08 23:17:15 +08:00
4d98fe23a2 [enhancement](rpc) should print fe address in error msg during thrift rpc call (#33381)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-04-08 23:10:17 +08:00
dbf2326f62 [regression-test](case) fix unstable test case in multi fe env (#33385)
* [regression-test](case) fix unstable test case in multi fe env
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-04-08 23:10:17 +08:00
0234976ab7 [refactor](meta scan) Remove RPC from execute threads (#33378) 2024-04-08 20:28:02 +08:00
5e5fffe4e3 Set enable_unique_key_partial_update to false in statistics session variable. (#33220) 2024-04-08 16:49:58 +08:00
a8232c67f9 [pipelineX](runtime filter) Fix task timeout caused by runtime filter (#33332) (#33369) 2024-04-08 16:30:32 +08:00