Commit Graph

18446 Commits

Author SHA1 Message Date
a391cf6bfe [fix](Nereids) rewritten mv should check output set and should not return null (#34288)
1. we should check output set, since we will remove top project and
   result output size will diff with its child output size if there are
   dup slot in result list
2. should not return null, instead we should return rewritten plan
   itself, because we will use return result in many place and do not
   check null at all
2024-05-07 07:44:16 +08:00
9d0d7293f0 [fix](json) fix be crash while load json data (#34283) 2024-05-07 07:42:53 +08:00
8db4d48731 [fix](statistics) Use column update rows to decide min/max stats are valid or not (#34263)
This is a following pr of #33685
After #33703 merged, need to check update rows in column level instead of table level.
2024-05-07 07:41:28 +08:00
971e10a9db [fix](csv-reader) fix column split error when there is escape character (#34364) 2024-05-07 07:38:35 +08:00
ad35968236 [Fix](Job)Job repaly logic error (#34378) 2024-05-07 07:37:14 +08:00
3fd3dfe16f [Feat](Job) Job supports task execution statistics (#34109)
* Support statistics

* - Fix Failed task not showing up in the task list
- Task metadata add jobName
- Fix Finished job clear time error
- Job metadata add successCount, failedCount, totalTaskCount

* add test
2024-05-07 07:36:54 +08:00
956ae2f83d [opt](Nereids) let behavior of function char same with legacy planner (#34415)
1. first argument must be string like literal
2. only support utf-8 charset
2024-05-07 07:34:34 +08:00
8fdfbcb3c4 Revert "[Opt](func) opt the percentile func performance (#34373) (#34416)"
This reverts commit 509ae425e416b4779ae94eab9c2b21f9850e03c3.
2024-05-07 07:23:48 +08:00
2d4da7d177 [fix](kerberos)enable hadoop auto renew tgt (#34439) 2024-05-07 00:36:20 +08:00
e19d57261c [improvement](spill) improve cancel (#34451)
* [improvement](spill) improve cancel

* fix
2024-05-07 00:07:20 +08:00
a81beb19c2 [fix](load) fix repeatedly open tablets_channel when tablets_channel already cancelled (#34442) 2024-05-06 23:15:33 +08:00
f7900b53ce [enhancement](function) floor/ceil/round/round_bankers can use column as scale argument (#34391) 2024-05-06 22:18:36 +08:00
c22f42121b [fix](compaction test) show single replica compaction status and fix test (#33076) (#34285) (#34438) 2024-05-06 21:00:34 +08:00
b7b843d944 [fix](load) acquire latest token instead of oldest token in TokenManager (#34424)
* [fix](load) acquire latest token instead of oldest token

* fixup
2024-05-06 20:19:36 +08:00
aa156f0781 [opt](memory) BE memory info compatible with Cgroup (#34262) 2024-05-06 20:11:20 +08:00
11ca738261 [fix](memory) Fix thread context init in MacOS and not use memory tracker (#34125) 2024-05-06 20:11:20 +08:00
509ae425e4 [Opt](func) opt the percentile func performance (#34373) (#34416) 2024-05-06 20:10:35 +08:00
3cb0deae9c [opt](ranger) modify and enhance the feature of ranger access controller (#34392) (#34426)
bp #34392
2024-05-06 17:08:47 +08:00
ab5ee81811 [fix](memory) Fix page cache memory tracker consumption in prune (#34320) 2024-05-06 12:53:11 +08:00
7ae5de316b [feature](Nereids) support set and use statement syntax only (#33979) (#34409)
pick from master #33979
commit id 65fb7d43b7e838c48502d4e8a69e2541dc73aa88

This PR:
1. add a new Command type: UnsupportedCommand to handle the statement only support parse but could not execute.
2. support syntax about set and use
3. add keyword VAULT to follow legacy planner

TODO
1. support all statment syntax in Nereids
2024-05-06 11:36:01 +08:00
d4c2f70673 [fix](regression_test) failed in distinct_streaming_agg (#34404) 2024-05-06 10:11:39 +08:00
85ae773996 [fix](spill) incorrect revocable mem size of hash join (#34379) 2024-05-06 06:53:12 +08:00
7248420cfd [chore](session_variable) Add 'data_queue_max_blocks' to prevent the DataQueue from occupying too much memory. (#34017) (#34395) 2024-05-05 21:20:33 +08:00
c3096cabe2 [Fix](executor)normal group not auth #34377 2024-05-02 15:17:19 +08:00
Pxl
0d106fe4c2 [Bug](runtime-filter) release rf count dependency when query canceled (#34367)
* release rf count dependency when query canceled

* update

* update
2024-05-02 09:56:17 +08:00
8da260ee0d [fix](hdfs)read 'fs.defaultFS' from core-site.xml for hdfs load which has no default fs (#34217) (#34372)
bp #34217
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-05-01 00:31:49 +08:00
581e168ee1 [Fix](executor)Fix normal workload group alter may failed #34356 2024-04-30 22:17:40 +08:00
5fc1f11cf1 [improvement](hive)add the queryid to the temporary file path (#34278) (#34368)
bp #34278

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-04-30 22:00:05 +08:00
8abd136ba2 [Improvement](executor)Refactor Workload group memory GC (#33797)
* just gc group's overcommit query when minor gc

* add process usage
2024-04-30 19:34:31 +08:00
50855f01c7 [fix](nereids) when runtimefilter target is null, skip the rf #34358 2024-04-30 18:48:50 +08:00
35f8563a75 [feature](iceberg) support iceberg equality delete (#34223) (#34327)
bp #34223

Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
2024-04-30 11:51:29 +08:00
7d77fd0286 [fix](profile) Fix reporting the profile while building the pipeline profile. (#34215) (#34326) 2024-04-30 11:38:03 +08:00
843c89f109 [fix] fix nullptr when clear cache due to move (#34323) 2024-04-30 09:52:41 +08:00
6f873c5907 [improvement](join) Avoid merging blocks more than once on the build side (#34291) 2024-04-30 08:37:53 +08:00
53c06ad9d2 [fix](spill) handel canceled status in spill (#34268) 2024-04-30 08:35:52 +08:00
75470ede1a [fix](test) Fix some testcases #34203 2024-04-30 08:35:03 +08:00
b15fc2a906 [Cherry-pick](branch-2.1) Pick #34043 and #34112 (#34318)
* [Enhancement](full compaction) Add run status support for full compaction (#34043)

* The usage is `curl http://{ip}:{host}/api/compaction/run_status?tablet_id={tablet_id}`
e.g. `curl http://127.0.0.1:8040/api/compaction/run_status?tablet_id=10084`

If full compaction is running, the output will be
```
{
"status" : "Success",
"run_status" : true,
"msg" : "compaction task for this tablet is running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```
else the ouput will be
```
{
"status" : "Success",
"run_status" : false,
"msg" : "compaction task for this tablet is not running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```

* 2

* 2

* [Fix](partial update) Fix rowset not found error when doing partial update (#34112)

Cause: In the logic of partial column updates, the existing data columns are read first, and then the data is supplemented and written back. During the reading process, initialization involves initially fetching rowset IDs, and the actual rowset object is fetched only when needed later. However, between fetching the rowset IDs and the rowset object, compaction may occur, turning the old rowset into a stale rowset. If too much time passes, the stale rowset might be directly deleted. Thus, when the rowset object is needed for an update, it cannot be found. Although the update operation with partial column logic should be able to read all keys and should not encounter new keys, if the rowset disappears, the Backend (BE) will consider these keys as missing. Consequently, it will check whether other columns have default values or are nullable. If this check fails, the aforementioned error is thrown.

Solution: To avoid such issues during partial column updates, the initialization step should involve fetching both the rowset IDs and the shared pointer to the rowset object simultaneously. This ensures that the rowset can always be found during data retrieval.
2024-04-30 07:26:23 +08:00
d1df0b8878 [fix](mtmv)Solving the problem of calling each other in toString() loops (#34277) (#34317)
bp #34277
2024-04-29 21:46:29 +08:00
f76bf66903 [chore](third-party) Fix compilation errors on arm platform (#34261) 2024-04-29 21:26:56 +08:00
996222c30e [fix](test) let test_ntile_function happy with Nereids (#34294) 2024-04-29 20:59:48 +08:00
a173513e27 [fix](pipelinex) exchange sink not set ready when source limit #34241 2024-04-29 20:58:50 +08:00
3495ed58e0 [Enhancement](jdbc catalog) Change Jdbc connection pool to hikari (#34045) (#34310) 2024-04-29 20:22:48 +08:00
7cb00a8e54 [Feature](hive-writer) Implements s3 file committer. (#34307)
Backport #33937.
2024-04-29 19:56:49 +08:00
1bfe0f0393 [feature](iceberg)support read iceberg complex type,iceberg.orc format and position delete. (#33935) (#34256)
master #33935
2024-04-29 14:40:12 +08:00
9b7e007ef6 [Bug](union) fix union operator set eos is not incorrect (#34250)
* [test](case) fix unstable case without order by distinct row

* [Bug](union) fix union operator set eos is not incorrect
2024-04-29 13:38:03 +08:00
20bd0c2987 [FIX](cases )fix ipv6 value for regress case 2024-04-29 13:37:29 +08:00
222289697d [improve](regression) Support qt_target_sql (#34236) (#34270) 2024-04-29 11:50:35 +08:00
5277a55791 (pick 34003) release fd for shutdown tablets (#34224) 2024-04-29 10:51:19 +08:00
946d28646a [fix](outfile)Fixed orcOutputStream.close() throwing an exception during destruction causing the program to hang. (#34254)
bp #34243
2024-04-28 19:54:34 +08:00
417431fd83 [Enhancement](hdfs-file-system) Change fs_handler ptr to shared_ptr and remove ref count operations. (#34049)
Backport #33959.
2024-04-28 19:45:30 +08:00