Commit Graph

7181 Commits

Author SHA1 Message Date
5f125bbaaa [improvement](binlog)Support inverted index in CCR (#31743) (#32101) 2024-03-12 15:34:08 +08:00
61928f7df5 [pipelineX](scanner) Use the actual instances num when ignore data distribution (#32081) 2024-03-12 14:20:39 +08:00
ffa904c487 [enhance](Cooldown) Skip cooldown if the tablet is dropped (#32079) 2024-03-12 14:20:39 +08:00
b41b17ad0a [fix](spill) fix storage engine type cast error (#32071) 2024-03-12 14:20:18 +08:00
aea9ddc3cb [Fix](Outfiel) fix be core when the open method of vfile_result_writer failed #32042 2024-03-12 14:20:18 +08:00
4268634115 [fix](memory) Fix Allocator cancel pipelinex query #32048 2024-03-12 14:20:18 +08:00
ccd21a6ea4 [Improve](InPredict) enhance in predict with array type (#31828) 2024-03-12 14:19:14 +08:00
68a5319da3 [fix](pipelineX) _local_channel_dependency is null in non pipelineX (#32054) 2024-03-12 14:19:04 +08:00
fe3988c686 [fix](compile) fix BE compile failure on Mac (#32044) 2024-03-12 14:17:55 +08:00
c0f2d0188b [feature](pipelineX) add mem control in local exchange sink (#31982) 2024-03-12 14:17:48 +08:00
b0b7161ad0 [feature](rf) add filter info profile when rf run as expr (#31822) 2024-03-12 14:17:48 +08:00
2470634859 [RuntimeFilter] fix <=> runtime filter failed bug (#32003) 2024-03-12 14:13:13 +08:00
3358f76a7f [feature](spill) Implement spill to disk for hash join, aggregation and sort for pipelineX (#31910)
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
2024-03-12 14:12:09 +08:00
ae6003504c [pipelineX](partition sort) Add some nessacery metrics (#32020)
Add some necessary metrics
2024-03-09 19:55:48 +08:00
c5390d00bb [Improvement]Add schema table backend_active_tasks (#31945) 2024-03-09 19:55:48 +08:00
Pxl
38caa35a73 [Chore](third-party) update clucene version (#32014)
update clucene version
2024-03-09 19:55:48 +08:00
d2e7a68d11 [enhancement](util) print if using nereids planner when be coredump (#31981) 2024-03-09 19:55:47 +08:00
621803c547 [FIX](InPredict) fix in params in to context for thread fragment query (#31935) 2024-03-09 19:45:46 +08:00
6ef4ab631d [Opt](func) reduce the useless mem alloc and const opt the concat code (#31983) 2024-03-09 19:45:46 +08:00
609761567c [Fix](partial-update) Fix wrong column number passing to BE when partial and enable nereids (#31461)
* Problem:
Inconsistent behavior occurs when executing partial column update `UPDATE` statements and `INSERT` statements on merge-on-write tables with the Nereids optimizer enabled. The number of columns passed to BE differs; `UPDATE` operations incorrectly pass all columns, while `INSERT` operations correctly pass only the updated columns.

Reason:
The Nereids optimizer does not handle partial column update `UPDATE` statements properly. The processing logic for `UPDATE` statements rewrites them as equivalent `INSERT` statements, which are then processed according to the logic of `INSERT` statements. For example, assuming a MoW table structure with columns k1, k2, v1, v2, the correct rewrite should be:
* `UPDATE` table t1 set v1 = v1 + 1 where k1 = 1 and k2 = 2
 * =>
 * `INSERT` into table (v1) select v1 + 1 from table t1 where k1 = 1 and k2 = 2

However, the actual rewriting process does not consider the logic for partial column updates, leading to all columns being included in the `INSERT` statement, i.e., the result is:
* `INSERT` into table (k1, k2, v1, v2) select k1, k2, v1 + 1, v2 from table t1 where k1 = 1 and k2 = 2

This results in `UPDATE` operations incorrectly passing all columns to BE.

Solution:
Having analyzed the cause, the solution is straightforward: when rewriting partial column update `UPDATE` statements to `INSERT` statements, only retain the updated columns and all key columns (as partial column updates must include all key columns). Additionally, this PR includes error injection cases to verify the number of columns passed to BE is correct.

* 2

* 3

* 4

* 5
2024-03-09 19:45:42 +08:00
e8aa5ee7d5 [Improve](Variant) support bloom filter for variant subcolumns (#31347)
* [Improve](Variant) support bloom filter for variant subcolumns

* rebase
2024-03-09 19:45:03 +08:00
f968d96545 [profile](pipelineX) Add lost metrics (#31964) 2024-03-09 19:45:03 +08:00
93d298d34a [fix](agg) wrong result of two or more map_agg functions in query (#31928) 2024-03-09 19:45:03 +08:00
b2de83f250 [agg](conf) Add a knob to control distinct agg (#31930)
Add a knob to control distinct agg
2024-03-09 19:44:54 +08:00
0da010603e [Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)
Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse
2024-03-09 19:44:42 +08:00
779ca464a5 [Fix](Status) Handle returned overall Status correctly (#31692)
Handle returned overall Status correctly
2024-03-09 19:44:39 +08:00
Pxl
981ea73466 [Bug](top-n) init query_ctx runtime predicate before _build_pipelines (#31896)
init query_ctx runtime predicate before _build_pipelines
2024-03-09 19:43:21 +08:00
21ce85dc14 [fix](money_format) fix money_format #31883 2024-03-07 16:53:19 +08:00
da5a40077f [fix](http stream) http stream support memtable_on_sink_node header (#31866) 2024-03-07 16:53:19 +08:00
9bf22a872a [Bug](fix) fix or and "<=>" cause coredump in query (#31884) 2024-03-07 16:53:19 +08:00
ff8c679851 [fix](group commit) group commit may heap-use-after-free if execute plan failed (#31839) 2024-03-07 16:12:23 +08:00
28f0b7eb32 [Improvement](profile)Add tvf active_be_tasks() #31815 2024-03-07 16:12:23 +08:00
538032a75c [fix](partition) add log when tablet partition id eq 0 (#31796) 2024-03-07 16:11:25 +08:00
2e94512e07 [fix](build)Fix index_tool build error (#31834)
Co-authored-by: Luennng <luennng@gmail.com>
2024-03-07 16:11:25 +08:00
Pxl
2913a111f6 [Bug](top-n) init query_ctx runtime predicate before _build_pipelines #31895 2024-03-07 10:19:42 +08:00
Pxl
b4dbb087c0 [Bug](top-n) init query_ctx runtime predicate before operators prepare #31876 2024-03-06 19:09:10 +08:00
1dcb99519d [fix](move-memtable) use tuple desc from schema param for delta writer (#31747) 2024-03-06 13:08:30 +08:00
4f174c4fb9 [feature](function) Support for aggregate function foreach combiner (#31526) 2024-03-06 13:08:30 +08:00
9af64d848f [fix](pipelineX) fix error distribution in DistinctStreamingAggOperatorX (#31804) 2024-03-06 13:08:30 +08:00
c43bc8349f [fix](move-memtable) free resources when fail to open stream (#31762) 2024-03-06 13:07:59 +08:00
eea9b56f69 [fix](group commit) handle group commit create plan error (#31757) 2024-03-06 13:07:59 +08:00
808563470f [pipelineX](debug) Refactor code and complete debug string (#31733) 2024-03-06 13:07:49 +08:00
7c30cb20fd [Fix](partial update) Fix partial update load false when schema includes auto increment column (#31725)
Problem:
When partially updating columns without specifying the auto-increment column, and the imported data contains new keys, an error stating the auto-increment column could not be found occurs.

Reason:
The logic for partial column updates does not account for new keys in auto-increment columns. Since auto-increment columns can be generated by the system, it's possible to omit this column data during import. However, partial column updates treat this as a regular column, expecting it to be nullable or have a default value for automatic filling, overlooking the fact that auto-increment columns can also be auto-filled. This oversight leads to the error.

Solution:
Incorporate a check for auto-increment columns into the partial column update logic, and include the logic for generating auto-increment column values in the process of completing partial updates.
2024-03-06 13:06:27 +08:00
cca0773261 [feature](function) round function defaults to rounding normally 2024-03-06 13:06:26 +08:00
231768db0d [Performance](exec) Support runtime filter in <=> join (#31754) 2024-03-06 13:06:26 +08:00
Pxl
25d1934289 [Feature](topn) support multiple topn filter on backend (#31665)
support multiple topn filter on backend
2024-03-06 13:05:22 +08:00
3777ffb43f [enhancement](nereids)support null partition for list partition (#31613) 2024-03-06 13:05:22 +08:00
91efb6a43d Revert "[fix](meta-tool) Fix compile meta tool (#31457)"
This reverts commit 60de835d48536977e25c1841f4eed6abb0f2a311.
2024-03-04 21:49:07 +08:00
65d45daf8a [Bug](coredump) fix regresstion test coredump in multi thread access map (#31664) 2024-03-03 19:30:55 +08:00
4386a71709 [bugfix](wgcore) map at only get reference and it will core in multithread
f
2024-03-03 15:54:26 +08:00