Commit Graph

18263 Commits

Author SHA1 Message Date
e9b67bc82d [bugfix](paimon)merge meta-inf/services for paimon FileIOLoader (#33166)
We introduced paimon's oss and s3 packages, but did not register them in meta-info/service. As a result, when be used the s3  or oss interface, an error was reported and the class could not be found(`Could not find a file io implementation for scheme 's3' in the classpath.`).

FYI:
https://stackoverflow.com/questions/47310215/merging-meta-inf-services-files-with-maven-assembly-plugin
https://stackoverflow.com/questions/1607220/how-can-i-merge-resource-files-in-a-maven-assembly
2024-04-07 22:13:00 +08:00
d9d950d98e [fix](iceberg) fix iceberg predicate conversion bug (#33283)
Followup #32923

Some cases are not covered in #32923
2024-04-07 22:12:38 +08:00
190763e301 [bugfix](iceberg)Convert the datetime type in the predicate according to the target column (#32923)
Convert the datetime type in the predicate according to the target column.
And add a testcase for #32194
related #30478 #30162
2024-04-07 22:12:33 +08:00
ecb4372479 [Fix](pipelinex) Fix MaxScannerThreadNum calculation error in file scan operator when turn on pipelinex. (#33037)
MaxScannerThreadNum in file scan operator when turn on pipelinex is incorrect, it will cost many memory and causing performance degradation. This PR fix it.
2024-04-07 22:11:27 +08:00
32d6a4fdd5 [opt](rowcount) refresh external table's rowcount async (#32997)
In previous implementation, the row count cache will be expired after 10min(by default),
and after expiration, the next row count request will miss the cache, causing unstable query plan.

In this PR, the cache will be refreshed after Config.external_cache_expire_time_minutes_after_access,
so that the cache entry will remain fresh.
2024-04-07 22:11:14 +08:00
ebf45bff20 [fix](variables) change column type of @@autocommit to BIGINT (#33282)
Some of mysql connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection.
eg, `select @@autocommit` should with column type `BIGINT`, not `BIT`, otherwise it will throw error like:

```
System.FormatException: The input string 'True' was not in a correct format.
   at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
   at System.Convert.ToInt32(String value)
   at MySql.Data.MySqlClient.Driver.LoadCharacterSetsAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken)
   at MySql.Data.MySqlClient.Driver.ConfigureAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken)
   at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Boolean execAsync, CancellationToken cancellationToken)
   at MySql.Data.MySqlClient.MySqlConnection.Open()
```

In this PR, I add a new field of `VarAttr`: `convertBoolToLongMethod`, if set, it will convert boolean to long.
And set it for `autocommit`
2024-04-07 22:02:28 +08:00
59b8bf24b1 [chore](license) fix incomplete license header (#33306)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-04-07 15:00:14 +08:00
92d7333810 [Fix](point query) avoid nullptr in _block_pool (#33120)
`resize` will make nullptrs in _block_pool if _block_pool.size() < s_preallocted_blocks_num
2024-04-07 13:02:37 +08:00
132dbeda7f [BugFix](Iceberg Catalog) Fix iceberg catalog of hms and hadoop not support iceberg properties (#33113)
* fix iceberg catalog of  hms and hadoop not support iceberg properties

* remove unused import
2024-04-07 13:01:24 +08:00
62699c8eea [improve](function) the offset params in lead/lag function could use 0 (#33174) 2024-04-07 12:58:03 +08:00
77349ca71a [pipelineX](fix) Fix coredump by incorrect cancel order (#33294) 2024-04-07 12:06:12 +08:00
0d0cb6d8a4 [fix](nereids)SimplifyRange didn't process NULL value correctly (#33296) 2024-04-07 11:02:32 +08:00
950ca68fac [fix](move-memtable) fix timeout to get tablet schema (#33256) (#33260) 2024-04-04 21:45:55 +08:00
df8e397dd8 [Fix](executor)Fix normal group can not be appended when image exits #33197 2024-04-03 20:37:12 +08:00
df197c6a14 [fix](move-memtable) fix initial use count of streams for auto partition (#33165) (#33236)
Co-authored-by: Kaijie Chen <ckj@apache.org>
2024-04-03 20:31:29 +08:00
Pxl
05a84bd485 [Bug](runtime-filter) set need_local_merge to false when rf is broadcast (#33211)
set need_local_merge to false when rf is broadcast
2024-04-03 19:14:09 +08:00
Pxl
113bada7ed [Chore](runtime-filter) add check is broadcast on nlj (#33088)
add check is broadcast on nlj
2024-04-03 19:14:05 +08:00
797b8fa456 [FIX](agg) fix vertical_compaction_reader for agg table with array/map type (#33130) 2024-04-03 18:09:45 +08:00
fff5c85a71 [bugfix](stop) should skip the loop when graceful stop (#33212)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-04-03 17:10:32 +08:00
7675383c40 [bugfix](deadlock) fix dead lock in cancel fragment (#33181)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-04-03 13:41:24 +08:00
4d18fc1e4c [profile](name) add table rollup name in profile (#33137) 2024-04-02 22:36:30 +08:00
Pxl
34f5521643 [Bug](min-max) store string data in MinMaxNumFunc to avoid use after free when cancel (#33152)
* store string data in MinMaxNumFunc to avoid use after free when cancel

* update
2024-04-02 22:35:58 +08:00
6150c54df5 [bugfix](asyncwriter) async writer's lock should not include finish or close method (#33077)
close or finish method will take a lot of time, and the lock will hold a lot of time. If there is a bug in close or finish method, it will affect pipeline execute thread.
writer's close method will need this lock, so that it will hang when close method is called.
2024-04-02 14:23:00 +08:00
1be38e798d push topn-filter to both sides of inner join (#33112) 2024-04-01 22:46:28 +08:00
7ace3ff6de [branch-2.1](pick) pick 2 prs about nereids create variant table (#33125)
* [fix](Nereids) support variant column with index when create table (#32948)

* [opt](Nereids) support create table with variant type (#32953)

---------

Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
2024-04-01 19:00:39 +08:00
bec153b369 [fix](timeout) query timeout was not correctly set #33045 2024-03-30 22:33:03 +08:00
9f2520537f 2.1.1-rc05 2024-03-30 21:26:27 +08:00
ed48d321d0 [fix](pipelineX) fix error open in scan (#33068) 2024-03-30 20:05:49 +08:00
2f699e27a6 [log](pipeline)add more log in scan localstate #33062 #33063 2024-03-30 12:22:50 +08:00
425c00a0d1 [fix](agg) incorrect result with having conjuncts and limit (#33040) 2024-03-30 10:14:44 +08:00
db3179edaf [pipelineX](local exchange) Fix potential timeout problem (#33022) 2024-03-29 22:24:23 +08:00
f72befe05e [fix](path-gc) Fix pending rowset guard check failure when ordered data compaction failed (#33029) 2024-03-29 17:47:51 +08:00
0c13977ee5 [Fix](segment compaction) _check_and_set_is_doing_segcompaction should be the last condition (#33043)
introduced by #33001
2024-03-29 17:35:30 +08:00
9d6fb39573 [regression-test](Variant) add order by to make test stable (#33014) (#33039) 2024-03-29 17:25:26 +08:00
5d576b41d7 [opt](invert index) use lowercase by default #32405 (#32940) 2024-03-29 14:37:40 +08:00
3a196c8b0f [Pick](Variant) pick 2 prs about bugfix of variant (#33011)
* [Fix](Variant) forbit table with variant type doing segment compaction temporarily

TODO fix this corretly in later works

* [Bug](Variant) use lower case name for variant's root, since backend treat parent column as lower case

This PR address the problem as blow:
```
errCode = 2, detailMessage = (172.16.56.137)[CANCELLED]failed to initialize storage reader. tablet=17136, res=[INTERNAL_ERROR]Not found field_name, field_name:Tags.tag_key1, schema:[Thread(8), Tags(9), Source(5), tags.tag_key1(-1), Title(6), Level(3), Time(2), CreateDate(1), Message(7), IP(4), AppId(0)]

```
2024-03-29 11:12:28 +08:00
67856d9c22 change to 2.1.1 rc04 2024-03-29 10:52:03 +08:00
6600e92b12 [scan](status) Finish execution if scanner failed (#32966) 2024-03-29 10:51:15 +08:00
bea05da638 [shuffle](fix) Do not use copy assignment for TUniqueId (#32969) 2024-03-29 10:37:26 +08:00
fc9d5de56d [case](rowpolicy)fix row policy has been exist (#32880) (#32990) 2024-03-28 22:06:16 +08:00
827d2c46de [fix](nereids) planner should always gennerate bitmap runtime filter #32960 2024-03-28 15:36:49 +08:00
2edf9ef301 [fix](nereids) do not push '<=>' rf through nlj outer join (#32952)
f
2024-03-28 13:54:52 +08:00
9fa6c3f2a7 [fix](auth)show grants error (#32951)
Inconsistent number of columns in title and info
2024-03-28 13:05:29 +08:00
3620a93da8 2.1.1-rc03 2024-03-28 00:16:50 +08:00
3e6d0fa35b [branch-2.1](auditlog) remove auditlog build command in build.sh 2024-03-28 00:07:09 +08:00
6ad63bb51b 2.1.1-rc02 2024-03-27 21:46:40 +08:00
5e35410e6e null-safe-eq-rf denies outer join (#32930) 2024-03-27 21:31:56 +08:00
36d8c73408 [Bug](Variant) forbit variant column with filter, group by or order by (#32928) 2024-03-27 21:18:24 +08:00
9c6180d9ba [revert](jni) revert part of #32455 #32904 2024-03-27 20:45:44 +08:00
71e16e6f35 [fix](iceberg) fix iceberg catalog bug and p2 test cases (#32898)
1. Fix iceberg catalog bug

    This PR #30198 change the logic of `IcebergHMSExternalCatalog.java`,
    to get locationUrl by calling hive metastore's `getCatalog()` method.
    But this method only exists in hive 3+. So it will fail if we using hive 2.x.

    I temporary remove this logic, because this logic is only used from iceberg table writing.
    Which is still under development. We will rethink this logic later.

2. Fix test cases

    Some of P2 test cases missed `order_qt`. And because the output format of the floating point
    type is changed, some result in `out` files need to be regenerated.
2024-03-27 20:44:38 +08:00