Commit Graph

18429 Commits

Author SHA1 Message Date
5d576b41d7 [opt](invert index) use lowercase by default #32405 (#32940) 2024-03-29 14:37:40 +08:00
3a196c8b0f [Pick](Variant) pick 2 prs about bugfix of variant (#33011)
* [Fix](Variant) forbit table with variant type doing segment compaction temporarily

TODO fix this corretly in later works

* [Bug](Variant) use lower case name for variant's root, since backend treat parent column as lower case

This PR address the problem as blow:
```
errCode = 2, detailMessage = (172.16.56.137)[CANCELLED]failed to initialize storage reader. tablet=17136, res=[INTERNAL_ERROR]Not found field_name, field_name:Tags.tag_key1, schema:[Thread(8), Tags(9), Source(5), tags.tag_key1(-1), Title(6), Level(3), Time(2), CreateDate(1), Message(7), IP(4), AppId(0)]

```
2024-03-29 11:12:28 +08:00
67856d9c22 change to 2.1.1 rc04 2024-03-29 10:52:03 +08:00
6600e92b12 [scan](status) Finish execution if scanner failed (#32966) 2024-03-29 10:51:15 +08:00
bea05da638 [shuffle](fix) Do not use copy assignment for TUniqueId (#32969) 2024-03-29 10:37:26 +08:00
fc9d5de56d [case](rowpolicy)fix row policy has been exist (#32880) (#32990) 2024-03-28 22:06:16 +08:00
827d2c46de [fix](nereids) planner should always gennerate bitmap runtime filter #32960 2024-03-28 15:36:49 +08:00
2edf9ef301 [fix](nereids) do not push '<=>' rf through nlj outer join (#32952)
f
2024-03-28 13:54:52 +08:00
9fa6c3f2a7 [fix](auth)show grants error (#32951)
Inconsistent number of columns in title and info
2024-03-28 13:05:29 +08:00
3620a93da8 2.1.1-rc03 2024-03-28 00:16:50 +08:00
3e6d0fa35b [branch-2.1](auditlog) remove auditlog build command in build.sh 2024-03-28 00:07:09 +08:00
6ad63bb51b 2.1.1-rc02 2024-03-27 21:46:40 +08:00
5e35410e6e null-safe-eq-rf denies outer join (#32930) 2024-03-27 21:31:56 +08:00
36d8c73408 [Bug](Variant) forbit variant column with filter, group by or order by (#32928) 2024-03-27 21:18:24 +08:00
9c6180d9ba [revert](jni) revert part of #32455 #32904 2024-03-27 20:45:44 +08:00
71e16e6f35 [fix](iceberg) fix iceberg catalog bug and p2 test cases (#32898)
1. Fix iceberg catalog bug

    This PR #30198 change the logic of `IcebergHMSExternalCatalog.java`,
    to get locationUrl by calling hive metastore's `getCatalog()` method.
    But this method only exists in hive 3+. So it will fail if we using hive 2.x.

    I temporary remove this logic, because this logic is only used from iceberg table writing.
    Which is still under development. We will rethink this logic later.

2. Fix test cases

    Some of P2 test cases missed `order_qt`. And because the output format of the floating point
    type is changed, some result in `out` files need to be regenerated.
2024-03-27 20:44:38 +08:00
81162fb7f4 [fix](nereids)BE cannot find min-max runtime filter on nested loop join node-pick #32886 #32917 2024-03-27 19:55:43 +08:00
e8d843d75f [Fix](executor)Fix regression test for test_active_queries/test_backend_active_tasks #32899 2024-03-27 19:42:26 +08:00
02d623f2ce [improvement](mtmv) Support more join types for query rewriting by materialized view (#32685) (#32909)
This pattern of rewriting is supported for multi-table joins and supported join types is as following:

INNER JOIN
LEFT OUTER JOIN
RIGHT OUTER JOIN
FULL OUTER JOIN
LEFT SEMI JOIN
RIGHT SEMI JOIN
LEFT ANTI JOIN
RIGHT ANTI JOIN
2024-03-27 19:31:01 +08:00
5d89d86a50 [fix](nereids)EliminateGroupByConstant should replace agg's output wafter removing constant group by keys (#32877) 2024-03-27 11:04:32 +08:00
309de6c82b [fix](smooth-upgrade) Fix function now upgrade compatible #32807 2024-03-27 10:49:34 +08:00
352617a34d [fix](scanner) cached blocks may be empty when VFileScanner return NOT_FOUND (#32745)
Cached blocks may be empty when VFileScanner return NOT_FOUND. This feature is introduced by https://github.com/apache/doris/pull/15226. Move this function inner `VFileScanner`.
2024-03-27 10:01:05 +08:00
0a447273f0 [Bug](pipelineX) fix memleak in stream agg (#32862)
* [Bug](pipelineX) fix memleak in stream agg

* change by review
2024-03-27 09:21:21 +08:00
39b768b9ea [branch-2.1](thirdparty) upgrade arrow to 15.0.2 #32827 2024-03-27 08:35:40 +08:00
45e1f9f4c4 [branch-2.1](cherry-pick)Role comment 2.1 #32837 2024-03-27 08:33:54 +08:00
Pxl
ca64fa7954 [Bug](materialized-view) do not check key/value column when index is dup or mow (#32695)
do not check key/value column when index is dup or mow
2024-03-27 02:56:11 +08:00
ad688171f5 [fix](Nereids): handle distinct and nullable property when rewriting sum literal (#32778)
Refactored the SQL query to handle the distinct and nullable values when computing the sum of 'v' and 'v + 1' in the 't' table like:

select sum(v + 1), sum(distinct v + 1) from t
=>
select sum(v) + count(v), sum(distinct v) + count(distinct v)
2024-03-26 20:36:07 +08:00
Pxl
f579eceb34 [Improvementation](profile) add some profile on vcollect_iterator (#32794)
add some profile on vcollect_iterator
2024-03-26 20:33:16 +08:00
b5a1914740 [Fix](nereids) Fix deletestmt getting catalog (#32701) 2024-03-26 20:29:03 +08:00
820bf7e590 [fix](mtmv) use min value to check the first partition (#32765) 2024-03-26 20:28:52 +08:00
3658dfd500 [enhance](auth)node priv can show proc (#32751) 2024-03-26 20:28:00 +08:00
0122b8a6b4 [Update](inverted index) add config for inverted index query cache shards (#32666) 2024-03-26 20:27:33 +08:00
1b6b92a19d [improvement](mtmv) Support hll function roll up when query rewrite by materialized view (#32431)
Support hll roll up, the hll fucntion supportd is as following:

+-----------------------------------------------------------------------------------------------------------------------------------------------------+
|                      in query                    |                          in materialized view                           |        rolluped        |
+ ------------------------------------------------ + ----------------------------------------------------------------------- + ---------------------- +
| HLL_UNION_AGG(hll column)                        | hll_union(column) or hll_raw_agg(column) as column1                     | HLL_UNION_AGG(column1) |
| HLL_RAW_AGG(hll column) or HLL_UNION(hll column) |                                                                         | HLL_UNION(column)      |
| approx_count_distinct(not hll column)            | hll_union(HLL_HASH(column)) or hll_raw_agg(HLL_HASH(column)) as column1 | HLL_UNION_AGG(column1) |
| HLL_UNION_AGG(HLL_HASH(column))                  |                                                                         | HLL_UNION_AGG(column)  |
| hll_cardinality(hll_union(HLL_HASH(column)))     | hll_union(HLL_HASH(column)) or hll_raw_agg(HLL_HASH(column)) as column1 |                        |
| hll_cardinality(hll_raw_agg(HLL_HASH(column)))   | hll_union(HLL_HASH(column)) or hll_raw_agg(HLL_HASH(column)) as column1 |                        |
| HLL_RAW_AGG(HLL_HASH(column))                    | hll_union(HLL_HASH(column)) or hll_raw_agg(HLL_HASH(column)) as column1 | HLL_RAW_AGG(column1)   |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
2024-03-26 20:26:16 +08:00
0655d49a21 [enhancement](nereids) only push having as agg's parent if having just use slots from agg's output (#32414)
1. only push having as agg's parent if having just use slots from agg's output
2. show user friendly error message when item in select list but not in aggregate node's output
2024-03-26 20:26:04 +08:00
60c3372d8e [fix](Nereids): fix elimiate join by pkfk when there are multi joins (#32703) 2024-03-26 20:24:39 +08:00
2d398dfe1f [fix](planner)the decimal type of's precision and scale setope ration codn is wrong (#32787) 2024-03-26 20:22:29 +08:00
Pxl
fdf6b8fe8d [Bug](repeat) fix core dump coz output slot'order on repeat node not match with pre repeat exprs (#32662)
fix core dump coz output slot'order on repeat node not match with pre repeat exprs
2024-03-26 20:22:20 +08:00
0a44de67bf [bug](distinct agg) fix distinct streaming agg not output all data (#32760)
fix distinct streaming agg not output all data
2024-03-26 20:19:36 +08:00
ad2d20348a [fix](pipeline) fix use error row desc when origin block clear #32803 (#32849)
* fix

* add case
2024-03-26 20:02:46 +08:00
Pxl
bda35f9ff0 [Bug](runtime-filter) fix unknown filter on nested loop join sink #32851 2024-03-26 19:10:31 +08:00
efe684572e [Enhancement](Load) Nereids supports http_stream and group_commit with stream load (#31259) 2024-03-26 17:22:42 +08:00
34cb83fb6e [feature](merge-cloud) Add init CloudEnv and CloudInternalCatalog (#29962) 2024-03-26 17:11:05 +08:00
f9ae03ac3c [feature](Nereids) support data masking policy (#32526)
support data masking policy

note:
if a user send the query
```sql
select name from tbl limit 1
```
and the user have row policy on `tbl.name` with the filter `name = 'Beijing'`, and have data masking policy on `tbl.name` with the masking `concat(substring(name, 1, 4), '****')`, we will rewrite the query to
```sql
select concat(substring(name, 1, 4), '****') as name
from tbl
where name = 'Beijing' -- note that this name is from tbl, not from the alias in the select list
limit 1
```

the result would be `Beij****`
2024-03-26 15:31:08 +08:00
0a2d7379fc [enhance](auth)row policy support catalog and match name instead id (#32310)
Follow up #32137

storage name instead id to meta,Prevent table deletion and reconstruction and causing ID changes
2024-03-26 15:31:08 +08:00
c0d7a5660e [fix](paimon) support paimon with hive2 (#32455)
In order to support paimon with hive2, we need to modify the origin HiveMetastoreClient.java
to let it compatible with both hive2 and hive3.
And this modified HiveMetastoreClient should be at the front of the CLASSPATH, so that
it can overwrite the HiveMetastoreClient in hadoop jar.

This PR mainly changes:

1. Copy HiveMetastoreClient.java in FE to BE's preload jar.

2. Split the origin `preload-extensions-jar-with-dependencies.jar` into 2 jars
    1. `preload-extensions-project.jar`, which contains the modified HiveMetastoreClient.
    2. `preload-extensions-jar-with-dependencies.jar`, which contains other dependency jars.

3. Modify the `start_be.sh`, to let `preload-extensions-project.jar` be loaded first.

4. Change the way the assemble the jni scanner jar
    Only need to assemble the project jar, without other dependencies.
    Because actually we only use classed under `org.apache.doris` package.
    So remove other unused dependency jars can also reduce the output size of BE.

5. fix bug that the prefix of paimon properties should be `paimon.`, not `paimon`

6. Support paimon with hive2
    User can set `hive.version` in paimon catalog properties to specify the hive version.
2024-03-26 15:31:07 +08:00
37c8cc040d [feature-wip](ranger)support datamask and row filter (#32137)
doris ranger support datamask and row filter

hive ranger support row filter
2024-03-26 15:31:07 +08:00
ec43f65235 [feature](hudi) support hudi incremental read (#32052)
* [feature](hudi) support incremental read for hudi table

* fix jdk17 java options
2024-03-26 15:31:07 +08:00
8714dde34f [monir] remove unused cluster code (#31360)
* [monir] remove unused cluster code

* 2
2024-03-26 15:31:07 +08:00
983e5df812 Fix compile (#32818)
[fix](compile) fix code style (#32819)

* Fix compile

* fix style
2024-03-26 15:02:46 +08:00
cf24b571a3 [Pick 2.1](inverted index) fix writer searcher cache fd leak(#32564) #32812 2024-03-26 13:58:36 +08:00