database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,668 Commits 1 Branch 0 Tags
ff6fa33021896f7ea577ed0dfdda3fed20bda89c
Commit Graph

5093 Commits

Author SHA1 Message Date
zzzxl
ff6fa33021 [opt](inverted index) mow supports index optimization #(#38180) 
## Proposed changes

https://github.com/apache/doris/pull/37428
https://github.com/apache/doris/pull/37429

<!--Describe your changes.-->
 2024-08-06 11:18:13 +08:00
feiniaofeiafei
ab3057b2d4 [Feat](nereids) support date function in partition prune (#38743) (#38898) 
cherry-pick #38743 to branch-2.1
 2024-08-06 09:13:13 +08:00
zzzxl
bcea54147c [feature](inverted index) String type inverted index match function c… (#38872) 
https://github.com/apache/doris/pull/38170
 2024-08-06 09:06:05 +08:00
zhiqiang
c7b59b38ef [fix](hist) Fix unstable result of aggregrate function hist #38608 (#38893) 
cherry pick from #38608
 2024-08-06 08:52:03 +08:00
feiniaofeiafei
0f0b0e9b37 [Feat](nereids) Support date_trunc function in partition prune (#38025) (#38849) 
cherry-pick #38025 to branch-2.1
 2024-08-05 18:29:10 +08:00
starocean999
40567b5d69 [fix](nereids)support group_concat with distinct and order by (#38871) 
## Proposed changes

pick from master https://github.com/apache/doris/pull/38080

<!--Describe your changes.-->
 2024-08-05 18:23:55 +08:00
meiyi
7d4ff34d1f [fix](regression) fix test_primary_key_simple_case (#38798) (#38844) 
pick https://github.com/apache/doris/pull/38798
 2024-08-05 16:32:41 +08:00
Uniqueyou
65154f8abe [branch-2.1] (doris-future) Support auto partition name function (#38853) 
cherry-pick https://github.com/apache/doris/pull/34258 to branch-2.1
 2024-08-05 16:04:24 +08:00
Kang
808397e0d2 [fix](testcase) add order by to fix unstable output of passwordLeaked #38813 (#38855) 
cherry pick from #38813
 2024-08-05 13:51:54 +08:00
meiyi
aaee1d9bbd [fix](regression) fix prepare_insert when execute prepare stmt in observer fe (#38545) (#38850) 
pick https://github.com/apache/doris/pull/38545
 2024-08-05 13:45:13 +08:00
feiniaofeiafei
de9b9d6a39 [Fix](nereids) change char(0) to char(1), varchar(0) to varchar(65533) when create table (#38427) (#38530) 
cherry-pick #38427 to branch-2.1

---------

Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
 2024-08-05 09:18:18 +08:00
daidai
607c0b82a9 [opt](serde)Optimize the filling of fixed values ​​into block columns without repeated deserialization. (#37377) (#38245) (#38810) 
## Proposed changes
pick pr: #38575  and fix this pr bug :  #38245
 2024-08-05 09:13:08 +08:00
amory
2653087843 [pick](array-funcs)fix array with empty arg in be behavior (#38708) 
## Proposed changes
backport: https://github.com/apache/doris/pull/36845
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-05 09:08:28 +08:00
zhangstar333
1b3d4b4d31 [cherry-pick](branch-21)fix operator do_projections should use local_state intermediate_projections (#38612) (#38765) 
## Proposed changes

cherry-pick from master https://github.com/apache/doris/pull/38612

<!--Describe your changes.-->
 2024-08-05 09:07:16 +08:00
daidai
5d02c48715 [feature](hive)Support reading renamed Parquet Hive and Orc Hive tables. (#38432) (#38809) 
bp #38432 

## Proposed changes
Add `hive_parquet_use_column_names` and `hive_orc_use_column_names`
session variables to read the table after rename column in `Hive`.

These two session variables are referenced from
`parquet_use_column_names` and `orc_use_column_names` of `Trino` hive
connector.

By default, these two session variables are true. When they are set to
false, reading orc/parquet will access the columns according to the
ordinal position in the Hive table definition.

For example:
```mysql
in Hive :
hive> create table tmp (a int , b string) stored as parquet;
hive> insert into table tmp values(1,"2");
hive> alter table tmp  change column  a new_a int;
hive> insert into table tmp values(2,"4");

in Doris :
mysql> set hive_parquet_use_column_names=true;
Query OK, 0 rows affected (0.00 sec)

mysql> select  * from tmp;
+-------+------+
| new_a | b    |
+-------+------+
|  NULL | 2    |
|     2 | 4    |
+-------+------+
2 rows in set (0.02 sec)

mysql> set hive_parquet_use_column_names=false;
Query OK, 0 rows affected (0.00 sec)

mysql> select  * from tmp;
+-------+------+
| new_a | b    |
+-------+------+
|     1 | 2    |
|     2 | 4    |
+-------+------+
2 rows in set (0.02 sec)
```

You can use `set
parquet.column.index.access/orc.force.positional.evolution = true/false`
in hive 3 to control the results of reading the table like these two
session variables. However, for the rename struct inside column parquet
table, the effects of hive and doris are different.
 2024-08-05 09:06:49 +08:00
hui lai
f76397277e [fix](routine load) fix show routine load task result incorrect (#38523) (#38826) 
pick (#38523)

Create a job:
```
CREATE ROUTINE LOAD testShow ON test_show_routine_load
COLUMNS TERMINATED BY ","
PROPERTIES
(
"max_batch_interval" = "5",
"max_batch_rows" = "300000",
"max_batch_size" = "209715200"
)
FROM KAFKA
(
"kafka_broker_list" = "127.0.0.1:19092",
"kafka_topic" = "test_show_routine_load",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
```
show routine load task:
```
SHOW ROUTINE LOAD TASK WHERE JobName = "testShow";
```
result:
```
ERROR 1105 (HY000): errCode = 2, detailMessage = The job named testshowdoes not exists or job state is stopped or cancelled
```

Do not  use `toLowerCase` method;
 2024-08-04 22:18:25 +08:00
hui lai
79b07d0b8a [fix](routine load) fix enclose and escape can not set in routine load job (#38402) (#38825) 
pick (#38402)
 2024-08-04 22:17:12 +08:00
hui lai
8e4fad99a1 [test](routine load) add routine load case with timestamp as offset(#38567) (#38822) 
pick (#38567)
 2024-08-04 22:05:19 +08:00
hui lai
eef8c87fb5 [chore](test) disable fault injection to make pipeline task check happy (#38665) (#38821) 
pick (#38665)

test_delta_writer_v2_back_pressure_fault_injection would make pipeline
task can not finish, disable it temporarily to make pipeline task check
happy.
 2024-08-04 11:18:56 +08:00
Luwei
0603ec1d9d [enhancement](compaction) optimizing memory usage for compaction (#37099) (#37486) 2024-08-04 10:49:18 +08:00
HappenLee
7bdc508ac7 [Bug](fix) fix coredump case in (not null, null) execpt (not null, not null) case (#38756) 
## Proposed changes

Issue Number: close #38612

<!--Describe your changes.-->
 2024-08-04 10:44:10 +08:00
qiye
fe3e3d0fab [fix](test)Fix build index fault test (#38736) (#38762) 
## Proposed changes

backport #38736
 2024-08-03 23:48:29 +08:00
qiye
74908c123a [fix](test)Fix unstable test drop index fault #38768 (#38772) 
## Proposed changes

bp #38768

<!--Describe your changes.-->
 2024-08-03 23:47:55 +08:00
bobhan1
64b69ed1ba [branch-2.1] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" (#38682) 
## Proposed changes

picks https://github.com/apache/doris/pull/38487
 2024-08-02 20:05:31 +08:00
amory
556f0fc784 [pick](json-keys) support json_keys function (#38631) 
## Proposed changes
backport: https://github.com/apache/doris/pull/36411
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 19:10:00 +08:00
morrySnow
f24d55fc94 [fix](syntax) multi statements must delim with semicolon (#38670) (#38753) 
pick from master #38670
 2024-08-02 14:49:51 +08:00
Mingyu Chen
da7b2cf578 [refactor](catalog) set "use_meta_cache" default to true (#38244)(#38352)(#38619) (#38355) 
bp #38244 #38352 #38619

---------

Co-authored-by: Yulei-Yang <yulei.yang0699@gmail.com>
 2024-08-02 14:13:38 +08:00
amory
9b07cd2069 [pick](json-serde)pick jsonb string deserialize with spec char (#38711) 
## Proposed changes
backport: https://github.com/apache/doris/pull/37176
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 13:37:41 +08:00
amory
7bcda89881 [pick](case) fix one_nested_types cases (#38723) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38410
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 12:07:14 +08:00
qiye
f21d7e3833 [test](inverted index)Add cases for inverted index format v2 (#38132)(#38443) (#38222) 
## Proposed changes

backport #38132 #38443
 2024-08-02 12:04:26 +08:00
amory
1d982ada45 [pick](array-funcs)pick array func array_enumerate_uniq bugfix (#38721) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38384
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:25:17 +08:00
amory
f5bc65989c [pick](array-range)improve array_range func for large param (#38707) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38284
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:22:46 +08:00
amory
84d9b2fcf4 [pick](nestedtypes) support nested type with agg replace_if_not_null (#38719) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38304
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:18:33 +08:00
zzzxl
0da388ade5 [fix](inverted index) fix match_phrase_ edge query result error #38327 (#38740) 2024-08-01 23:17:53 +08:00
qiye
4d980b8235 [feature](http action)Add http action to show nested inverted index file (#38272) (#38672) 
backport #38272
 2024-08-01 19:30:59 +08:00
qiye
555dccb1a4 [fix](bloom filter)Fix rename column with bloom filter (#38431) (#38662) 
backport #38431
 2024-08-01 19:01:14 +08:00
qiye
e59a866d6a [fix](test)Fix unstable index compaction fault injection case (#38501) (#38673) 
## Proposed changes

bp #38501
 2024-08-01 18:58:51 +08:00
meiyi
e8690b62ee [fix](group commit) Pick add debug log show why group commit not work; delete wal when replay success (#38611) (#38659) 
Pick https://github.com/apache/doris/pull/38611
 2024-08-01 16:59:54 +08:00
slothever
b0943064e0 [fix](kerberos)fix and refactor ugi login for kerberos and simple authentication (#38607) 
pick from  (#37301)
 2024-08-01 14:01:32 +08:00
HappenLee
28998300d4 [Bug](fix) fix ubsan use int32_t pointer access bool value (#38621) 
## Proposed changes

Issue Number: close #38617

<!--Describe your changes.-->
 2024-08-01 13:52:12 +08:00
wuwenchi
057ee1905f [bugfix](hudi)add timetravel for nereids for 2.1 (#38324) (#38582) 
## Proposed changes

bp #38324
 2024-08-01 11:37:57 +08:00
amory
338fa32303 [pick](simdjson) fix simdjson with object array when jsonroot is not empty (#38633) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38490
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-01 11:04:54 +08:00
wuwenchi
41fa7bc9fd [bugfix](paimon)Fixed the reading of timestamp with time zone type data for 2.1 (#37716) (#38592) 
bp: #37716
 2024-08-01 10:23:06 +08:00
amory
184b8cbbe4 [pick](json)fix jsonb deseriaze (#38630) 
## Proposed changes
backport: https://github.com/apache/doris/pull/37251
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-01 10:18:27 +08:00
amory
6bd93b119f [pick](cast)Feature cast complexttype2 json (#38632) 
## Proposed changes
backport: https://github.com/apache/doris/pull/36548
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-01 09:18:15 +08:00
feiniaofeiafei
4c330e3fc6 [Fix](test) fix pull up literal predicate regression (#38564) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-31 22:59:08 +08:00
Tiewei Fang
c8037ca833 [fix](regression-test) fix test_numbers test (#38600) 
bp: #38050
 2024-07-31 22:55:01 +08:00
Tiewei Fang
b21b906306 [Fix](outfile) FE check the hdfs URI of outfile (#38602) 
bp: #38203

1. Previously, if the root path of the HDFS URI started with two
slashes, the outfile would be successfully exported without errors, but
the exported path would not be the expected path.
Currently, we will delete repeated '/' which specified by users in FE.

2. move the test case for outfile HDFS from p2 to p0.
 2024-07-31 22:46:37 +08:00
wuwenchi
ef8a1918c3 [case][fix](iceberg)move rest cases from p2 to p0 and fix iceberg version issue for 2.1 (#37898) (#38589) 
bp: #37898
 2024-07-31 22:41:56 +08:00
wuwenchi
1c176db010 [bugfix](paimon)add support for 'in' and 'not in' for 2.1 #38390 (#38576) 
## Proposed changes

bp: #38390
 2024-07-31 22:38:27 +08:00