193be20c86
[feature](csv)Supports reading CSV data using LF and CRLF as line separators. ( #37687 ) ( #38099 )
...
bp #37687
2024-07-22 22:53:04 +08:00
bb2b7774df
[feature](iceberg) iceberg write support insert overwrite and optimize hive write transaction statistics and ( #37191 ) ( #38097 )
...
bp #37191
Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com >
Co-authored-by: lik40 <lik40@chinatelecom.cn >
2024-07-19 09:45:41 +08:00
ba66ff5768
[fix](multi-catalog)fix paimon meta properties convert ( #37249 ) ( #37958 )
...
bp #37249
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com >
2024-07-17 01:04:33 +08:00
f7068b5658
[cherry-pick](branch-2.1) Make doris read hive text table parameters and behavior consistent with hive ( #37840 )
...
## Proposed changes
pick from master https://github.com/apache/doris/pull/37638
<!--Describe your changes.-->
2024-07-16 22:24:50 +08:00
9fd4795080
[opt](iceberg)Add a new appearance to display the pushDown count for 2.1 ( #37046 ) ( #34928 ) ( #37810 )
...
## Proposed changes
bp: #37046 #34928
2024-07-16 16:03:44 +08:00
e7a001c420
[enhance](mtmv)support partition tvf ( #37795 )
...
pick from: https://github.com/apache/doris/pull/36479 and
https://github.com/apache/doris/pull/37201
2024-07-16 09:27:44 +08:00
e5339a4014
[feature](ES Catalog)Support control scroll level by config #37180 ( #37290 )
...
## Proposed changes
backport #37180
2024-07-15 16:41:38 +08:00
7bd6818350
[branch-2.1][improvement](jdbc catalog) Added support for Oracle Raw type ( #37776 )
...
pick (#37078 )
In previous versions, we adopted the strategy of reading the object
address for Oracle's raw type, which would lead to unstable and
meaningless results. Here I changed it to read hexadecimal or UTF8
2024-07-15 14:43:05 +08:00
16de141743
[regression](kerberos)add hive kerberos docker regression env ( #37657 )
...
## Proposed changes
pick:
[regression](kerberos)fix regression pipeline env when write hosts
(#37057 )
[regression](kerberos)add hive kerberos docker regression env (#36430 )
2024-07-15 09:35:39 +08:00
56a207c3f0
[case](paimon/iceberg)move cases from p2 to p0 ( #37276 ) ( #37738 )
...
bp #37276
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com >
2024-07-13 10:01:05 +08:00
f2556ba182
[feature](insert)support external hive truncate table DDL ( #37659 )
...
pick: #36801
2024-07-12 22:37:47 +08:00
ca0e44f83f
[fix](case) fix struct format out files ( #37350 ) ( #37499 )
...
bp #37350
2024-07-09 10:11:50 +08:00
f8cee439b6
[feature](ES Catalog) map nested/object type in ES to JSON type in Doris ( #37101 ) ( #37182 )
...
backport #37101
2024-07-05 10:48:32 +08:00
55636e8035
[test](migrate) move 3 cases from p2 to p0 ( #36957 ) ( #37264 )
...
bp #36957
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com >
2024-07-04 20:09:59 +08:00
ceef9ee123
[feature](serde) support presto compatible output format ( #37039 ) ( #37253 )
...
bp #37039
2024-07-04 13:56:05 +08:00
3613413a54
[fix](hive) support find serde info from both tbl properties and serde properties ( #37043 ) ( #37188 )
...
bp #37043
2024-07-04 13:55:38 +08:00
bf3ea1839c
[test]Mv external p2 test case to p0. ( #37070 ) ( #37140 )
...
backport: https://github.com/apache/doris/pull/37070
2024-07-04 11:19:31 +08:00
a9f9113c48
[branch-2.1][test](external)move hive cases from p2 to p0 ( #37149 )
...
pk (#36855 )
test_hive_same_db_table_name
test_hive_special_char_partition
test_complex_types
test_wide_table
2024-07-03 19:44:52 +08:00
e5695e058f
[test](migrate) move 2 cases from p2 to p0 ( #36935 ) ( #37200 )
...
bp #36935
Co-authored-by: zhangdong <493738387@qq.com >
2024-07-03 17:29:01 +08:00
e857680661
[Migrate-Test](multi-catalog) Migrate p2 tests from p2 to p0. ( #37175 )
...
Backport #36989 .
2024-07-03 11:08:49 +08:00
4761090848
[fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF ( #37161 )
...
bp: #35489
2024-07-03 10:58:08 +08:00
e7e1e967cf
[test](migrate) move 2 cases from p2 to p0 for 2.1 ( #37139 )
...
pick #37004
2024-07-02 22:50:53 +08:00
b445c783eb
[test](tvf) move p2 tvf tests from p2 to p0 ( #37081 ) ( #37152 )
...
bp: #37081
2024-07-02 22:38:22 +08:00
74086189d3
[test](tvf) move p2 tvf tests from p2 to p0 ( #36871 ) ( #37150 )
...
bp: #36871
2024-07-02 22:37:43 +08:00
bd24a8bdd9
[Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values ( #37153 )
...
bp: #36668
2024-07-02 22:12:17 +08:00
b63e6ac350
[fix](ES Catalog)Add array types support in esquery function ( #36936 ) ( #37054 )
...
backport #36936
2024-07-02 17:48:35 +08:00
cf86eb8647
[test](migrate) move test_hive_text_complex_type from p2 to p0 ( #37007 ) ( #37123 )
...
bp: #37007
2024-07-02 17:36:37 +08:00
4dcceaefea
[test](ES Catalog) Add test cases for ES 5.x ( #34441 ) ( #36993 )
...
backport #34441
2024-06-28 16:58:07 +08:00
fcc26cc671
[test](migrate) move some cases from p2 to p0 ( #36750 )( #36787 ) ( #36922 )
...
bp #36750 and #36787
2024-06-27 20:59:50 +08:00
5c1eef5f06
[feature](tvf) support max_filter_ratio ( #35431 ) ( #36911 )
...
bp #35431
Co-authored-by: 苏小刚 <suxiaogang223@icloud.com >
2024-06-27 20:58:53 +08:00
ac0f6e75d2
[bugfix](iceberg)Read error when timestamp does not have time zone for 2.1 ( #36435 )
...
bp: #36141
2024-06-20 18:32:31 +08:00
cbaff8a700
[fix](nereids)change the decimal's precision and scale for cast(xx as decimal) ( #36540 )
...
pick from master #36316
expression cast( xx as decimal )'s datatype maybe decimalv3 or decimalv2
depending on enable_decimal_conversion value in fe conf file. if
enable_decimal_conversion is true, the datatype is decimalv3(9, 0), but
the datatype was decimalv3(38, 9) in 2.0 releases. So this pr change the
datatype same as 2.0 releases to keep the behavior consistent.
2024-06-20 17:46:11 +08:00
4008a04da7
[bugfix](paimon)Fix field case issues for 2.1 ( #36288 )
...
bp: #36239
2024-06-17 18:38:00 +08:00
d8eac07178
[branch-2.1](test) fix external p0 unstable test ( #36262 )
...
Fix some unstable external p0 tests
2024-06-13 20:55:41 +08:00
9e972cb0b9
[bugfix](iceberg)Fix the datafile path error issue for 2.1 ( #36066 )
...
bp: #35957
2024-06-08 21:51:46 +08:00
bd6b913e00
[bugfix](paimon)paimon's field length judgment error for 2.1 ( #36049 )
...
bp #35981
2024-06-07 21:13:08 +08:00
cb96a79d07
[bugfix](iceberg)fix datetime conversion error and data path error ( #35708 )
...
## Proposed changes
Issue #31442
<!--Describe your changes.-->
1. The unit of the seventh parameter of `ZonedDateTime.of` is
nanosecond, so we should multiply the microsecond by 1000.
2. When writing to a non-partitioned iceberg table, the data path has an
extra slash
2024-06-01 00:42:48 +08:00
bc062a2595
[fix](orc)fix orc reader missing column. ( #35735 )
...
## Proposed changes
bp #35583
Issue Number: close #xxx
<!--Describe your changes.-->
2024-05-31 22:51:44 +08:00
b91d2caab8
[Feature](iceberg-writer) Implements iceberg sink basic functionality for inserting into table. ( #35587 )
...
backport #34929
2024-05-29 16:40:54 +08:00
84e9a14063
[Fix](hive-writer) Fix partition column orders issue when the partition fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. ( #35543 )
...
## Proposed changes
backport #35347
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org ](mailto:dev@doris.apache.org ) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
2024-05-28 18:11:55 +08:00
68eda58a8c
[Fix](multi-catalog) Fix string dict filtering when use null related function in parquet and orc reader. ( #35335 )
...
The following sql and when the dictionary column contains functions related to null, the results will be incorrect.
```
select * from ( select IF(o_orderpriority IS NULL, 'null', o_orderpriority) AS o_orderpriority from test_string_dict_filter_orc ) as A where o_orderpriority = 'null';
```
```
select * from ( select IFNULL(o_orderpriority, 'null') AS o_orderpriority from test_string_dict_filter_parquet ) as A where o_orderpriority = 'null'
```
```
select * from ( select COALESCE(o_orderpriority, 'null') AS o_orderpriority from test_string_dict_filter_parquet ) as A where o_orderpriority = 'null';
```
2024-05-27 15:25:29 +08:00
f98ed4e4c5
[bugfix](hive)Misspelling of class names ( #34981 )
2024-05-27 15:24:38 +08:00
b1795d44ec
[bugfix](hive)fix testcase for test_hive_write_different_path ( #35209 )
...
Hive's test environment uses docker, so when using 127.0.0.1,
BE will write the file to the docker of its own machine.
But if FE and are not on the same machine,
FE cannot read this file because it can only read docker on its own machine.
Therefore, the address 127.0.0.1 cannot be used in the test environment.
2024-05-27 15:24:30 +08:00
f6beeb1ddd
[Enhencement](tvf) select tvf supports using resource ( #35139 )
...
Create an S3/HDFS resource that TVF can use it directly to access the data source.
2024-05-24 16:23:58 +08:00
adc364a6fd
[feature](Paimon) support deletion vector for Paimon naive reader ( #34743 ) ( #35241 )
...
bp #34743
Co-authored-by: 苏小刚 <suxiaogang223@icloud.com >
2024-05-23 00:01:30 +08:00
24990383ff
[refactor](jdbc catalog) split clickhouse jdbc executor ( #34794 ) ( #35174 )
...
pick master #34794
2024-05-22 19:09:05 +08:00
291cf57c54
[Configurations](multi-catalog) Add enable_parquet_filter_by_min_max and enable_orc_filter_by_min_max Session variables. ( #35012 ) ( #35164 )
...
backport #35012
2024-05-22 19:06:12 +08:00
c0fd98abe5
[Fix](tvf) Fix that tvf reading empty files in compressed formats. ( #34926 )
...
1. Fix the issue with tvf reading empty compressed files.
2. move two test cases (`test_local_tvf_compression` and `test_s3_tvf_compression`) from p2 to p0
2024-05-21 12:59:31 +08:00
22f85be712
[fix](hive-ctas) support create hive table with full quolified name ( #34984 )
...
Before, when executing `create table hive.db.table as select` to create table in hive catalog,
if current catalog is not hive catalog, the default engine name will be filled with `olap`, which is wrong.
This PR will fill the default engine name base on specified catalog.
2024-05-18 18:42:43 +08:00
4dd5379951
[bugfix](hive)fix error for writing to hive for 2.1 ( #34518 )
...
mirror #34520
2024-05-14 23:27:29 +08:00
