database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
18,869 Commits 1 Branch 0 Tags
aaa89ec768080ceecbfff5b3ecce8f140f514f4e
Commit Graph

144 Commits

Author SHA1 Message Date
苏小刚
7381cd56b0 [docker](hive) sync for hive initializing (#35479) 
Add healthy checking for hive2 and hive3
 2024-05-29 15:03:06 +08:00
Qi Chen
68eda58a8c [Fix](multi-catalog) Fix string dict filtering when use null related function in parquet and orc reader. (#35335) 
The following sql and when the dictionary column contains functions related to null, the results will be incorrect.
```
select * from ( select IF(o_orderpriority IS NULL, 'null', o_orderpriority) AS o_orderpriority from test_string_dict_filter_orc ) as A where o_orderpriority = 'null';
```
```
select * from ( select IFNULL(o_orderpriority, 'null') AS o_orderpriority from test_string_dict_filter_parquet ) as A where o_orderpriority = 'null'
```
```
select * from ( select COALESCE(o_orderpriority, 'null') AS o_orderpriority from test_string_dict_filter_parquet ) as A where o_orderpriority = 'null';
```
 2024-05-27 15:25:29 +08:00
HHoflittlefish777
50f50cf8cc Revert "[fix][docker] fix kafka test scritps (#33417)" (#35229) 
This reverts commit c35b2becdd08ab9255b3a0c2a19d74970f621388.
 2024-05-22 20:33:14 +08:00
zgxme
bc70968019 [chore](regression) Modify character encoding to be consistent with Doris (#35228) 2024-05-22 20:04:50 +08:00
苏小刚
4ecc3edc21 [test](hive)revert hive container to host mode (#34322) 
Revert hive container to host mode to fix pipeline problem
 2024-05-07 10:36:01 +08:00
Qi Chen
99af54f779 [Fix](orc-reader) Fix the issue when string col has mixed plain and dict encoding in different stripes. (#34146) (#34248) 
backport #34146
 2024-04-28 19:43:57 +08:00
Qi Chen
acc2b532e7 [Test](hive-writer) Adjust test_hive_write_partitions regression test to resolve special characters issue with git on windows. (#34026) 2024-04-26 15:05:47 +08:00
Mingyu Chen
7f4b7b04ad [test](hive)add subnet for hive docker compose (#34000) (#34157) 
bp #34000
Co-authored-by: 苏小刚 <suxiaogang223@icloud.com>
 2024-04-26 13:49:33 +08:00
Mingyu Chen
50f9d47e96 [test](hive) run suite cases both in hive2 and hive3 (#33874) (#34156) 
bp #33874

Co-authored-by: 苏小刚 <suxiaogang223@icloud.com>
 2024-04-26 13:48:09 +08:00
苏小刚
1c025c0488 [docker](hive) add hive3 docker compose and modify scripts (#33115) 
add hive3 docker compose from:
big-data-europe/docker-hive#56
 2024-04-17 23:42:13 +08:00
Qi Chen
4963d60a07 [Fix](multi-catalog)Fix the issue of not initializing the writer caused by refactoring and add hive writing regression test. (#32721) (#33446) 
backport #32721.
 2024-04-10 11:42:22 +08:00
XuJianxu
c35b2becdd [fix][docker] fix kafka test scritps (#33417) 
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>
 2024-04-09 16:11:09 +08:00
Mingyu Chen
73de61ed84 [opt](hive) skip hidden file and dir (#32412) 
When query hive table, we should skip all hidden dirs and files, like:
```
/visible/.hidden/path
/visible/.hidden.txt
```
 2024-03-21 14:07:24 +08:00
wuwenchi
926908ece2 [fix](hive) fix spelling mistakes for "separatorChar" #32061 2024-03-12 14:20:18 +08:00
zy-kkk
2e9bd268cd [improvement](jdbc catalog) support sqlserver timestamp type read (#31805) 2024-03-06 13:08:04 +08:00
Ashin Gau
248ea20901 Revert "[test](regression) add regression test for schange change of complex …" (#31660) 
This reverts commit dcd2afdb4e857791fed66a46f28ab3adc25494e1.
Reverts #31207
 2024-03-01 19:06:59 +08:00
zy-kkk
07224686ef [feature](jdbc catalog) support db2 jdbc catalog (#31627) 2024-03-01 14:19:28 +08:00
XuJianxu
5276cc4db6 [docker][fix] update routine load cases (#31553) 
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>
 2024-02-29 16:44:39 +08:00
zy-kkk
3c37fb085c [refactor](jdbc catalog) split jdbc executor for different data sources (step-1) (#31406) 2024-02-29 12:38:03 +08:00
Ashin Gau
e3b4b83bca [test](regression) add regression test for schange change of complex type (#31207) 
Add regression test for #31128
 2024-02-22 19:50:07 +08:00
zy-kkk
b3ac2128dd [Refactor](catalog) Refactor Jdbc Catalog external name case mapping rules (#28414) 2024-02-19 17:22:03 +08:00
Guangdong Liu
8db2824c44 [bugfix](es catalog) add constant_keyword wildcard data type (#30947) 2024-02-19 17:20:21 +08:00
Qi Chen
92cad69fc4 [Fix](parquet-reader) Fix reading fixed length byte array decimal in parquet reader. (#30535) 2024-01-31 23:53:40 +08:00
zhangdong
658c869aac [improvement](mtmv)mtmv support partition by hms table (#29989) 2024-01-29 19:02:46 +08:00
chunping
9b100207d2 [docker](container)create data path after rm residual data avoiding docker start fail (#30141) 2024-01-23 10:09:54 +08:00
qiye
fac0580eae [opt](docker)optimize ES docker compose (#30068) 
1. add volume for es logs
2. optimize health check, waiting for es status to be green
3. fix es6 valume path error
4. optimize disk watermark to avoid es disk watermark error
5. fix es6 create index error
6. add custom elasticsearch.yml for es6
7. add log4j2.properties for es6, es7, es8
 2024-01-19 15:48:56 +08:00
wuwenchi
44ba9e102c [feature](statistics)support statistics for iceberg/paimon/hudi table (#29868) 2024-01-18 12:03:07 +08:00
zy-kkk
5d0732fcf4 [typo](docker thirdparties) Modify script prompt information error (#29770) 2024-01-12 11:53:57 +08:00
Dongyang Li
67986a8a6f [fix](case) external case, specify networks (#29699) 
Co-authored-by: stephen <hello-stephen@qq.com>
 2024-01-12 11:46:29 +08:00
zy-kkk
3cd1c7745a [fix](jdbc catalog) Fix the precision of decimal type mapping to 0 (#29407) 2024-01-12 11:39:57 +08:00
qiye
2c4e52e44e [fix](es catalog) only es_query function can push down to ES (#29320) 
Issue Number: close #29318 
1. Only push down `es_query` function to ES
2. Add null check where ES query result not have `_source` or `fields` fields.
 2023-12-30 09:33:26 +08:00
wuwenchi
7da86c37ec [fix](hive) add support for quoteChar and seperatorChar for hive (#28613) 
add support for quoteChar and seperatorChar .
 2023-12-19 19:35:03 +08:00
XuJianxu
608baae001 [docker][regregression]update routine load cases #28450 
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>
 2023-12-16 17:57:39 +08:00
bobhan1
01c94a554d [fix](autoinc) Fix broker load when target table has autoinc column (#28402) 2023-12-14 18:02:54 +08:00
Jibing-Li
a271fee3c5 [test](statistics)Add external empty table test case. (#28267) 2023-12-13 21:48:01 +08:00
Qi Chen
60bc3be8a2 [Opt](Compression) Opt zstd block decompression by ZSTD_decompressDCtx(). (#27534) 
Opt zstd block decompression by `ZSTD_decompressDCtx()` to replace streaming decompression.
It will improve performance but consume more memory. 

Test result: 
- env: 1 node(16 cores, 64G).
- parquet column: 100 million rows of char(255) column.
- result: 5.2 -> 4.6.
 2023-12-01 09:10:32 +08:00
Qi Chen
e4149c6e4c [Fix](parquet-reader) Fix null map issue in parquet reader. (#27777) 
Fix null map issue in parquet reader which cause result incorrect such as `min()`, `max()`.

In order to share null map between parquet converted src column and dst column to avoid copying. It is very tricky that will call mutable function `doris_nullable_column->get_null_map_column_ptr()` which will set `_need_update_has_null = true`. Because some operations such as agg will call `has_null()` to set `_need_update_has_null = false`.
 2023-11-30 13:55:37 +08:00
Qi Chen
cc395f5428 [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27563) 2023-11-25 10:29:39 +08:00
zy-kkk
b477839bce [enhancement](jdbc catalog) Add lowercase column name mapping to Jdbc data source & optimize database and table mapping (#27124) 
This PR adds the processing of lowercase Column names in Oracle Jdbc Catalog. In the previous behavior, we changed all Oracle columns to uppercase queries by default, but could not handle the lowercase case. This PR can solve this situation and improve All Jdbc Catalog works
 2023-11-17 23:51:47 +08:00
zy-kkk
df867a1531 [fix](catalog) Fix ClickHouse DataTime64 precision parsing (#26977) 2023-11-15 10:23:21 +08:00
zhangguoqiang
1edeacd0a5 [enhance](regression) enhance docker network by add network subnet (#26862) 2023-11-14 20:06:20 +08:00
daidai
3585c7e216 [test](parquet)append parquet reader byte_array_decimal and rle_bool case (#26751) 2023-11-14 15:05:10 +08:00
zy-kkk
7ce746654a [test](jdbc) add doris and sqlserver jdbc catalog test case (#26656) 2023-11-10 10:32:09 +08:00
Tiewei Fang
49cffd0bc9 [fix](JdbcCatalog) fix that the predicate column name does not have back quote when querying the JDBC appearance (#26479) 2023-11-10 09:54:39 +08:00
zy-kkk
8434389358 [fix](jdbc) fix clickhouse catalog arr nullable and add case (#26639) 2023-11-09 19:32:05 +08:00
wudongliang
22bf2889e5 [feature](tvf)(jni-avro)jni-avro scanner add complex data types (#26236) 
Support avro's enum, record, union data types
 2023-11-09 13:58:49 +08:00
Tiewei Fang
f6b7046a6e [fix](regression-test) add tests for jdbc catalog (#26608) 2023-11-09 11:59:35 +08:00
zy-kkk
5bcf6bfd46 [fix](jdbc catalog) fix mysql zero date (#26569) 2023-11-08 21:41:56 +08:00
Jibing-Li
80f654ec2a [Fix](statistics)Fix analyze min max sql syntax error. #26240 2023-11-02 09:22:32 +08:00
Jibing-Li
78204f7c92 [Fix](statistics)Fix external couldn't analyze database bug (#26025) 2023-10-31 11:32:47 +08:00