Commit Graph

7816 Commits

Author SHA1 Message Date
fe9571c2fd [typo](docs) fix typo in get-starting.md (#15345) 2022-12-25 21:56:44 +08:00
e640f49b6d [refactor](non-vec) remove non vectorized predicate and row_block (#15348)
remove non vectorized predicate and row_block
2022-12-25 21:45:00 +08:00
691790a46f [typo](docs) fix typo in tablet-repair-and-balance (#15341) 2022-12-25 10:12:02 +08:00
a291cb17be [fix](information-schema) fix bug that query tables in information_schema db will return error #15336 2022-12-25 10:09:40 +08:00
0cda82ad5a [typo](docs) fix typo in tablet-repair-and-balance (#15341) 2022-12-25 09:48:16 +08:00
d48abd91df [deps](fe)upgrade deps version (#15262)
upgrade hadoop version to 2.10.2
jackson-databind to 2.14.1
2022-12-24 22:18:10 +08:00
82d316b419 [bug](decimalv3) Fix wrong decimal scale for arithmetic expr (#15316) 2022-12-24 21:57:46 +08:00
6151a43e9c [Thirdparty](Protobuf) update protobuf from 3.14.0 to 3.15.0 (#15055) 2022-12-24 20:45:11 +08:00
fd764b3ccd [fix](fe)add session variable group_concat_max_len (#15254) 2022-12-24 20:07:14 +08:00
5cefd05869 [fix](multi-catalog) fix and optimize iceberg v2 reader (#15274)
Fix three bugs when read iceberg v2 tables:
1. The `delete position` in `delete file` represents the position of delete row in the entire file, but the `read range` in 
`RowGroupReader` represents the position in current row group. Therefore, we need to subtract the position of first 
row of current row group from `delete position`.
2. When only reading the partition columns, `RowGroupReader` skips processing the `delete position`.
3. If the `delete position` has delete all rows in a row group, the `read range` is empty, but we read the whole row 
group in such case.

Optimize four performance issues:
1. We change `delete position` to `delete range`, and then merge `delete range` and `read range` into the final read 
ranges. This process is too tedious and time-consuming. . we can merge `delete position` and `read range` directly.
2. `delete position` is ordered in a `delete file`, so we can use merge-sort, instead of ordered-set.
3. Initialize `RowGroupReader` when reading, instead of initialize all row groups when opening a `ParquetReader`, to 
save memory usage, and the same as `IcebergReader`.
4. Change the recursive call of `_do_lazy_read` to loop logic.
2022-12-24 16:02:07 +08:00
907cbcde69 [doc](compile) update docker compile image version (#15300)
Add new docker compile image tag: apache/doris:build-env-for-1.2
2022-12-24 15:28:03 +08:00
e72404c537 [fix](scan) fix that be may core dump when the predicates are all false (#15332) 2022-12-24 15:27:43 +08:00
acd16b62e2 [feature](multi-catalog) support use catalog.db when client connect to the doris server (#15293) 2022-12-24 15:20:00 +08:00
9d83c59764 [samples](golang)add golang stream load and read bitmap sample #15273
Co-authored-by: hourong <hourong@zhihu.com>
2022-12-23 22:22:52 +08:00
06f71f2bca [pipeline](fix) Fix bugs to pass all regression cases (#15306)
* [pipeline](fix) Fix bugs to pass all regression cases

* update

* update
2022-12-23 22:17:50 +08:00
cf9217c0ca [typo](docs)fix 404 err to Monitoring and alarming doc #15324 2022-12-23 22:15:54 +08:00
95b4a4a1ba [fix](s3 load) fix that FE can not access s3 objected-storage (#15323) 2022-12-23 22:15:32 +08:00
e31847e925 support rename hive/jdbc engine external table (#15308) 2022-12-23 21:04:37 +08:00
8c0de789e4 [feature](Nereids) support table generating function (#15121) 2022-12-23 20:36:33 +08:00
19cc65cc24 [fix](Nereids): fix bug of converting to NLJ. (#15290) 2022-12-23 19:33:45 +08:00
ede68e075d [fix](iceberg-v2) fix fe iceberg split, add regression case (#15299) 2022-12-23 19:33:00 +08:00
a98636a970 [bugfix](from_unixtime) fix timezone not work for from_unixtime (#15298)
* [bugfix](from_unixtime) fix timezone not work for from_unixtime
2022-12-23 19:05:09 +08:00
bfaaa2bd7c [feature](Nereids) support digital_masking function (#15252) 2022-12-23 18:59:08 +08:00
2f089be37e [feature](nereids) support bitAnd/ bitOr/ bitXor (#15261) 2022-12-23 18:39:39 +08:00
06d0035c02 [refactor](non-vec)remove schema change related non-vec code (#15313)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2022-12-23 18:33:04 +08:00
27d64964e6 [enhancement](Nereids) cast expression to the type with parameters (#14657) 2022-12-23 18:29:50 +08:00
ef3da105c9 [DOCS](refactor) refine en docs (#15244)
* Update basic-summary.md

* Update README.md
2022-12-23 16:47:51 +08:00
00fd5b1b1c [typo](doc) update Paxos spell mistake (#15171) 2022-12-23 16:47:12 +08:00
e7a077a81f [fix](jdbc catalog) fix bugs of jdbc catalog and table valued function (#15216)
* fix bugs

* add `desc function` test

* add test

* fix
2022-12-23 16:46:39 +08:00
e336178ef8 [Fix](multi catalog)Fix VFileScanner file not found status bug. #15226
The if condition to check NOT FOUND status for VFileScanner is incorrect, fix it.
2022-12-23 16:45:54 +08:00
b935fd0e7d [fix](fe)fix bug of the bucket shuffle join is not recognized (#15255)
* [fix](fe)fix bug of the bucket shuffle join is not recognized

* use broadcast join for empty table
2022-12-23 16:44:44 +08:00
1926239f09 [improvement](test) add --conf option for run-regression-test.sh for custom config file (#15287)
* add --conf option for run-regression-test.sh for custom config file

* fix shell check error
2022-12-23 16:43:18 +08:00
8a810cd554 [fix](bitmapfilter) fix core dump caused by bitmap filter (#15296)
Do not push down the bitmap filter to a non-integer column
2022-12-23 16:42:45 +08:00
8515a03ef9 [fix](compile) fix compile error caused by mysql_scan_node.cpp not being found when enabling WITH_MYSQL (#15277) 2022-12-23 16:25:28 +08:00
764b1db097 [fix](s3 outfile) Add theuse_path_style parameter for s3 outfile (#15288)
Currently, `outfile` did not support `use_path_style` parameter and use `virtual-host style` by default,
however some Object-storage may only support `use_path_style` access mode.

This pr add the`use_path_style` parameter for s3 outfile, so that different object-storage can use different access mode.
2022-12-23 16:22:06 +08:00
4b7f279cf9 [Enhancement](Nereids) change expression to conjuncts in filter (#14807) 2022-12-23 15:31:40 +08:00
fe562bc3e7 [Bug](Agg) fix crash when encountering not supported agg function like last_value(bitmap) (#15257)
The former logic inside aggregate_function_window.cpp would shutdown BE once encountering agg function with complex type like BITMAP. This pr makes it don't crash and would return one more concrete error message which tells the unsupported function signature to user.
2022-12-23 14:23:21 +08:00
cb295de981 [Bug](decimalv3) Fix wrong precision of DECIMALV3 (#15302)
* [Bug](decimalv3) Fix wrong precision of DECIMALV3

* update
2022-12-23 14:11:08 +08:00
b085ff49f0 [refactor](non-vec) delete non-vec data sink (#15283)
* [refactor](non-vec) delete non-vec data sink

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-12-23 14:10:47 +08:00
38530100d8 [fix](localgc) check gc only cache directory (#15238) 2022-12-23 10:40:55 +08:00
82fbfab77f [fix](union)the union node should not pass through children in some case (#15286)
the union node will make children pass through in wrong condition. If the children's materialized slots are different from union node, children can't be passed through.
2022-12-23 10:27:49 +08:00
020c47f528 [load](config) update max timeout (#15280) 2022-12-23 10:15:26 +08:00
Pxl
6b3721af23 [Bug](function) fix core dump on reverse() when big string input
fix core dump on reverse() when big string input
2022-12-23 10:14:09 +08:00
09a22813e4 [feature](Nereids) support syntax SELECT DISTINCT (#15197)
Add a new rule 'ProjectWithDistinctToAggregate' to support "select distinct xx from table".
This rule check's the logicalProject node's isDisinct property and replace the logicalProject node with a LogicalAggregate node.
So any rule before this, if createing a new logicalProject node, should make sure isDisinct property is correctly passed around.
please see rule BindSlotReference or BindFunction for example.
2022-12-22 23:54:08 +08:00
83a99a0f8b [refactor](non-vec) Remove non vec code from be (#15278)
* [refactor](removecode) remove some non-vectorization
Co-authored-by: yiguolei <yiguolei@gmail.com>
2022-12-22 23:28:30 +08:00
67647f0cf6 [fix](Nereids): fix bug of converting to NLJ. (#15268) 2022-12-22 23:05:39 +08:00
df5969ab58 [Feature] Support function roundBankers (#15154) 2022-12-22 22:53:09 +08:00
388df291af [pipeline](schedule) Add profile for except node and fix steal task problem (#15282) 2022-12-22 22:42:37 +08:00
e331e0420b [improvement](topn)add per scanner limit check for new scanner (#15231)
Optimize for key topn query like `SELECT * FROM store_sales ORDER BY ss_sold_date_sk, ss_sold_time_sk LIMIT 100` 
(ss_sold_date_sk, ss_sold_time_sk is prefix of table sort key). 

Check per scanner limit and set eof true to reduce the data need to be read.
2022-12-22 22:39:31 +08:00
d38461616c [Pipeline](error msg) format error message (#15247) 2022-12-22 20:55:06 +08:00