Commit Graph

12836 Commits

Author SHA1 Message Date
8ef6b4d996 [fix](json) fix json int128 overflow (#22917)
* support int128 in jsonb

* fix jsonb int128 write

* fix jsonb to json int128

* fix json functions for int128

* add nereids function jsonb_extract_largeint

* add testcase for json int128

* change docs for json int128

* add nereids function jsonb_extract_largeint

* clang format

* fix check style

* using int128_t = __int128_t for all int128

* use fmt::format_to instead of snprintf digit by digit for int128

* clang format

* delete useless check

* add warn log

* clang format
2023-08-25 11:40:30 +08:00
372f83df5c [opt](Nereids) remove between expression to simplify planner (#23421) 2023-08-25 11:28:12 +08:00
37b90021b7 [fix](planner)literal expr should do nothing in substituteImpl() method (#23438)
substitute a literal expr is pointless and wrong. This pr keep literal expr unchanged during substitute process
2023-08-25 11:21:35 +08:00
18094511e7 [fix](Outfile/Nereids) fix that csv_with_names and csv_with_names_and_types file format could not be exported on nereids (#23387)
This problem is casued by #21197

Fixed an issue that `csv_with_names` and `csv_with_names_and_types` file format could not be exported on nereids optimizer when using `select...into outfile`.
2023-08-25 11:12:04 +08:00
ffadf09eec [fix](catalog)add custom jar (#23406)
- allow put custom jar in `${DORIS_HOME}/lib/java_extensions/custom_extension` such as `paimon-s3-0.4.0-incubating.jar`
- add some note for paimon and fqdn
2023-08-25 11:10:53 +08:00
6614c219cb [opt](Nereids) use NUMERIC_PRECEDENCE in int div (#23403) 2023-08-25 11:03:50 +08:00
69e75f04ab [fix](feut) should not enable InternalSchemaDb in fe ut (#23400) 2023-08-25 11:03:37 +08:00
3786ffec51 [opt](Nereids) add some array functions (#23324)
1. rename TVFProperties to Properties
2. add generating function explode and explode_outer
3. fix concat_ws could not apply on array
4. check tokenize second argument format on FE
5. add test case for concat_ws, tokenize, explode, explode_outer and split_by_string
2023-08-25 11:01:50 +08:00
d331bfc513 [Performance](pipeline) support shared scan segment in mow (#23305) 2023-08-25 10:43:02 +08:00
Pxl
d9db3f5431 [Improvement](scan) Remove redundant predicates on scan node (#23374)
* Remove redundant predicates on scan node

* update

* fix
2023-08-25 10:41:37 +08:00
a305f2ffc2 [fix](pipeline) update status when prepare failed #23419 2023-08-25 10:34:37 +08:00
0a70cbfe99 [feature](move-memtable)[5/7] add olap table sink v2 and writers (#23458)
Co-authored-by: laihui <1353307710@qq.com>
2023-08-25 10:20:06 +08:00
2847c5e5b8 [Optimize](index) Optimize implement the new internal lucene index query interface (#23389) 2023-08-25 10:14:02 +08:00
d30bb8042e [fuzzy](hash join) disable fuzzy enable_hash_join_early_start_probe (#23413) 2023-08-25 10:11:20 +08:00
9cacf9535a [Opt](functions) Use preloaded cache to accelerate timezone parsing (#22694)
* opt

* bugfix

* fix ut

* fix stylecheck
2023-08-25 10:00:48 +08:00
7cfb3cc0aa [fix](functions) fix function substitute for datetimeV1/V2 (#23344)
* fix

* function fe
2023-08-25 09:59:38 +08:00
467e48053b [FIX](regresstest) fix regress test for export with scp files (#23417)
* fix regress test for export with scp

* fix outfile and outputfilepath
2023-08-25 09:22:34 +08:00
bc3d397759 [fix](case) update .out file, relate to #23272 (#23455)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-08-25 09:15:27 +08:00
e44f14507d [github](action) rm required tag for 'BE UT (Clang)' and 'BE UT (macOS)' #23422
These two checks often cost about 2 hours, which slowed down the CI process,
After discussion, maybe we can rm the required tag first, then set compile with clang in BE UT from TeamCity.
2023-08-25 09:12:20 +08:00
ceb931c513 [regression-test](hdfs_tvf)append regression test that hdfs_tvf read compression file (#23454) 2023-08-25 09:00:21 +08:00
441a9fff6d [fix](planner) fix now function param type error (#23446) 2023-08-25 00:12:21 +08:00
71071ba057 [feature](move-memtable)[4/7] add stream sink file writer (#23416)
Co-authored-by: laihui <1353307710@qq.com>
2023-08-25 00:08:27 +08:00
98d0a2f6c1 [feature](move-memtable)[3/7] add load stream manager and rpc service (#23415)
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
Co-authored-by: Yongqiang YANG <dataroaring@gmail.com>
Co-authored-by: laihui <1353307710@qq.com>
2023-08-25 00:08:04 +08:00
b65023e667 [fix](load) avoid using protobuf set_allocated_id in VDataStreamSender (#23435) 2023-08-25 00:07:09 +08:00
6a4976921d [fix](auth)Disable column auth temporarily (#23295)
- add config `enable_col_auth` to temporarily disable column permissions(because old/new planner has bug when select from view)
- Restore the old optimizer to the previous authentication method
- Support for new optimizer authentication(Legacy issue: When querying the view, the permissions of the base table will be authenticated. The view's own permissions should be authenticated and processed after the new optimizer is improved)
- fix: show grants for non-existent users
- fix: role:`admin` can not grant/revoke to/from user
2023-08-24 23:37:06 +08:00
966561a6ed [improvement and fix](statistics)Load the cache for external table row count while init table (#23170)
1. Load the cache for external table row count while init table, this could avoid no row number stats for the very first time to run an sql.
2. Show cardinality for an external scan node when explain the sql.
3. fix bugs introduced by https://github.com/apache/doris/pull/22963
2023-08-24 23:34:16 +08:00
f6c5c8f7b5 [Fix](Nereids) fix that select...from tablets() are invalidated when there exists predicates (#23365)
Problem: `select...from tablets()` are invalidated when there exists predicates, such as:
```sql
// The all data is:
mysql> select * from student3;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    1 | ftw  |   18 |
|    3 | yy   |   19 |
|    4 | xx   |   21 |
|    2 | cyx  |   20 |
+------+------+------+

// when we specified tablet to read:
mysql> select * from student3 tablet(131131);
+------+------+------+
| id   | name | age  |
+------+------+------+
|    1 | ftw  |   18 |
|    3 | yy   |   19 |
+------+------+------+

// Howerver, when there exists predicates, the `tablet(131131)` is invalidated
mysql> select * from student3 tablet(131131) where id > 1;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    4 | xx   |   21 |
|    3 | yy   |   19 |
|    2 | cyx  |   20 |
+------+------+------+
```

After the fix, we get promising data
```sql
mysql> select * from student3 tablet(131131) where id > 1;
+------+------+------+
| id   | name | age  |
+------+------+------+
|    3 | yy   |   19 |
+------+------+------+
```
2023-08-24 23:29:59 +08:00
caddcc6215 [Fix](orc-reader) Fix decimal type check for ColumnValueRange issue and use primitive_type. (#23424)
Fix decimal type check for ColumnValueRange issue and use primitive_type in orc_reader. Because in #22842 the `CppType` of `PrimitiveTypeTraits<TYPE_DECIMALXXX> ` were changed.
2023-08-24 23:26:41 +08:00
55e572df82 [pipelineX](analytic operator) Support analytic operator (#23444) 2023-08-24 23:05:29 +08:00
adafcc1f7a [typo](docs) Fix Sidebar Category Name of CN Version and Capitalize Files Name (#23050) 2023-08-24 22:56:23 +08:00
320eda78e6 [fix](nereids) remove useless cast in in-predicate (#23171)
consider sql "select * from test_simplify_in_predicate_t where a in ('1992-01-31', '1992-02-01', '1992-02-02', '1992-02-03', '1992-02-04');"
before:

```
|   0:VOlapScanNode                                                                                                                                                                                      |
|      TABLE: default_cluster:bugfix.test_simplify_in_predicate_t(test_simplify_in_predicate_t), PREAGGREGATION: OFF. Reason: No aggregate on scan.                                                      |
|      PREDICATES: CAST(a[#0] AS DATETIMEV2(0)) IN ('1992-01-31 00:00:00', '1992-02-01 00:00:00', '1992-02-02 00:00:00', '1992-02-03 00:00:00', '1992-02-04 00:00:00') AND __DORIS_DELETE_SIGN__[#1] = 0 |
|      partitions=0/1, tablets=0/0, tabletList=                                                                                                                                                          |
|      cardinality=1, avgRowSize=0.0, numNodes=1                                                                                                                                                         |
|      pushAggOp=NONE                                                                                                                                                                                    |
|      projections: a[#0]                                                                                                                                                                                |
|      project output tuple id: 1                                                                                                                                                                        |
|      tuple ids: 0  
```
after:

```
|   0:VOlapScanNode                                                                                                                                 |
|      TABLE: default_cluster:bugfix.test_simplify_in_predicate_t(test_simplify_in_predicate_t), PREAGGREGATION: OFF. Reason: No aggregate on scan. |
|      PREDICATES: a[#0] IN ('1992-01-31', '1992-02-01', '1992-02-02', '1992-02-03', '1992-02-04') AND __DORIS_DELETE_SIGN__[#1] = 0                |
|      partitions=0/1, tablets=0/0, tabletList=                                                                                                     |
|      cardinality=1, avgRowSize=0.0, numNodes=1                                                                                                    |
|      pushAggOp=NONE                                                                                                                               |
|      projections: a[#0]                                                                                                                           |
|      project output tuple id: 1                                                                                                                   |
|      tuple ids: 0  

```
2023-08-24 18:14:43 +08:00
6c5072ffc5 [FIX](array-func) fix array index func with decimal (#23399)
fix array index func with decimal
in old analyzer when sql with array_position or array_contains with decimal , may loss precision to which will make result wrong
2023-08-24 17:58:20 +08:00
96164f3bdc [pipelinex](sort) Fix expression initialization order (#23405) 2023-08-24 17:29:24 +08:00
e262c8b4be [chore](fe)correct some doc errors (#23398) 2023-08-24 17:05:13 +08:00
c6ac925f5a [fix](common) implement the move assignment operator for Status (#23372) 2023-08-24 14:41:19 +08:00
303d5ac3f7 Revert "[enhancement](thirdparty) upgrade thirdparty libs (#23290)" (#23420)
This reverts commit 2185268bab2f5890a2c3c9a3c315375a971c4416.

Co-authored-by: stephen <hello-stephen@qq.com>
2023-08-24 14:40:51 +08:00
c775f8e7bd [feature](move-memtable)[2/7] add protos for memtable on sink node (#23348)
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
Co-authored-by: laihui <1353307710@qq.com>
2023-08-24 11:11:46 +08:00
687c676160 [FIX](map)fix column map for offset next_array_item_rowid order (#23250)
* fix column map for offset next_array_item_rowid order

* add regress test
2023-08-24 10:57:40 +08:00
044423f902 [fix](Nereids) use Stopwatch to do timeout checker (#23383)
1. avoid thread leak if exception thrown in planning
2. avoid memory release delays since the timer task hold CascadesContext
   object
2023-08-24 09:49:35 +08:00
Pxl
376d54de41 [Chore](function) add check for comparison input type #23330 2023-08-24 08:28:31 +08:00
f0bc2c2eff [fix](tablet clone) fix partition rebalance apply move exception (#23222) 2023-08-23 21:50:50 +08:00
156f7b7699 [improvement](transaction) make commit txn fail hint more understandable (#23227) 2023-08-23 21:50:24 +08:00
448b7755c6 [feature](jdbc catalog) support doris jdbc catalog array type (#23056) 2023-08-23 21:17:16 +08:00
8bdb75e752 [Enhancement](inverted index) remove time-consuming log prints in bkd reader (#23381) 2023-08-23 21:03:01 +08:00
daa4db097e [fix](Nereids) array_difference and array_position get wrong result (#23331)
1. change array_difference signature to let it return same type as arg
2. do not change precision when signature not use wildcard type
2023-08-23 20:38:09 +08:00
2185268bab [enhancement](thirdparty) upgrade thirdparty libs (#23290)
protobuf 3.15.0 -> 21.11
glog 0.4.0 -> 0.6.0
lz4 1.9.3 -> 1.9.4
curl 7.79.0 -> 8.2.1
brpc 1.4.0 -> 1.6.0
zstd 1.5.2 -> 1.5.5
arrow 7.0.0 -> 13.0.0
abseil 20220623.1 -> 20230125.3
orc 1.7.2 -> 1.9.0
jemalloc for arrow 5.2.1 -> 5.3.0
xsimd 7.0.0 -> 13.0.0
opentelemetry-proto 0.19.0 -> 1.0.0
opentelemetry 1.8.3 -> 1.10.0

new:
c-ares -> 1.19.1
grpc -> 1.54.3
2023-08-23 20:25:49 +08:00
dad1587799 [opt](stats) don't load stats when task is killed (#23354)
Sync cache after analyze task is killed is meaningless and make error message confused
2023-08-23 20:24:08 +08:00
1b7d692d72 [fix](planner & nereids) convert to double if div decimal overflow (#23272) 2023-08-23 20:10:53 +08:00
bdb2f09e9f [fix](test) fix unstable pk uk case (#23390) 2023-08-23 20:09:37 +08:00
9d1f2cd8e0 [Improvement](pipeline) Terminate early for short-circuit join (#23378) 2023-08-23 19:40:17 +08:00