Commit Graph

12163 Commits

Author SHA1 Message Date
bfa7f8df6d [fix](Nereids) parse logical binary stack overflow (#22308)
1. not use recursive parse to avoid stack overflow
2. To create a balanced tree instead of left deep tree
TODO: add expr_depth_limit to Nereids' parser
2023-07-28 09:48:17 +08:00
f0f3548dfe [regression](multi-catalog)add EMR cloud env test tools (#21788)
add emr test tools for aliyun, huawei cloud, tencent cloud.
2023-07-28 09:45:10 +08:00
8caa5a9ba4 [Fix](mutli-catalog) Fix null partitions error in iceberg tables. (#22185)
### Issue
when partition has null partitions, it throws error
`Failed to fill partition column: t_int=null`

### Resolution
- Fix the following null partitions error in iceberg tables by replacing null partition to '\N'.
- Add regression test for hive null partition.
2023-07-27 23:57:35 +08:00
00863f25e9 [improvement](profile) add table name for file scan node (#22299)
```
VFILE_SCAN_NODE(region)  (id=0):(Active:  3.537us,  %  non-child:  0.00%)
                                -  RuntimeFilters:  :  
                              -  UseSpecificThreadToken:  False
                              -  AcquireRuntimeFilterTime:  501ns
                              -  AllocateResourceTime:  105.598us
```
2023-07-27 23:54:31 +08:00
442ae632e3 [fix](fs-cache) add 'scheme://authority' to fs cache key (#22263)
This file system cache key should contains `scheme://authority`, eg: `hdfs//nameservices1`.
Or it will encounter error:

```
Wrong FS: hdfs//abc/xxxx, expected: hdfs://def
```
2023-07-27 23:53:54 +08:00
f7d5453be8 [fix](nereids) fix cte bucket shuffle path (#22311) 2023-07-27 22:44:51 +08:00
b5fa29e138 [fix](bitmap) incorrect result of function 'bitmap_from_array' (#22305) 2023-07-27 22:44:06 +08:00
5584d7a5ba [Improve](point query) Improve lookup connection cache from DoubleBuffer to LRU cache for better item pruning (#22041) 2023-07-27 22:22:50 +08:00
8371171e44 [Feature](inverted index) add inverted index tool (#22207) 2023-07-27 21:28:34 +08:00
461c4dfaae [fix](tablet clone) fix single replica load failed during migration (#22077) 2023-07-27 20:38:03 +08:00
e39d234db9 [opt](inverted index) add more check for create inverted index (#22297) 2023-07-27 20:33:24 +08:00
716d58f5ff [fix](Nereids) decimal divide should not return null if numerator is zero (#22309) 2023-07-27 20:23:04 +08:00
687d97e648 [improvement][default_config] enlarge default value compaction related (#22286)
configs

1. Because vertical compaction is enabled by default, it consumes less
memory, we can enlarge default value of compaction related configs.
2. Enlarge default value of shard size related to lock.
2023-07-27 20:17:43 +08:00
816fd50d1d [Enhancement](binlog) Add binlog enable diable check in BinlogManager (#22173)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-07-27 20:16:21 +08:00
a87d34b19b [Fix](multi catalog statistics)Improve external table statistics collection (#22224)
Improve external table statistics collection, including log, observability and fix some bugs.
1. Add Running state for statistics job.
2. Add progress for show analyze job. (n/m tasks finished, n/m task failed and so on)
3. Add analyze time cost for show analyze task.
4. Make task failure message more clear.
5. Synchronize the job status updating code in updateTaskStatus.
6. Fix NPE in HMSAnalyzeTask. (Avoid refreshing statistics cache if the collection sql failed)
7. Return error message for with sync collection while timeout. 
8. Log level improvement
9. Fix misuse of logCreateAnalysisJob for tasks.
2023-07-27 20:01:14 +08:00
391131c7fd [typo](docs)correct describe for grant (#20910) 2023-07-27 20:00:01 +08:00
2c849c619d [fix](nereids) only allow inner join in dphyper join reorder (#22307)
current dphyper join reorder hasn't consider the join conjunct referencing only one side of the child. This is common case in outer join conjunct. So we need disable outer join reorder in dphyper until this problem is addressed.
2023-07-27 19:46:37 +08:00
8a03766c58 [fix](test) change some name in regression test to avoid conflict name running in parallel (#22273) 2023-07-27 19:38:01 +08:00
ae5e39ad26 [opt](Nereids) add double signature back for round like function (#22284)
add double signature back for round like function
2023-07-27 19:10:43 +08:00
d0c369d61b [fix](vec) Arena was not initialized in PartitionMethodSerialized (#22295) 2023-07-27 18:55:57 +08:00
Pxl
87b9425772 [Bug](materialized-view) fix where clause not analyzed after fe restart (#22268)
fix where clause not analyzed after fe restart
2023-07-27 18:34:44 +08:00
b51fcbd9c7 [opt](stats) Scale replica of stats table to 3 when it's possible (#22227)
So that we could improve the availability of stats.
2023-07-27 17:36:54 +08:00
Pxl
05d18b2f68 [Chore](regression-test) remove insert into select on enable nereids dml (#22291)
remove insert into select on enable nereids dml
2023-07-27 17:18:49 +08:00
e29c703ce4 [typo](docs) fix some typo (#21031) 2023-07-27 16:55:48 +08:00
6f1c03c766 [fix](jdbc_catalog) fix int and bigint in mysql view when use doris catalog (#22251) 2023-07-27 16:50:42 +08:00
94d2ba5ba0 [Docs](community) Add Join Community Files of EN & CN Verison (#22252) 2023-07-27 16:47:10 +08:00
b597eef436 [fix](doc)description of check_rpc_channel api in http-actions-be (#21568) 2023-07-27 16:09:20 +08:00
0512e0b168 [test](regression) add cases for partial update with sequence_type (#22215) 2023-07-27 15:51:01 +08:00
4f6a3c5bf0 [feature](catalog) support clob type in oracle jdbc catalog (#21532) 2023-07-27 15:49:15 +08:00
0670e38d5c [pipeline](update)exclude block case test_round (#22290) 2023-07-27 15:38:00 +08:00
e78afedd0a [minor](refactor) refine function logics (#22280) 2023-07-27 15:09:23 +08:00
aa75f79fad [fix](executor)cancel exchange buffer rpc when query is cancelled (#22226)
when brpc client make a request to a server, if the server doesn't response and may not response forever(such as BE restart), the query can be cancelled at once, but the ExchangeSinkBuffer can not be cancelled until rpc timeout.
So we hope when the query is cancelled, the ExchangeSinkBuffer can be closed at once.
2023-07-27 14:38:25 +08:00
9a95d664b9 [chore](third-party) Fix the build order for libunwind (#22244)
1. libunwind depends on lzma
2. Fix the missing headers issues reported by GCC-13
2023-07-27 14:07:08 +08:00
Pxl
05be45bd35 [Improvement](brpc) adjust brpc_light_work_pool_threads/brpc_heavy_work_pool_threads (#22241)
adjust brpc_light_work_pool_threads/brpc_heavy_work_pool_threads
2023-07-27 14:03:46 +08:00
ddfdf62993 [opt](planner) support to parse scientific notation(aEb) (#22248) 2023-07-27 13:31:37 +08:00
a630f127ce [fix](planner) fix bug of push down conjuncts through agg (#22202)
should use both contains and comeFrom method to check if the conjunct can be pushed down throgh agg node
2023-07-27 13:20:50 +08:00
8b51bfa384 [fix](planner) fix bug of unexpected nest loop join (#22236)
use isLiteral instead of isConstant to check if the expr is a literal. This prevent the unexpected nest loop join, see the test case for detail
2023-07-27 13:20:29 +08:00
41a230b721 [fix] iceberg catalog to specify the version and time (#22209)
problem:
1. create a iceberg_type catalog:
2. use iceberg catalog to specify verison
```
mysql> show catalog iceberg;
+----------------------+--------------------------+
| Key                  | Value                    |
+----------------------+--------------------------+
| type                 | iceberg                  |
| iceberg.catalog.type | hms                      |
| hive.metastore.uris  | thrift://127.0.0.1:9083 |
| hadoop.username      | hadoop                   |
| create_time          | 2023-07-25 16:51:00.522  |
+----------------------+--------------------------+
5 rows in set (0.02 sec)

mysql> select * from iceberg.iceberg_db.tb1 FOR VERSION AS OF 8783036402036752909;
ERROR 5090 (42000): errCode = 2, detailMessage = Only iceberg/hudi external table supports time travel in current version
```

change:
Add `ICEBERG_EXTERNAL_TABLE` type for specify the version and time
2023-07-27 12:04:41 +08:00
31c856351a [enhancement](default_config) change default value of rpc related (#22149)
configs

Bdbje elect timeout is 30 seconds, so we enlarge thrift_rpc_timeout_ms
and txn_commit_rpc_timeout_ms to 60s.

BTW: enlarge bdbje_lock_timeout_second from 1 to 5.
2023-07-27 11:12:26 +08:00
619a2857e1 [improvement](jdbc catalog) improve mysql jdbc catalog read bytea`s types & else improve (#22233) 2023-07-27 10:18:37 +08:00
052a416d49 [Enhencement](binlog) db enable binlog (#22256)
* Improve db update binlog properties (binlog.enable = "true") with check
all table enable binlog

* Add more test_alter_database_property regression test
2023-07-27 10:03:51 +08:00
341c45974c [round](decimalv2) round precise decimalv2 value (#22258) 2023-07-27 10:00:36 +08:00
163a38a527 [opt](Nereids) support sql cache (#22144)
1. let Nereids support sql cache
2. let legacy planner's sql cache supports union all
2023-07-27 09:57:31 +08:00
82fe78ce84 Update table binlog config disable failure when db binlog is enable && (#22253)
modify table binlog more than one property

Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-07-27 09:54:24 +08:00
8fb28ecc9e [test](partial-update) add some cases for partial-update (#22210) 2023-07-27 09:52:40 +08:00
dcd6844ea5 [improvement](regression-test) add partial update with schema change case (#22213) 2023-07-27 09:51:42 +08:00
fb41265c27 [opt](Nereids) add boolean type signature for sum aggregate function (#21959) 2023-07-27 09:41:19 +08:00
12222eb145 forbid: regression-test/pipeline/p0/conf/regression-conf.groovy (#22271) 2023-07-26 23:27:18 +08:00
7ed997cba8 [improvement](s3) increase the connection num of s3 client (#22049)
The default maxConnection of s3 client is 25.
It should be increased to improve the query performance.

In my test, a tpch 300 benchmark with data stored on object storage, the total time
can reduce from 430s -> 330s
2023-07-26 22:52:40 +08:00
Pxl
560731f392 [Bug](runtime-filter) fix probe expr prepared twice on minmax runtime filter (#22229)
fix probe expr prepared twice on minmax runtime filter
2023-07-26 19:44:35 +08:00