Commit Graph

12184 Commits

Author SHA1 Message Date
05abfbc5ef [improvement](regression-test) add compression algorithm regression test (#22303) 2023-07-28 17:28:52 +08:00
3eeca7ee55 [enhance](regresstion case)add external group mark 0727 (#22287)
* add external group mark 0727

* add external pipeline regression conf
 0727

* update pipeline regression config 0727

* open es config from docker 0727
2023-07-28 17:11:19 +08:00
ef218d79da [fix](case) add sync after stream load (#22232)
add sync after stream load
2023-07-28 17:05:20 +08:00
ec1a4d172b (vertical compaction) fix vertical compaction core (#22275)
* (vertical compaction) fix vertical compaction core
co-author:@zhannngchen
2023-07-28 16:41:00 +08:00
25f26198f4 [fix](executor) only mysql connect to set GlobalPipelineTask (#22205) 2023-07-28 16:19:34 +08:00
5a0ad09856 [fix](nereids) SubqueryToApply may lost conjunct (#22262)
consider sql:
```
SELECT *
        FROM sub_query_correlated_subquery1 t1
        WHERE coalesce(bitand( 
        cast(
            (SELECT sum(k1)
            FROM sub_query_correlated_subquery3 ) AS int), 
            cast(t1.k1 AS int)), 
            coalesce(t1.k1, t1.k2)) is NULL
        ORDER BY  t1.k1, t1.k2;
```
is Null conjunct is lost in SubqueryToApply rule. This pr fix it
2023-07-28 15:08:56 +08:00
80673406b1 [fix](Nereids) project hidden columns when show_hidden_columns is true (#22285) 2023-07-28 15:08:18 +08:00
0c734a861e [Enhancement](delete) eliminate reading the old values of non-key columns for delete stmt (#22270) 2023-07-28 14:37:33 +08:00
9f565cf835 [fix](ut) fix ut of stats test #22325
After auto retry merged, it's hard to determine the execute times of doExecute method in compile time, and if the expected execute times in the expectation block is missed, unexpected invocation  exception would be thrown, so just remove the expected execute times
2023-07-28 14:23:35 +08:00
c2155678ca [fix](functions) fix now(null) crash (#22321)
before: BE crash
now:

mysql [test]>select now(null);
+-----------+
| now(NULL) |
+-----------+
| NULL      |
+-----------+
1 row in set (0.06 sec)
2023-07-28 14:07:56 +08:00
1c6246f7ee [improve](agg) support distinct agg node (#22169)
select c_name from customer union select c_name from customer
this sql used agg node to get distinct row of c_name,
so it's no need to wait for inserted all data to hash map,
could output the data which it's inserted into hash map successed.
2023-07-28 13:54:10 +08:00
ad080c691f [chore](log)Move non-user-friendly error message to be.WARNING (#22315)
Move non-user-friendly error message to be.WARNING
2023-07-28 13:15:25 +08:00
2d538d5959 [regression](fix) include case test_round (#22326)
include case test_round
2023-07-28 13:10:54 +08:00
7be349a10b [opt](inverted index) add session variable enable_inverted_index_query to control whether query with inverted index (#22255) 2023-07-28 12:43:26 +08:00
5da5fac37a [refactor](Nereids) add result sink node (#22254)
use ResultSink as query root node to let plan of query statement
has the same pattern with insert statement
2023-07-28 11:31:09 +08:00
697745bb58 [chore](workflow) Increase the build space for building third-party libraries (#22322)
Increase the build space for building third-party libraries
2023-07-28 11:27:51 +08:00
313d4fd257 [doc](fix) Improve the document example of alter table replace (#21958)
Improve the document example of alter table replace
The old doc example was wrong
2023-07-28 10:21:47 +08:00
e87174dd6b [feature](planner) modify multi partition prefix value (#22098)
modify multi partition prefix value: 'p_'
2023-07-28 10:21:32 +08:00
adc44d9f46 [regression-test] add list partition case and multi partition keys case (#22042)
* [regression-test] add list partition case and multi partition keys case

* fix delete failed
2023-07-28 10:12:35 +08:00
0d7d9b92db [fix](multi-catalog) complex types parsing failed, with unexpected nulls and rows (#22228)
Fix tow bugs:
1. Unexpected null values in array column. If 65535 consecutive values are not null in nullable array column, this error will be triggered. The reason is that the array parser did not handle boundary conditions.
2. The number of rows of key filed, and that of value field in map column are not equal. Similarly, the number of rows among fields in struct column are not the same. This would be triggered when the number of rows are not equal among parquet pages of different columns in a row group.
2023-07-28 10:03:08 +08:00
7d5d416b25 [Fix](EsCatalog) fix be core when query the table of Es catalog with null fields (#22279) 2023-07-28 09:53:55 +08:00
bfa7f8df6d [fix](Nereids) parse logical binary stack overflow (#22308)
1. not use recursive parse to avoid stack overflow
2. To create a balanced tree instead of left deep tree
TODO: add expr_depth_limit to Nereids' parser
2023-07-28 09:48:17 +08:00
f0f3548dfe [regression](multi-catalog)add EMR cloud env test tools (#21788)
add emr test tools for aliyun, huawei cloud, tencent cloud.
2023-07-28 09:45:10 +08:00
8caa5a9ba4 [Fix](mutli-catalog) Fix null partitions error in iceberg tables. (#22185)
### Issue
when partition has null partitions, it throws error
`Failed to fill partition column: t_int=null`

### Resolution
- Fix the following null partitions error in iceberg tables by replacing null partition to '\N'.
- Add regression test for hive null partition.
2023-07-27 23:57:35 +08:00
00863f25e9 [improvement](profile) add table name for file scan node (#22299)
```
VFILE_SCAN_NODE(region)  (id=0):(Active:  3.537us,  %  non-child:  0.00%)
                                -  RuntimeFilters:  :  
                              -  UseSpecificThreadToken:  False
                              -  AcquireRuntimeFilterTime:  501ns
                              -  AllocateResourceTime:  105.598us
```
2023-07-27 23:54:31 +08:00
442ae632e3 [fix](fs-cache) add 'scheme://authority' to fs cache key (#22263)
This file system cache key should contains `scheme://authority`, eg: `hdfs//nameservices1`.
Or it will encounter error:

```
Wrong FS: hdfs//abc/xxxx, expected: hdfs://def
```
2023-07-27 23:53:54 +08:00
f7d5453be8 [fix](nereids) fix cte bucket shuffle path (#22311) 2023-07-27 22:44:51 +08:00
b5fa29e138 [fix](bitmap) incorrect result of function 'bitmap_from_array' (#22305) 2023-07-27 22:44:06 +08:00
5584d7a5ba [Improve](point query) Improve lookup connection cache from DoubleBuffer to LRU cache for better item pruning (#22041) 2023-07-27 22:22:50 +08:00
8371171e44 [Feature](inverted index) add inverted index tool (#22207) 2023-07-27 21:28:34 +08:00
461c4dfaae [fix](tablet clone) fix single replica load failed during migration (#22077) 2023-07-27 20:38:03 +08:00
e39d234db9 [opt](inverted index) add more check for create inverted index (#22297) 2023-07-27 20:33:24 +08:00
716d58f5ff [fix](Nereids) decimal divide should not return null if numerator is zero (#22309) 2023-07-27 20:23:04 +08:00
687d97e648 [improvement][default_config] enlarge default value compaction related (#22286)
configs

1. Because vertical compaction is enabled by default, it consumes less
memory, we can enlarge default value of compaction related configs.
2. Enlarge default value of shard size related to lock.
2023-07-27 20:17:43 +08:00
816fd50d1d [Enhancement](binlog) Add binlog enable diable check in BinlogManager (#22173)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-07-27 20:16:21 +08:00
a87d34b19b [Fix](multi catalog statistics)Improve external table statistics collection (#22224)
Improve external table statistics collection, including log, observability and fix some bugs.
1. Add Running state for statistics job.
2. Add progress for show analyze job. (n/m tasks finished, n/m task failed and so on)
3. Add analyze time cost for show analyze task.
4. Make task failure message more clear.
5. Synchronize the job status updating code in updateTaskStatus.
6. Fix NPE in HMSAnalyzeTask. (Avoid refreshing statistics cache if the collection sql failed)
7. Return error message for with sync collection while timeout. 
8. Log level improvement
9. Fix misuse of logCreateAnalysisJob for tasks.
2023-07-27 20:01:14 +08:00
391131c7fd [typo](docs)correct describe for grant (#20910) 2023-07-27 20:00:01 +08:00
2c849c619d [fix](nereids) only allow inner join in dphyper join reorder (#22307)
current dphyper join reorder hasn't consider the join conjunct referencing only one side of the child. This is common case in outer join conjunct. So we need disable outer join reorder in dphyper until this problem is addressed.
2023-07-27 19:46:37 +08:00
8a03766c58 [fix](test) change some name in regression test to avoid conflict name running in parallel (#22273) 2023-07-27 19:38:01 +08:00
ae5e39ad26 [opt](Nereids) add double signature back for round like function (#22284)
add double signature back for round like function
2023-07-27 19:10:43 +08:00
d0c369d61b [fix](vec) Arena was not initialized in PartitionMethodSerialized (#22295) 2023-07-27 18:55:57 +08:00
Pxl
87b9425772 [Bug](materialized-view) fix where clause not analyzed after fe restart (#22268)
fix where clause not analyzed after fe restart
2023-07-27 18:34:44 +08:00
b51fcbd9c7 [opt](stats) Scale replica of stats table to 3 when it's possible (#22227)
So that we could improve the availability of stats.
2023-07-27 17:36:54 +08:00
Pxl
05d18b2f68 [Chore](regression-test) remove insert into select on enable nereids dml (#22291)
remove insert into select on enable nereids dml
2023-07-27 17:18:49 +08:00
e29c703ce4 [typo](docs) fix some typo (#21031) 2023-07-27 16:55:48 +08:00
6f1c03c766 [fix](jdbc_catalog) fix int and bigint in mysql view when use doris catalog (#22251) 2023-07-27 16:50:42 +08:00
94d2ba5ba0 [Docs](community) Add Join Community Files of EN & CN Verison (#22252) 2023-07-27 16:47:10 +08:00
b597eef436 [fix](doc)description of check_rpc_channel api in http-actions-be (#21568) 2023-07-27 16:09:20 +08:00
0512e0b168 [test](regression) add cases for partial update with sequence_type (#22215) 2023-07-27 15:51:01 +08:00
4f6a3c5bf0 [feature](catalog) support clob type in oracle jdbc catalog (#21532) 2023-07-27 15:49:15 +08:00