database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
8,410 Commits 1 Branch 0 Tags
fdc042bb397661d8acc0447563add0fac5e9cf3e
Commit Graph

8410 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
chenlinzhong
fdc042bb39 [fix](vresultsink) BufferControlBlock may block all fragment handle threads (#16231) 
BufferControlBlock may block all fragment handle threads leads to be out of work

modify include:

BufferControlBlock cancel after max timeout
StmtExcutor notify be to cancel the fragment when unexcepted occur
more details see issue #16203
 2023-01-30 16:53:21 +08:00
谢健
342f3168b5 [fix](Nereids) return null when encryption is invalid rather than throw exception (#16234) 2023-01-30 16:52:42 +08:00
yiguolei
90b12143a3 [refactor](remove unused code) remove runtime tuple structure and useless utils class (#16237) 2023-01-30 16:45:14 +08:00
Xiangyu Wang
7c3c5576b9 [Fix](doc) fix hive catalog docs with kerberos enabled. (#16236) 
* [Fix](doc) fix hive catalog docs with kerberos enabled.

* [Fix](doc) fix hive catalog docs with kerberos enabled.

---------

Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
 2023-01-30 16:44:27 +08:00
Jerry Hu
a9671b6dfd [feature](agg)support two level-hash map in aggregation node (#15967) 2023-01-30 16:43:33 +08:00
shee
6bebf92254 [fix][FE] fix be coredump when children of FunctionCallExpr is folded (#16064) 
Co-authored-by: shizhiqiang03 <shizhiqiang03@meituan.com>
fix be coredump when children of FunctionCallExpr is folded
 2023-01-30 15:25:00 +08:00
yiguolei
4b6a4b3cf7 [refactor](remove unused code) Remove unused mempool declare or function params (#16222) 
* Remove unused mempool declare or function params

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-01-30 13:03:18 +08:00
Coen
f87f80a9ef [typo](doc) fix spelling mistaks word uninstall (#16219) 2023-01-30 12:32:53 +08:00
mch_ucchi
6254258357 [Feature](Nereids) eliminate project when child is empty relation (#16210) 2023-01-30 11:28:07 +08:00
jakevin
60b622cefe [minor](git): ignore metastore_db/ (#16201) 2023-01-30 11:13:00 +08:00
morrySnow
5c00caa259 [refactor](Nereids) refactor BindSlotReference for easy merge all bind process in one rule (#16156) 2023-01-30 10:57:39 +08:00
jakevin
bd1b7e190c [fix](Nereids): fix field(). (#16214) 2023-01-30 10:55:02 +08:00
Ashin Gau
07d58e531a [improvement](filecache) add profile for file cache (#16223) 2023-01-30 10:46:31 +08:00
Kai Qiang Wu
28fcc093a8 [improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost (#15972) 
As AVX512 is available in most modern processors, it is good to use them if have performance boost.
In latest bitshuffle, AVX512 have been added. We could make it integrated in doris for AVX512 case.

Tested with master branch, queries(SSB query q1.1.sql~q4.3.sql total 13 queries) can be boost from 1.4%~3.2%. (use run-ssb-queries.sh 5 times, each time with 100 iterations.)

Signed-off-by: Wu, Kaiqiang <kaiqiang.wu@intel.com>
Co-authored-by: vesslanjin <jun.i.jin@intel.com>
 2023-01-30 10:33:01 +08:00
Zhengguo Yang
ec4a56922f [enhancement](memory) reduce memory usage for failed broker loads (#15895) 
* [enhancement](memory) reduce memory usage for failed  broker loads
 2023-01-30 10:22:31 +08:00
WenYao
69e748b076 [fix](schema scanner)change schema_scanner::get_next_row to get_next_block (#15718) 2023-01-30 10:01:50 +08:00
HappenLee
7d437d5706 [Bug](function) running_difference function coredump in regression test (#16215) 2023-01-30 09:58:27 +08:00
Xinyi Zou
17885acd09 [improvement](metrics) Metrics add all rowset nums and segment nums (#16208) 2023-01-30 09:55:32 +08:00
Gabriel
f72efd4523 [Improvement](decimal) do not log fatal when precision is invalid (#16207) 2023-01-30 09:54:22 +08:00
Xiangyu Wang
b8a7297109 [Enhancement](profile) fill user field for profile. (#16212) 
Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
 2023-01-30 09:15:02 +08:00
fracly
9894d239b7 [typo](doc)Add an example of how to assign a value to a field of Date type (#16199) 
* 添加Date类型字段赋值示例，Date类型字段需要使用Epoch Day

* Update flink-doris-connector.md
 2023-01-30 08:22:52 +08:00
Adonis Ling
d56043ab5a [feature-wip](MTMV) Support setting variables in query statement (#16060) 
## Use case

```shell
mysql> CREATE TABLE t_user (
    ->   event_day DATE,
    ->   id bigint,
    ->   username varchar(20)
    -> )
    -> DISTRIBUTED BY HASH(id) BUCKETS 10
    -> PROPERTIES ('replication_num' = '1');
Query OK, 0 rows affected (0.07 sec)

mysql> CREATE TABLE t_user_pv(
    ->   event_day DATE,
    ->   id bigint,
    ->   pv bigint
    -> )
    -> DISTRIBUTED BY HASH(id) BUCKETS 10
    -> PROPERTIES ('replication_num' = '1');
Query OK, 0 rows affected (0.09 sec)

mysql> CREATE MATERIALIZED VIEW mv
    -> BUILD IMMEDIATE REFRESH COMPLETE
    -> KEY (username)
    -> DISTRIBUTED BY HASH(username) BUCKETS 10
    -> PROPERTIES ('replication_num' = '1')
    -> AS SELECT /*+ SET_VAR(exec_mem_limit=1048576, query_timeout=3600) */ t1.username ,t2.pv FROM t_user t1 LEFT JOIN t_user_pv t2 on t1.id = t2.id;
Query OK, 0 rows affected (0.10 sec)
```
 2023-01-30 01:05:41 +08:00
HappenLee
f0ab71aa84 [Thirdpart](lib) Add lib fast_float to replace std::from_chars() convert float/double (#16204) 2023-01-30 00:41:56 +08:00
谢健
98649ec9f8 [fix](Nereids): Fix some functions error (#16197) 
* fix bugs in regexp_extract_all

* fix rpad

* fix weekofday

* fix cryptor

* fix timestamp

* fix st_ function
 2023-01-30 00:41:31 +08:00
jakevin
7d648a94d0 [fix](Nereids): fix scalar_function A-F. (#16209) 
* [fix](Nereids): fix scalar_function A-F.

* [Fix](regression-test)fix regression test framework cannot compare double value nan and inf.

* revert dround()
 2023-01-30 00:37:34 +08:00
luozenglin
62ff20d462 [fix](compile) fix the compile error when WITH_MYSQL is ON. (#16195) 2023-01-29 22:52:44 +08:00
yiguolei
5eaa995704 [refactor](some mempool) not memset 0 in default value iterator (#16194) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-01-29 22:50:39 +08:00
Gabriel
217db3e4c8 [refactor](built-in function) remove symbols for vectorized function (#16189) 
* [refactor](built-in function) remove symbols for vectorized function

* update

* update
 2023-01-29 21:30:09 +08:00
YueW
a38f36365b [optimize](inverted index) use InvertedIndexReaderType::FULLTEXT instead of InvertedIndexParserType to check predicate whether handle by fulltext or not (#16191) 2023-01-29 20:45:51 +08:00
mch_ucchi
1db7882bb5 [Fix](Nereids): fix error of X-Z function for nereids (#16171) 2023-01-29 20:42:30 +08:00
starocean999
1ec88cbff6 [fix](nereids) AggregationNode process null as key column in wrong way (#16125) 
in AggregationNode, _merge_with_serialized_key_helper method should convert the key column to full column if the key column is null literal.
 2023-01-29 20:12:07 +08:00
Pxl
46347a51d2 [Bug](exec) enable warning on ignoring function return value for vctx (#16157) 
* enable warning on ignoring function return value for vctx
 2023-01-29 17:23:21 +08:00
plat1ko
f97c51d786 [enhancement](compaction) Optimize calculating level size of input rowset in SizeBasedCumulativeCompactionPolicy (#15633) 
* Optimize calculating level size of input rowset in SizeBasedCumulativeCompactionPolicy
 2023-01-29 17:18:50 +08:00
morrySnow
1ad6ef939b [refactor](Nereids) use immutable collections as far as possible (#16193) 2023-01-29 16:48:58 +08:00
jakevin
04ed83cb36 [fix](Nereids): remove DataV2Type in ConvertTz SIGNATURES (#16170) 
* [fix](Nereids): remove `DataV2Type` in ConvertTz SIGNATURES

* remove it in doris_builtins_functions.py
 2023-01-29 16:11:17 +08:00
jakevin
b11d056fe5 [fix](Nereids): fix function name. (#16188) 2023-01-29 15:21:24 +08:00
jakevin
abc50c6fe5 [enhance](Nereids): remove duplicated alias Function. (#16187) 2023-01-29 14:56:20 +08:00
huangzhaowei
c6bc0a03a4 [feature](Load)Suppot MySQL Load Data (#15511) 
Main subtask of [DSIP-28](https://cwiki.apache.org/confluence/display/DORIS/DSIP-028%3A+Suppot+MySQL+Load+Data)

## Problem summary
Support mysql load syntax as below: 
```sql
LOAD DATA
    [LOCAL]
    INFILE 'file_name'
    INTO TABLE tbl_name
    [PARTITION (partition_name [, partition_name] ...)]
    [COLUMNS TERMINATED BY 'string']
    [LINES TERMINATED BY 'string']
    [IGNORE number {LINES | ROWS}]
    [(col_name_or_user_var [, col_name_or_user_var] ...)]
    [SET (col_name={expr | DEFAULT} [, col_name={expr | DEFAULT}] ...)]
    [PROPERTIES (key1 = value1 [, key2=value2]) ]
```

For example, 
```sql
            LOAD DATA 
            LOCAL
            INFILE 'local_test.file'
            INTO TABLE db1.table1
            PARTITION (partition_a, partition_b, partition_c, partition_d)
            COLUMNS TERMINATED BY '\t'
            (k1, k2, v2, v10, v11)
            set (c1=k1,c2=k2,c3=v10,c4=v11)
            PROPERTIES ("auth" = "root:", "strict_mode"="true")
```

Note that in this pr the property named `auth` must be set since stream load need auth. I will optimize it later.
 2023-01-29 14:44:59 +08:00
Mingyu Chen
c9f66250a8 [docker](iceberg) add iceberg docker compose and modify scripts (#16175) 
Add iceberg docker compose
Rename start-thirdparties-docker.sh to run-thirdparties-docker.sh and support start to stop specified components.
 2023-01-29 14:31:27 +08:00
abmdocrt
eb7da1c0ee [fix](datatype) fix some bugs about data type array datetimev2 and decimalv3 (#16132) 2023-01-29 14:26:08 +08:00
yiguolei
3235b636cc [refactor](remove unused code) remove thread pool manager (#16179) 
* remove thread resource manager

* remove  string buffer
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-01-29 13:03:08 +08:00
lihangyu
578a855b3e [Bug](topn-opt) filter condition for analytic info for two phase read opt (#16173) 
two phase read optimization should not be enabled when query has analytic info
 2023-01-29 12:06:18 +08:00
jakevin
ce487e2b11 [fix](Nereids): fix dceil() dfloor() (#16174) 2023-01-29 11:59:23 +08:00
Jibing-Li
35398ad8d9 [fix](multi-catalog)Use -1 for column_statistics internal table idx_id default value instead of null, for external catalog (#16177) 
The internal statistic table column_statistics has a non-null field idx_id, the insert sql for hive table set the default value to NULL, which will failed to insert the result. Change it to -1.
 2023-01-29 11:29:25 +08:00
Pxl
2b5f95f08a [Bug](function) remove datev2 signature of hour_ceil/hour_floor #16168 2023-01-29 11:27:56 +08:00
Xinyi Zou
e9afd3210c [improvement](memory) Optimize the log of process memory insufficient and support regular GC cache (#16084) 
1. When the process memory is insufficient, print the process memory statistics in a more timely and detailed manner.
2. Support regular GC cache, currently only page cache and chunk allocator are included, because many people reported that the memory does not drop after the query ends.
3. Reduce system available memory warning water mark to reduce memory waste
4. Optimize soft mem limit logging
 2023-01-29 10:02:04 +08:00
Hu Yanjun
46ce66cbd8 [docs](multi-catalog)update en docs (#16160) 2023-01-29 00:36:31 +08:00
mch_ucchi
b7379daffa [test](Nereids) changing test data for more effectively testing for nereids_function_p0 (#16163) 2023-01-28 21:23:40 +08:00
jakevin
3151d94e9e [fix](Nereids): fix Ceiling. (#16164) 2023-01-28 20:26:20 +08:00
yixiutt
8ca3aa85e2 [bugfix](compaction) fix segment writer finalize footer twice (#16145) 2023-01-28 20:15:52 +08:00