database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
13,721 Commits 1 Branch 0 Tags
b38b8b4494de38b43c1c6c19a39de23538e12b22
Commit Graph

13721 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mingyu Chen
f02aa7e3e2 [thirdparty](libhdfs3) update libhdfs3 to 2.3.7 to fix kms seek bug (#16265) 2023-01-31 08:56:00 +08:00
AKIRA
e7c1d81419 [fix](planner) Pushdown constant predicate to all scan node in the lieteral view. #16217 
Before this PR, planner might push a constant FALSE predicate to the wrong scan nodes in the literal view, and make this predicate useless
 2023-01-30 22:18:43 +08:00
HappenLee
1020fe165b [Bug](function) positive function coredump in decimal (#16230) 2023-01-30 22:17:50 +08:00
YueW
75c8670286 [Feature-WIP](inverted index) Filter out and remain predicates that do not support apply by inverted index, and add inverted index regression case (#16167) 
1. Filter out and remain predicates that do not support applying on inverted index,
    like `BF` predicate, `IS_NULL` predicate, `IS_NOT_NULL` predicate.
2. Add inverted index regression case that based on tpcds_sf1 data set.
 2023-01-30 22:16:08 +08:00
yongkang.zhong
8b2aa9f3a2 fix date_sub function doc (#16250) 2023-01-30 21:53:45 +08:00
Jerry Hu
9aa0d86fec [fix](olap) Incorrect reserving size for PredicateColumn converted from ColumnDictionary (#16249) 2023-01-30 20:28:22 +08:00
fengzhengshu
53e50d2b05 [typo](doc): fix SELECT.md (#16245) 2023-01-30 19:39:30 +08:00
jakevin
c6014975a4 [fix](Nereids): fix AesDecrypt, AesEncrypt. (#16213) 2023-01-30 18:52:56 +08:00
AKIRA
b480db2e11 [test](pipeline) Run nereids cases in p1/p2 (#16130) 2023-01-30 18:33:31 +08:00
Pxl
322dc2a104 [Bug](function) fix now(int) use_default_implementation_for_nulls && fix dround signature (#16238) 2023-01-30 18:01:26 +08:00
yiguolei
c59a8cb15d [refactor](remove unused code) remove log error hub (#16183) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-01-30 16:53:56 +08:00
chenlinzhong
fdc042bb39 [fix](vresultsink) BufferControlBlock may block all fragment handle threads (#16231) 
BufferControlBlock may block all fragment handle threads leads to be out of work

modify include:

BufferControlBlock cancel after max timeout
StmtExcutor notify be to cancel the fragment when unexcepted occur
more details see issue #16203
 2023-01-30 16:53:21 +08:00
谢健
342f3168b5 [fix](Nereids) return null when encryption is invalid rather than throw exception (#16234) 2023-01-30 16:52:42 +08:00
yiguolei
90b12143a3 [refactor](remove unused code) remove runtime tuple structure and useless utils class (#16237) 2023-01-30 16:45:14 +08:00
Xiangyu Wang
7c3c5576b9 [Fix](doc) fix hive catalog docs with kerberos enabled. (#16236) 
* [Fix](doc) fix hive catalog docs with kerberos enabled.

* [Fix](doc) fix hive catalog docs with kerberos enabled.

---------

Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
 2023-01-30 16:44:27 +08:00
Jerry Hu
a9671b6dfd [feature](agg)support two level-hash map in aggregation node (#15967) 2023-01-30 16:43:33 +08:00
shee
6bebf92254 [fix][FE] fix be coredump when children of FunctionCallExpr is folded (#16064) 
Co-authored-by: shizhiqiang03 <shizhiqiang03@meituan.com>
fix be coredump when children of FunctionCallExpr is folded
 2023-01-30 15:25:00 +08:00
yiguolei
4b6a4b3cf7 [refactor](remove unused code) Remove unused mempool declare or function params (#16222) 
* Remove unused mempool declare or function params

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-01-30 13:03:18 +08:00
Coen
f87f80a9ef [typo](doc) fix spelling mistaks word uninstall (#16219) 2023-01-30 12:32:53 +08:00
mch_ucchi
6254258357 [Feature](Nereids) eliminate project when child is empty relation (#16210) 2023-01-30 11:28:07 +08:00
jakevin
60b622cefe [minor](git): ignore metastore_db/ (#16201) 2023-01-30 11:13:00 +08:00
morrySnow
5c00caa259 [refactor](Nereids) refactor BindSlotReference for easy merge all bind process in one rule (#16156) 2023-01-30 10:57:39 +08:00
jakevin
bd1b7e190c [fix](Nereids): fix field(). (#16214) 2023-01-30 10:55:02 +08:00
Ashin Gau
07d58e531a [improvement](filecache) add profile for file cache (#16223) 2023-01-30 10:46:31 +08:00
Kai Qiang Wu
28fcc093a8 [improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost (#15972) 
As AVX512 is available in most modern processors, it is good to use them if have performance boost.
In latest bitshuffle, AVX512 have been added. We could make it integrated in doris for AVX512 case.

Tested with master branch, queries(SSB query q1.1.sql~q4.3.sql total 13 queries) can be boost from 1.4%~3.2%. (use run-ssb-queries.sh 5 times, each time with 100 iterations.)

Signed-off-by: Wu, Kaiqiang <kaiqiang.wu@intel.com>
Co-authored-by: vesslanjin <jun.i.jin@intel.com>
 2023-01-30 10:33:01 +08:00
Zhengguo Yang
ec4a56922f [enhancement](memory) reduce memory usage for failed broker loads (#15895) 
* [enhancement](memory) reduce memory usage for failed  broker loads
 2023-01-30 10:22:31 +08:00
WenYao
69e748b076 [fix](schema scanner)change schema_scanner::get_next_row to get_next_block (#15718) 2023-01-30 10:01:50 +08:00
HappenLee
7d437d5706 [Bug](function) running_difference function coredump in regression test (#16215) 2023-01-30 09:58:27 +08:00
Xinyi Zou
17885acd09 [improvement](metrics) Metrics add all rowset nums and segment nums (#16208) 2023-01-30 09:55:32 +08:00
Gabriel
f72efd4523 [Improvement](decimal) do not log fatal when precision is invalid (#16207) 2023-01-30 09:54:22 +08:00
Xiangyu Wang
b8a7297109 [Enhancement](profile) fill user field for profile. (#16212) 
Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
 2023-01-30 09:15:02 +08:00
fracly
9894d239b7 [typo](doc)Add an example of how to assign a value to a field of Date type (#16199) 
* 添加Date类型字段赋值示例，Date类型字段需要使用Epoch Day

* Update flink-doris-connector.md
 2023-01-30 08:22:52 +08:00
Adonis Ling
d56043ab5a [feature-wip](MTMV) Support setting variables in query statement (#16060) 
## Use case

```shell
mysql> CREATE TABLE t_user (
    ->   event_day DATE,
    ->   id bigint,
    ->   username varchar(20)
    -> )
    -> DISTRIBUTED BY HASH(id) BUCKETS 10
    -> PROPERTIES ('replication_num' = '1');
Query OK, 0 rows affected (0.07 sec)

mysql> CREATE TABLE t_user_pv(
    ->   event_day DATE,
    ->   id bigint,
    ->   pv bigint
    -> )
    -> DISTRIBUTED BY HASH(id) BUCKETS 10
    -> PROPERTIES ('replication_num' = '1');
Query OK, 0 rows affected (0.09 sec)

mysql> CREATE MATERIALIZED VIEW mv
    -> BUILD IMMEDIATE REFRESH COMPLETE
    -> KEY (username)
    -> DISTRIBUTED BY HASH(username) BUCKETS 10
    -> PROPERTIES ('replication_num' = '1')
    -> AS SELECT /*+ SET_VAR(exec_mem_limit=1048576, query_timeout=3600) */ t1.username ,t2.pv FROM t_user t1 LEFT JOIN t_user_pv t2 on t1.id = t2.id;
Query OK, 0 rows affected (0.10 sec)
```
 2023-01-30 01:05:41 +08:00
HappenLee
f0ab71aa84 [Thirdpart](lib) Add lib fast_float to replace std::from_chars() convert float/double (#16204) 2023-01-30 00:41:56 +08:00
谢健
98649ec9f8 [fix](Nereids): Fix some functions error (#16197) 
* fix bugs in regexp_extract_all

* fix rpad

* fix weekofday

* fix cryptor

* fix timestamp

* fix st_ function
 2023-01-30 00:41:31 +08:00
jakevin
7d648a94d0 [fix](Nereids): fix scalar_function A-F. (#16209) 
* [fix](Nereids): fix scalar_function A-F.

* [Fix](regression-test)fix regression test framework cannot compare double value nan and inf.

* revert dround()
 2023-01-30 00:37:34 +08:00
luozenglin
62ff20d462 [fix](compile) fix the compile error when WITH_MYSQL is ON. (#16195) 2023-01-29 22:52:44 +08:00
yiguolei
5eaa995704 [refactor](some mempool) not memset 0 in default value iterator (#16194) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-01-29 22:50:39 +08:00
Gabriel
217db3e4c8 [refactor](built-in function) remove symbols for vectorized function (#16189) 
* [refactor](built-in function) remove symbols for vectorized function

* update

* update
 2023-01-29 21:30:09 +08:00
YueW
a38f36365b [optimize](inverted index) use InvertedIndexReaderType::FULLTEXT instead of InvertedIndexParserType to check predicate whether handle by fulltext or not (#16191) 2023-01-29 20:45:51 +08:00
mch_ucchi
1db7882bb5 [Fix](Nereids): fix error of X-Z function for nereids (#16171) 2023-01-29 20:42:30 +08:00
starocean999
1ec88cbff6 [fix](nereids) AggregationNode process null as key column in wrong way (#16125) 
in AggregationNode, _merge_with_serialized_key_helper method should convert the key column to full column if the key column is null literal.
 2023-01-29 20:12:07 +08:00
Pxl
46347a51d2 [Bug](exec) enable warning on ignoring function return value for vctx (#16157) 
* enable warning on ignoring function return value for vctx
 2023-01-29 17:23:21 +08:00
plat1ko
f97c51d786 [enhancement](compaction) Optimize calculating level size of input rowset in SizeBasedCumulativeCompactionPolicy (#15633) 
* Optimize calculating level size of input rowset in SizeBasedCumulativeCompactionPolicy
 2023-01-29 17:18:50 +08:00
morrySnow
1ad6ef939b [refactor](Nereids) use immutable collections as far as possible (#16193) 2023-01-29 16:48:58 +08:00
jakevin
04ed83cb36 [fix](Nereids): remove DataV2Type in ConvertTz SIGNATURES (#16170) 
* [fix](Nereids): remove `DataV2Type` in ConvertTz SIGNATURES

* remove it in doris_builtins_functions.py
 2023-01-29 16:11:17 +08:00
jakevin
b11d056fe5 [fix](Nereids): fix function name. (#16188) 2023-01-29 15:21:24 +08:00
jakevin
abc50c6fe5 [enhance](Nereids): remove duplicated alias Function. (#16187) 2023-01-29 14:56:20 +08:00
huangzhaowei
c6bc0a03a4 [feature](Load)Suppot MySQL Load Data (#15511) 
Main subtask of [DSIP-28](https://cwiki.apache.org/confluence/display/DORIS/DSIP-028%3A+Suppot+MySQL+Load+Data)

## Problem summary
Support mysql load syntax as below: 
```sql
LOAD DATA
    [LOCAL]
    INFILE 'file_name'
    INTO TABLE tbl_name
    [PARTITION (partition_name [, partition_name] ...)]
    [COLUMNS TERMINATED BY 'string']
    [LINES TERMINATED BY 'string']
    [IGNORE number {LINES | ROWS}]
    [(col_name_or_user_var [, col_name_or_user_var] ...)]
    [SET (col_name={expr | DEFAULT} [, col_name={expr | DEFAULT}] ...)]
    [PROPERTIES (key1 = value1 [, key2=value2]) ]
```

For example, 
```sql
            LOAD DATA 
            LOCAL
            INFILE 'local_test.file'
            INTO TABLE db1.table1
            PARTITION (partition_a, partition_b, partition_c, partition_d)
            COLUMNS TERMINATED BY '\t'
            (k1, k2, v2, v10, v11)
            set (c1=k1,c2=k2,c3=v10,c4=v11)
            PROPERTIES ("auth" = "root:", "strict_mode"="true")
```

Note that in this pr the property named `auth` must be set since stream load need auth. I will optimize it later.
 2023-01-29 14:44:59 +08:00
Mingyu Chen
c9f66250a8 [docker](iceberg) add iceberg docker compose and modify scripts (#16175) 
Add iceberg docker compose
Rename start-thirdparties-docker.sh to run-thirdparties-docker.sh and support start to stop specified components.
 2023-01-29 14:31:27 +08:00