database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
4,735 Commits 1 Branch 0 Tags
4df1106e1e73880fedbfbf2279dcfc7daccbc4ef
Commit Graph

4735 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
smallhibiscus
4df1106e1e [improvement](script) Add jvm parameters and the process will automatically stop when oom occurs in fe. (#9765) 2022-05-30 09:44:12 +08:00
dujl
8092439634 [feature](hudi) Step2: Support query hudi external table(include cow and mor table) (#9752) 
support query cow and mor hudi table.
 2022-05-30 09:43:36 +08:00
Jing Shen
7b98dd438d [feature](function) Add nvl function (#9726) 2022-05-30 09:43:00 +08:00
EmmyMiao87
0683181fef [API changed](parser) Remove merge join syntax (#9795) 
Remove merge join sql and merge join node
 2022-05-30 09:04:21 +08:00
smallhibiscus
080c70f32d [regression test]Add regression test case for multiple column delete. (#9817) 
Co-authored-by: smallhibiscus <844981280>
 2022-05-30 09:01:25 +08:00
smallhibiscus
88db94c220 [doc]Fix typo in create-table documentation. (#9834) 
Co-authored-by: smallhibiscus <844981280>
 2022-05-30 08:28:15 +08:00
Gabriel
a96b41db7a [Improvement] Simplify expressions for _vconjunct_ctx_ptr (#9816) 2022-05-29 23:05:21 +08:00
Amos Bird
63aab5ee5d [Bugfix(Vec)] Fix some memory leak issues (#9824) 2022-05-29 23:04:11 +08:00
xueweizhang
c5369d3220 [bugfix] Fix create table like when having hidden columns (#9694) 2022-05-29 18:02:16 +08:00
morrySnow
9f4291dcc9 [enhancement] (community) update intellij style file (#9783) 2022-05-29 17:34:25 +08:00
compiletheworld
635b5630e3 [fix](third-party) Fix build issues of third-party in ubuntu docker (#9805) 
Fixes:
1. Fully static linkage of protoc may lead to core dump when using it.
2. Building libmyclient may fail due to miss-used option `-v` of `make`
3. Building lzma implicitly requires autopoint
4. Building xml2 implicitly requires pkg-config
 2022-05-29 17:33:59 +08:00
Stalary
e231273ddf [fix](sql-block-rule) sql block rule NPE (#9778) 2022-05-29 16:21:00 +08:00
yixiutt
1aeb16d153 [improvement](load) reduce useless err_msg format in VOlapTableSink send (#9531) 2022-05-29 16:02:57 +08:00
Mingyu Chen
9fe3827239 [fix](ut) fix BE ut (#9831) 
introduced from #8923, the github checks has some problem that failed to check BE ut in #8923
 2022-05-29 12:25:41 +08:00
htyoung
3ee804f7b3 [doc]join优化文档笔误更改 (#9825) 2022-05-28 21:36:25 +08:00
Pxl
f33ef32d92 [Bug] [Bitmap] change to_bitmap to always_not_nullable (#9716) 2022-05-28 17:33:55 +08:00
Dongyang Li
92c1c82422 [tools]add ssb flat test scripts (#9796) 
Co-authored-by: stephen <hello-stephen@qq.com>
 2022-05-28 17:28:03 +08:00
Dayue Gao
4d1e926b6c [feature][config] introduce a new BE config storage_page_cache_shard_size (#9821) 
Co-authored-by: gaodayue <gaodayue@bytedance.com>
 2022-05-28 10:17:09 +08:00
Adonis Ling
cfe640c37b [doc] Add a note about the version of checkstyle (#9785) 2022-05-27 21:58:49 +08:00
pengxiangyu
ee1bed46be [config] Add backend_rpc_timeout_second in FE config (#9779) 2022-05-27 21:58:09 +08:00
LOVEGISER
77de4869ed [doc] website document update (#9776) 2022-05-27 21:57:06 +08:00
Kang
efdb3b79a5 [feature] add zstd compression codec (#9747) 
ZSTD compression is fast with high compression ratio. It can be used to archive higher compression ratio
than default Lz4f codec for storing cost sensitive data such as logs.

Compared to Lz4f codec, we see zstd codec get 35% compressed size off, 30% faster at first time read without OS page 
cache, 40% slower at second time read with OS page cache in the following comparison test.

test data: 25GB text log, 110 million rows
test table: test_table(ts varchar(30), log string)
test SQL: set enable_vectorized_engine=1; select sum(length(log)) from test_table
be.conf: disable_storage_page_cache = true
set this config to disable doris page cache to avoid all data cached in memory for test real decompression speed.
test result

master branch with lz4f codec result: 
- compressed size 4.3G
- SQL first exec time(read data from disk + decompress + little computation) : 18.3s
- SQL second exec time(read data from OS pagecache + decompress + little computation) : 2.4s

this branch with zstd codec (hardcode enable it) result:
- compressed size: 2.8G
- SQL first exec time: 12.8s
- SQL second exec time: 3.4s
 2022-05-27 21:56:18 +08:00
Lightman
b2c2cdb122 [feature] Support compression prop (#8923) 2022-05-27 21:52:05 +08:00
shaqf
ea4eaf1411 [doc] fix typos in bloomfilter.md (#9806) 
修改错别字
 2022-05-27 20:47:18 +08:00
Luwei
af2cfa2db4 [fix] Fix bug of bloom filter hash value calculation error (#9802) 
* Fix bug of bloom filter hash value calculation error

* fix code style
 2022-05-27 20:44:26 +08:00
Jerry Hu
a52e91a140 [chore] Update .gitignore to ignore generated files in tools (#9782) 2022-05-27 20:43:06 +08:00
hanfengcan
b2b9463537 [doc] Updated the compilation and deployment chapter in the documentation (#9702) 
Updated the compilation and deployment chapter in the documentation
 2022-05-27 17:11:31 +08:00
smallhibiscus
8019da4504 fix the format error of en doc. (#9756) 
fix the format error of en doc
 2022-05-27 17:10:47 +08:00
lit
dbc2feb31a modify bloomfilter.md (#9798) 
modify bloomfilter
 2022-05-27 17:10:08 +08:00
jiafeng.zhang
b07004e2f7 [fix]Documentation fixes (#9787) 
[fix]Documentation fixes （rollup)
 2022-05-27 12:57:00 +08:00
jiafeng.zhang
e7e551ffa7 [doc][fix]Sync job doc (#9790) 
binlog load documentation fix
 2022-05-27 12:56:19 +08:00
jiafeng.zhang
80e9c3395a [doc]Add export sql help documentation (#9797) 
Add export sql help documentation
 2022-05-27 12:55:45 +08:00
jiafeng.zhang
add3f5da85 [doc]Remove tcp-h (#9794) 
remove tcp-h
 2022-05-27 12:55:19 +08:00
Stalary
6698f63dec [fix](function) If function adds type inference (#9728) 2022-05-26 22:43:18 +08:00
yinzhijian
cbbda7857b [feature-wip](parquet-orc) Support orc scanner in vectorized engine (#9541) 2022-05-26 21:39:12 +08:00
jiafeng.zhang
dbc5af44ef [doc]Add Doris join optimization documentation (#9753) 
* [doc]Add Doris join optimization documentation
 2022-05-26 18:11:43 +08:00
morrySnow
e701c057dc [style](fe) wrap and whitespace rules (#9764) 
change below rules' severity to error and fix original code error:

- EmptyBlock
- EmptyCatchBlock
- LeftCurly
- RightCurly
- IllegalTokenText
- MultipleVariableDeclarations
- OneStatementPerLine
- StringLiteralEquality
- UnusedLocalVariable
- Indentation
- OuterTypeFilename
- MethodParamPad
- GenericWhitespace
- NoWhitespaceBefore
- OperatorWrap
- ParenPad
- WhitespaceAfter
- WhitespaceAround
 2022-05-26 16:56:20 +08:00
Pxl
13c1d20426 [Bug] [Vectorized] add padding when load char type data (#9734) 2022-05-26 16:51:01 +08:00
jacktengg
9236c2efc9 [improvement] Show detail status code string for be http api (#9771) 
1. move to_json method to common/status
2. modify related usage in http folder
 2022-05-26 15:09:21 +08:00
jacktengg
f4dd3bf013 [bugfix] fix memleak in olapscannode(#9736) 2022-05-26 15:06:54 +08:00
wudi
8898b11bb0 [docs]add sql mode markdown (#9742) 
Co-authored-by: wudi <>
 2022-05-26 15:06:23 +08:00
Gabriel
24631915ed [bugfix] fix correctness for vectorized compaction (#9773) 2022-05-26 15:05:50 +08:00
Gabriel
cd99c24844 [Improvement] remove unused code in vectorized compaction (#9774) 2022-05-26 15:05:27 +08:00
Mingyu Chen
32a210f426 [fix](help) fix bug of help command (#9761) 
This bug is introduced from #9306, that user need to execute
"help stream-load" to show the help doc.
But actually, it should be "help stream load".
 2022-05-26 08:44:00 +08:00
Mingyu Chen
0c70359404 [fix](resource-tag) Consider resource tags when assigning tasks for broker & routine load (#9492) 
This CL mainly changes:
1. Broker Load
    When assigning backends, use user level resource tag to find available backends.
    If user level resource tag is not set, broker load task can be assigned to any BE node,
    otherwise, task can only be assigned to BE node which match the user level tags.

2. Routine Load
    The current routine load job does not have user info, so it can not get user level tag when assigning tasks.
    So there are 2 ways:
    1. For old routine load job, use tags of replica allocation info to select BE nodes.
    2. For new routine load job, the user info will be added and persisted in routine load job.
 2022-05-26 08:42:09 +08:00
Adonis Ling
2a11a4ab99 [feature-wip][array-type] Support more sub types. (#9466) 
Please refer to #9465
 2022-05-26 08:41:34 +08:00
spaces-x
73e31a2179 [stream-load-vec]: memtable flush only if necessary after aggregated (#9459) 
Co-authored-by: weixiang <weixiang06@meituan.com>
 2022-05-25 21:12:24 +08:00
Gabriel
8470543144 [Improvement] fix typo (#9743) 2022-05-25 19:29:01 +08:00
Zhengguo Yang
f5bef328fe [fix] disable transfer data large than 2GB by brpc (#9770) 
because of brpc and protobuf cannot transfer data large than 2GB, if large than 2GB will overflow, so add a check before send
 2022-05-25 18:41:13 +08:00
Zhengguo Yang
be026addde [security] update canal version to fix fastjson security issue (#9763) 2022-05-25 18:22:37 +08:00