8d17eea22b
[pipeline](mysqlscan) support mysql scan node ( #14949 )
2022-12-09 16:25:29 +08:00
b213ce6ffd
[Bug](pipeline) Fix double prepare on pipeline engine ( #14959 )
2022-12-09 14:35:34 +08:00
873b128fde
[feature](pipeline) add inersect/except operators ( #14868 )
2022-12-09 14:13:48 +08:00
5a1c7f6314
[improvement](analytic) improve memory counter ( #14890 )
2022-12-09 14:13:17 +08:00
ec24a56956
[Exec][Pipeline] Remove AggContext's lock ( #14901 )
...
* [Exec][Pipeline] Remove AggContext's lock
* fix lock location
2022-12-09 14:11:28 +08:00
9d36931038
[Refactor](NLJ) refactor the nested loop join node ( #14911 )
...
* [Refactor](NLJ) refactor the nested loop join node
* change the logic of alloc/release resource
2022-12-09 14:10:26 +08:00
dffa3c0db2
[enhancement](memory) Support query memroy overcommit #14948
...
Add conf enable_query_memroy_overcommit
If true, when the process does not exceed the soft mem limit, the query memory will not be limited; when the process memory exceeds the soft mem limit, the query with the largest ratio between the currently used memory and the exec_mem_limit will be canceled.
If false, cancel query when the memory used exceeds exec_mem_limit, same as before.
2022-12-09 14:09:05 +08:00
00f44257e2
[feature-wip](file-reader) Merge hdfs reader to the new file reader ( #14875 )
2022-12-09 13:21:59 +08:00
20f2abb3d4
[vectorized](pipeline) support assert num rows operator ( #14923 )
2022-12-09 09:39:29 +08:00
b311ebef6c
[pipeline](refactor) do some refactor for code and comments ( #14934 )
2022-12-09 09:08:28 +08:00
41eafa5e9d
[fix](aggregate) agg table and unique table get different result when… ( #14822 )
...
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2022-12-08 22:54:36 +08:00
375e0e08ca
[Bug](predicate) fix ccore dump on varchar with in list predicate ( #14881 )
...
* fix ccore dump on varchar with in list predicate
* update case
* Update sqlsmith01.sql
2022-12-08 17:14:23 +08:00
0c817e6b3a
[Pipeline](hashjoin) Support hash join on pipeline engine ( #14898 )
2022-12-08 15:43:02 +08:00
dbaa02d3a0
[Pipeline](fix) fix enable_pipeline_engine variable not work ( #14909 )
2022-12-08 14:52:52 +08:00
4c5ddf70db
[bugfix](conf) remove useless conf ( #14918 )
2022-12-08 14:11:15 +08:00
962810b973
[Vectorized](jdbc) add check type for jdbc table ( #14501 )
2022-12-08 10:27:47 +08:00
6a26435e8d
[bugfix](compaction) fix promotion size bug ( #14836 )
2022-12-07 18:54:30 +08:00
204ab4c951
[enhancement](compaction) add some trigger and delete useless log ( #14796 )
...
1.add a vertical compaction segment file size config, make it more
flexible to set segment file size
2.add a config to close skip tablet compaction. If current skip logic
has some bug so we can still use old logic
3.delete some useless log
2022-12-07 18:53:55 +08:00
dfa58e7985
[improvement](config) update high_priority_flush_thread_num_per_store default value to 6 ( #14775 )
2022-12-07 15:31:00 +08:00
ec2539e2a3
[chore](macOS) Resolve the issue with missing python program ( #14864 )
2022-12-07 15:30:12 +08:00
48a9166aa4
[Pipeline](sink) support olap table sink operator ( #14872 )
...
* support olap table sink operator
* update config
2022-12-07 15:29:56 +08:00
cdbbf1e4ee
[enhancement](memory) Add Memory GC when the available memory of the BE process is lacking ( #14712 )
...
When the system MemAvailable is less than the warning water mark, or the memory used by the BE process exceeds the mem soft limit, run minor gc and try to release cache.
When the MemAvailable of the system is less than the low water mark, or the memory used by the BE process exceeds the mem limit, run fucc gc, try to release the cache, and start canceling from the query with the largest memory usage until the memory of mem_limit * 20% is released.
2022-12-07 15:28:52 +08:00
9d2cb133f2
[fix](jdbc) fix logger error of statusLogger unrecognized ( #14854 )
...
* [fix](jdbc) fix logger error of statusLogger unrecognized
* update
2022-12-07 11:43:05 +08:00
3286fb48ab
[fix](if) fix coredump of if const ( #14858 )
2022-12-07 09:43:10 +08:00
df2b190559
[pipeline](dategen) Support datagen node ( #14835 )
2022-12-07 08:47:35 +08:00
9e51e0263d
[fix](memory leak) Fix load fragment QueryFragmentsCtx is not destroyed ( #14840 )
2022-12-07 08:45:53 +08:00
5292880310
[refactor](odbc) move param to config ( #14596 )
...
move param to config
2022-12-06 17:38:52 +08:00
1304185adb
[Regression](Fix) fix the regression of pipeline and ConcurrentModificationException failed ( #14849 )
...
* [fix](ut) try to fix ConcurrentModifycationException bug
* [Regression](Fix) fix the regression of pipeline and ConcurrentModificationException failed
Co-authored-by: morningman <morningman@163.com >
2022-12-06 15:34:32 +08:00
fcea89bcf4
[fix](const_expr) fix coredump caused by unsupported cast const expr ( #14825 )
2022-12-06 10:31:15 +08:00
e279c90965
[fix](ColumnVector) ColumnVector::insert_date_column crashed #14839
...
ColumnVector::insert_date_column make BE crashed with large data(>512 rows).
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-12-06 09:06:57 +08:00
07472f7318
[fix](tcmalloc_gc) optimize policy of tcmalloc gc ( #14776 )
...
Release memory when memory pressure is above pressure limit and keep at lease 2% memory as tcmalloc cache.
2022-12-05 21:16:35 +08:00
b30cd86e9e
[Refactor](pipeline) Refactor operator and builder code of pipeline ( #14787 )
2022-12-05 18:35:00 +08:00
382d35c7e1
[fix](runtime profile) Fix HighWaterMarkCounter update max Bug #14769
2022-12-05 18:22:48 +08:00
8726bfa121
[enhancement](memory) Add tablet schema cache metrics ( #14742 )
2022-12-05 18:19:13 +08:00
016e47b857
[refactor](date function) simplify function template arguments ( #14814 )
2022-12-05 17:59:47 +08:00
1190fd4cd6
[Pipeline](regression) Add ssb flat for pipeline ( #14763 )
2022-12-05 15:05:23 +08:00
8c0e13ab51
[improvement](profile) add detail memory counter for exec nodes ( #14806 )
...
* [improvement](profile) improve accuraccy of memory usage and add detail memory counter
* fix
2022-12-05 11:51:52 +08:00
58bc254529
[enhancement](BE)add metric for too many version ( #14735 )
...
* add one funciton to get if exceeds version limit
add bvar to indicate version exceed
* resolve
* remove unnecessary header file
2022-12-05 11:37:14 +08:00
8a834566d0
[typo](docs) fix schema change DATA_QUALITY_ERROR typo and related error msg ( #14773 )
2022-12-05 09:50:20 +08:00
e141664339
[fix](statistics) fix missing scanBytes and scanRows in query statist… ( #14750 )
...
* [fix](statistics) fix missing scanBytes and scanRows in query statistics when enable_vectorized_engine=true.
Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com >
2022-12-05 09:17:51 +08:00
ed96442b85
[fix](multi-catalog) fix persist issue about jdbc catalog and class loader issue #14794
...
Fix a bug that JDBC catalog/database/table should be add to GsonUtil
Fix a class loader issue that sometime it will cause ClassNotFoundException
Fix regression test to use different catalog name.
Comment out 2 regression tests:
regression-test/suites/query_p0/system/test_query_sys.groovy
regression-test/suites/statistics/alter_col_stats.groovy
Need to be fixed later
2022-12-05 09:05:13 +08:00
ce95da8dfb
[improvement](multi-catalog) support specify hadoop username ( #14734 )
...
Support setting "hadoop.username" property when creating hms catalog.
2022-12-04 21:09:39 +08:00
12304bc0ee
[Pipeline](exec) Support pipeline exec engine ( #14736 )
...
Co-authored-by: Lijia Liu <liutang123@yeah.net >
Co-authored-by: HappenLee <happenlee@hotmail.com >
Co-authored-by: Jerry Hu <mrhhsg@gmail.com >
Co-authored-by: Pxl <952130278@qq.com >
Co-authored-by: shee <13843187+qzsee@users.noreply.github.com >
Co-authored-by: Gabriel <gabrielleebuaa@gmail.com >
## Problem Summary:
### 1. Design
DSIP: https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine
### 2. How to use:
Set the environment variable `set enable_pipeline_engine = true; `
2022-12-02 17:11:34 +08:00
505019e1dd
[minor](decimal) degrade log level for overflow decimal ( #14748 )
2022-12-02 15:55:26 +08:00
3dde97bff1
(compaction) opt compaction task producer and quick compaction ( #13495 ) ( #14535 )
...
1.remove quick_compaction's rowset pick policy, call cu compaction when trigger
quick compaction
2. skip tablet's compaction task when compaction score is too small
Co-authored-by: yixiutt <yixiu@selectdb.com >
2022-12-02 10:07:44 +08:00
5b29489c7f
(tcmalloc) gc does not work in somecases ( #14732 )
...
gc does not work in some cases
2022-12-02 09:18:23 +08:00
94a6ffb906
[feature](compaction) support vertical_compaction & ordered_data_compaction ( #14524 )
2022-12-01 22:15:41 +08:00
ba9a777554
[fix](function) StringRef should not be key of timezone cache ( #14719 )
2022-12-01 16:31:47 +08:00
9dd1d989e8
[test](decimalv3) add regression test cases for decimalv3 ( #14672 )
2022-12-01 15:18:40 +08:00
176f519fa1
[enhancement](memtracker) Optimize exec node memory tracking ( #14711 )
2022-12-01 14:52:21 +08:00
