d6b7640cf0
[fix](inverted index) fix check failed for block erase temp column ( #20924 )
2023-06-18 19:27:48 +08:00
48065fce19
[bugfix](merge-on-write) optimize rowset tree and tablet header lock ( #20911 )
2023-06-18 19:26:02 +08:00
ce9a20a375
[enhancement](merge-on-write) format logs about MoW and add more stats for publish ( #20853 )
2023-06-17 23:14:28 +08:00
0585a1f004
[fix](compaction) fix time series compaction policy to adjust vertical compaction max segment size ( #20889 )
2023-06-17 20:32:34 +08:00
cc3f9ed9b7
[Fix](fd) fix fd limit over 100% ( #20778 )
2023-06-17 19:54:10 +08:00
0a59580aa4
[Enhancement](function) fix compatibility issues of sum/count during upgrade process ( #20890 )
...
in order to solve agg of sum/count is not compatibility during the upgrade process.
in PR [refactor](agg_state) refactor agg_state type to support fixed length object type #20370 have changed the serialize type and serialize column of sum/count
before is ColumnVector, now sum/count change to use ColumnFixedLengthObject
so during the upgrade process, will be not compatible if exist Old BE and Newer BE
2023-06-17 12:51:01 +08:00
aea719627d
Revert "[enhencement](streamload) add on_close callback for httpserver ( #20826 )" ( #20927 )
...
This reverts commit 5b6761acb86852a93351b7b971eb2049fb567aaf.
2023-06-17 10:39:02 +08:00
ab32299ba4
[feature](nereids) Support multi target rf #20714
...
Support multi target runtime filter, mainly for set operation, such as union/intersect/except.
2023-06-16 20:26:00 +08:00
baed26acc6
[fix](merge-on-write) fix publish timeout ( #20891 )
2023-06-16 18:11:41 +08:00
2e295a1ee9
[Enhancement](http) unify http auth config ( #20864 )
2023-06-16 16:55:46 +08:00
97135a1cbb
[Feature] (json)add json_contains function ( #20824 )
2023-06-16 15:10:12 +08:00
f80bf3c1bc
[fix](compaction) fix time series compaction policy ( #20837 )
2023-06-16 13:25:55 +08:00
b7a50a09fe
[Opt](orc-reader) Optimize orc reader by dict filtering. ( #20806 )
...
Optimize orc reader by dict filtering. It is similar with #17594 .
Test result
**ssb-flat-100**: (3 nodes)
| Query | before opt | after opt |
| ------------- |:-------------:| ---------:|
Q1.1 | 1.239 | 1.145
Q1.2 | 1.254 | 1.128
Q1.3 | 1.931 | 1.644
Q2.1 | 1.359 | 1.006
Q2.2 | 1.229 | 0.674
Q2.3 | 0.934 | 0.427
Q3.1 | 2.226 | 1.712
Q3.2 | 2.042 | 1.562
Q3.3 | 1.631 | 1.021
Q3.4 | 1.618 | 0.732
Q4.1 | 2.294 | 1.858
Q4.2 | 2.511 | 1.961
Q4.3 | 1.736 | 1.446
total | 22.004 | 16.316
2023-06-16 13:11:37 +08:00
1ec7f0e50a
[fix](memory) memory management thread exits gracefully
...
memory_maintenance_thread, memory_gc_thread, load_channel_tracker_refresh_thread, memory_tracker_profile_refresh_thread
2023-06-16 11:40:24 +08:00
9d41edd9eb
[Feature](binlog) Add binlog gc && Auth master_token ( #20854 )
...
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-06-16 11:25:11 +08:00
db1da6b787
[Chore](pipeline) add some profile log when pipeline canceled ( #20825 )
...
add some profile log when pipeline canceled
2023-06-16 10:54:54 +08:00
420603317b
[improve](match) Improve performance for match query without inverted index ( #20815 )
2023-06-16 10:35:38 +08:00
b38417358b
[chore](compile) Remove fs/benchmark/fs_benchmark_tool.cpp in IO_FILES ( #20845 )
2023-06-15 21:16:59 +08:00
f1af09ef87
[Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments ( #20706 )
2023-06-15 21:11:39 +08:00
4bfceb7acb
[Bug](pipeline) fix pipeline task call finish_p_dependency more than once ( #20851 )
...
fix pipeline task call finish_p_dependency more than once
When pipeline task meet eos->PENDING_FINISH->CANCELED, this task will call finish_p_dependency twice.
2023-06-15 19:21:40 +08:00
bb5d36c5cb
[Log](load) change VLOG to INFO when write replica failing ( #20783 )
2023-06-15 16:11:14 +08:00
b6835840f7
[Bug](table-function) return InvalidArgument when explode_split meet empty delimiter ( #20795 )
...
return InvalidArgument when explode_split meet empty delimiter
2023-06-15 15:17:22 +08:00
01e53f4e67
[Bug](materialized-view) fix problems about create mv on ssb_flat q4.1 failed ( #20658 )
...
fix problems about create mv on ssb_flat q4.1 failed
2023-06-15 14:38:21 +08:00
15b9830859
[fix](partial-update) sequence column is not proceeded correctly #20813
...
When checking the keys in PrimaryKeyIndex, seq_col_length is not set to correct value, then we got a NOT_FOUND result for an existing key.
2023-06-15 14:07:00 +08:00
17a395f5e3
[Bug](runtime-filter) fix runtime filter not register on vdata_gen_scan_node ( #20787 )
...
fix runtime filter not register on vdata_gen_scan_node
2023-06-15 14:06:14 +08:00
2151f5d04d
[fix](bitmap) fix bug: incorrect orthogonal bitmap result in some cases ( #20819 ) ( #20822 )
...
Issue Number: close #20819
If there is only one aggregation (update finalize) phase, result field will not be updated.
This pr is aim to resolve it.
2023-06-15 14:05:24 +08:00
5b6761acb8
[enhencement](streamload) add on_close callback for httpserver ( #20826 )
...
Sometimes connection cannot be released properly during on_free. We need
on_close callback as the last resort.
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-06-15 13:44:02 +08:00
4b18cde0c7
[Update](clucene) synchronize clucene version to improve clcuene performance ( #20794 )
...
Improve clucene performance by zero copy memory and SIMD string compare.
Related PR: apache/doris-thirdparty#87
2023-06-15 10:35:51 +08:00
1ce8f13837
[fix](memory) fix mem tracker in NodeChannel rpc callback ( #20779 )
2023-06-15 10:35:25 +08:00
5c4f6b4ca9
[fix](MOW)Fix publish timeout when enable MOW ( #20828 )
2023-06-15 10:34:20 +08:00
4bf15b9788
[fix](load) fix race condition problem when insert commitinfo ( #20823 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-06-15 09:53:32 +08:00
460399f214
[fix](profile) remove same profile in join node ( #20734 )
2023-06-15 08:08:39 +08:00
2a2e485456
[Enhancement](compaction) time-series scenario cumulative compaction policy ( #20715 )
...
new compaction policy for log and time-series scenario
2023-06-14 23:48:44 +08:00
09d187ec77
[improvement](ck jdbc) Optimized reading of datetime and ip types of the ClickHouse JDBC Catalog ( #20804 )
2023-06-14 23:28:08 +08:00
bb617ee2cc
[fix](parquet-reader)fix page v2 header offset ( #20814 )
...
fix page v2 header offset.
get correct offset when read next page in file.
2023-06-14 23:27:31 +08:00
3727483c06
[Chore](build) update ldb_toolchain to v0.18 ( #20802 )
...
* update ldb_toolchain to v0.18
* update
2023-06-14 18:38:35 +08:00
0ecc98df82
[Bug](rowset) expire delayed rowsets should be ignored and should not be deleted in _tablet_meta ( #20803 )
2023-06-14 18:30:13 +08:00
31a4f96f01
[refactor](exprcontext) move close to expr context's dector method ( #20747 )
...
The close method does nothing. But I am not sure we could remove it. So that I add it to dector method and remove many many calls.
2023-06-14 18:01:07 +08:00
b97537b04b
[Fix](MOW) Fix load data publish timeout when enable unique key MOW ( #20720 )
2023-06-14 17:56:02 +08:00
615778924e
[feature](fs) add fs benchmark tool framework ( #20770 )
...
Add an optional executable binary fs_benchmark_tool, for test the performance of file system such as hdfs, s3.
Usage:
./fs_benchmark_tool --conf my.conf --fs_type=s3 --operation=read --iterations=5
in my.conf, you can add any config key value with following format:
key1=value1
key2=value2
By default, this binary will not be built. Only build it when setting BUILD_FS_BENCHMARK=ON.
The binary will be installed in output/be/lib.
For developer, you can add new subclass of BaseBenchmark to add your own benchmark.
See be/src/io/fs/benchmark/s3_benchmark.hpp for an example
2023-06-14 17:50:06 +08:00
a0d4f11667
[Bug](function) catch error state in function cast to avoid core dump ( #20751 )
...
catch error state in function cast to avoid core dump
2023-06-14 17:34:34 +08:00
d922a4a9fa
[Feature-WIP](inverted index) add inverted index file size method ( #20758 )
...
This PR calculates the size of the inverted index files. The changes consist of:
Introduction of a new get_inverted_index_size() method in different column writers such as ScalarColumnWriter, StructColumnWriter, ArrayColumnWriter, and MapColumnWriter. This method will fetch the size of the inverted index file associated with that column. If the file size cannot be fetched, it defaults to 0.
A new method file_size() has been added in InvertedIndexColumnWriter class which retrieves the size of the file stored on disk. If the file size cannot be fetched, it logs an error and returns -1.
Additionally, a new method get_inverted_index_file_size() is introduced in SegmentWriter which aggregates the inverted index file sizes of all the column writers.
2023-06-14 17:18:20 +08:00
dd5b82fe00
[Enhancement](merge-on-write) optimize contains_agg when calculate delete bitmap ( #20762 )
2023-06-14 16:25:11 +08:00
0f470fec0e
[Bug](topn opt) Fix Two-Phase read when some rowset swept ( #20732 )
...
* [Bug](topn opt) Fix Two-Phase read when some rowset swept
If this is a Two-Phase read query, and we need to delay the release of Rowset by row->update_delayed_expired_timestamp() to expand the lifespan of rowsets. This is necessary to avoid data loss during the second phase reading, where some stale rowsets may be swept and result in missing data.
2023-06-14 15:46:29 +08:00
f2025b9eed
[fix](memory) before compaction run, check memory exceed limit #20782
2023-06-14 14:20:48 +08:00
9b4b0d4bf9
[fix](cooldown) Fix bug when cooldown a dropped tablet ( #20750 )
2023-06-14 09:42:55 +08:00
fd97587aff
[fix](merge-on-write) fix the merged rows is not equal to missed rows when do cumulative compaction ( #20754 )
2023-06-13 22:18:59 +08:00
9244cb6553
[Chore](runtime-filter) do not make query fail when rf publish failed ( #20742 )
...
do not make query fail when rf publish failed
2023-06-13 18:23:46 +08:00
ad2f1b5647
[Update](clucene) synchronize clucene version to address PFOR adaptation issue ( #20736 )
2023-06-13 18:04:48 +08:00
feb21fc9e9
[fix](group_concat) use default seperator ',' instead of ', ' for group_concat, to be consistant with mysql ( #20741 )
2023-06-13 17:20:29 +08:00
