e6d20f842c
[Bug](compile) fix compile failed on function case ( #23335 )
2023-08-22 22:10:53 +08:00
5c2fae7ce5
[pipeline](exec) Refactor the table sink code in remove unless code ( #23223 )
...
Refactor the table sink code in remove unless code
2023-08-22 20:42:14 +08:00
eeec26d68d
[refactor](load) rename flush_memtable_and_wait to flush_async ( #23204 )
2023-08-22 20:07:50 +08:00
1609b6cbf2
[pipelineX](sort) Support sort operator ( #23322 )
2023-08-22 19:36:50 +08:00
8819d73abd
[fix](be) fix the crash when there is no tzfile in docker env ( #23071 )
2023-08-22 18:56:36 +08:00
1a1f86486d
[Improvement](function) opt for case when ( #23068 )
...
opt for case when
2023-08-22 18:31:40 +08:00
0b51e6d8e1
[refractor](FunctionArrayIndex) make the codes more simple
2023-08-22 17:48:59 +08:00
4a690f8b13
fix compile error with gcc12 ( #23190 )
2023-08-22 16:05:03 +08:00
bcdb481374
[refactor](fragment) refactor non pipeline fragment executor ( #23281 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-08-22 16:00:34 +08:00
9d2e23b1aa
[fix](parquet) A row of complex type may be stored across more pages ( #23277 )
...
A row of complex type may be stored across two(or more) pages, and the parameter `align_rows` indicates that whether the reader should read the remaining value of the last row in previous page.
2023-08-22 14:47:10 +08:00
5ff7b57fc1
[fix](parquet) parquet reader confuses logical/physical/slot id of columns ( #23198 )
...
`ParquetReader` confuses logical/physical/slot id of columns. If only reading the scalar types, there's nothing wrong, but when reading complex types, `RowGroup` and `PageIndex` will get wrong statistics. Therefore, if the query contains complex types and pushed-down predicates, the probability of the result set is incorrect.
2023-08-22 13:35:29 +08:00
7450bd504d
[improvement](scan) Reduce time to fetch function calls ( #23098 )
2023-08-22 11:24:55 +08:00
438010a214
[fix](merge-on-write) process error of delete bitmap calculation ( #23285 )
2023-08-22 10:53:50 +08:00
0d7a61ae8c
[fix](load) fix duplicate register of memtable writer in memory limiter ( #23205 )
2023-08-22 10:05:17 +08:00
12075f9853
[pipelineX](projection) Support projection and blocking agg ( #23256 )
2023-08-21 22:23:02 +08:00
8503d26e5c
[bugfix](vertical-compaction) Only can init the SegmentCacheHandle once ( #23246 )
2023-08-21 21:57:27 +08:00
3b4cc06a62
[feature](move-memtable)[1/8] support creating segments externally in rowset writer ( #23085 )
2023-08-21 21:02:34 +08:00
d4642b47b4
[fix](InternalService) add short-cut return when offer failed ( #23239 )
...
During offer_failed(), rpc done will be executed so the response will be
sent and released. Further access to such object will cause NPE error.
So let us just return after offer_failed().
2023-08-21 21:00:49 +08:00
dcd6c3c022
[pipelineX](refactor) propose a new pipeline execution model ( #22562 )
2023-08-21 15:38:45 +08:00
d4694167a8
[Enhancement](chore) Some Status relevant enhancement ( #23072 )
2023-08-21 14:14:38 +08:00
81dd00f6e4
[Feature](Compaction) Support do full compaction by table id ( #22010 )
2023-08-21 11:54:51 +08:00
37b49f60b7
[refactor](conf) add be conf for partition topn partitions threshold ( #23220 )
...
add be conf for partition topn partitions threshold
2023-08-21 10:52:41 +08:00
33dfa0c454
[Improve](serde) support text serde for nested type-array/map ( #22738 )
...
Now we can not support nested type array/map
so this pr aim to:
1. add format option for string convert defined datatype to keep with origin from_string
2. support array map can nested array and map
2023-08-21 10:32:28 +08:00
0967d7ec04
[improvement](agg) Do not serialize bitmap to string ( #23172 )
2023-08-21 10:10:15 +08:00
a11e0e3bc4
[Bug](agg) fix QUANTILE_UNION many problems ( #23181 )
...
fix QUANTILE_UNION many problems
2023-08-21 10:04:27 +08:00
4bf055c818
[fix](parquet) the key colum of map type in parquet may be nullable ( #23180 )
...
Fix errors when reading map type with nullable key column in parquet file. `ParquetReader` support to read nullable key column, but add a check to prevent reading nullable key column. Unfortunately, this check error was not thrown correctly, causing the BE to crash, and thrown meaningless error logs in be.out:
```
...
11# doris::vectorized::ParquetReader::get_columns(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, doris::TypeDescriptor, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, doris::TypeDescriptor> > >*, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) at /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:508
12# doris::vectorized::VFileScanner::_get_next_reader() in /root/yun_you_external/output/be/lib/doris_be
13# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:241
...
```
2023-08-20 22:59:18 +08:00
58b9bce954
[fix](load) add rowset builder init error handling ( #23166 )
2023-08-19 17:13:10 +08:00
433a6103ab
[Enhancement](scanner) allocate blocks in scanner_context on demand and free them on close ( #23182 )
...
Introduced #19389 , removed #20785
2023-08-19 12:13:24 +08:00
0838ff4bf4
[fix](Outfile) fix bug that the fileSize is not correct when outfile is completed ( #22951 )
2023-08-18 22:31:44 +08:00
26905e36e5
[fix](load) fix nullptr in memtable limiter flush ( #23149 )
2023-08-18 19:55:53 +08:00
419e922a69
[fix](json)Fix the bug that does not stop when reading json files ( #23062 )
...
* [fix](json)Fix the bug that does not stop when reading json files
2023-08-18 18:23:19 +08:00
477961dc21
[Chore](agg) refactor of hash map ( #22958 )
...
refactor of hash map
2023-08-18 17:59:30 +08:00
f0ad3ef244
[fix](merge-on-write) should use write lock of tablet's header lock in #23047 ( #23161 )
2023-08-18 17:50:44 +08:00
3d4ec1ac88
[pipeline](exec) support async writer in jdbc sink in pipeline query engine ( #23144 )
...
support async writer in jdbc sink in pipeline query engine
2023-08-18 17:07:57 +08:00
1c3cc77a54
[fix](function) to_bitmap parameter parsing failure returns null instead of bitmap_empty ( #21236 )
...
* [fix](function) to_bitmap parameter parsing failure returns null instead of bitmap_empty
* add ut
* fix nereids
* fix regression-test
2023-08-18 14:37:49 +08:00
cf368728be
[fix](merge-on-write) Fix a typo and remove useless member rowset in CommitTabletTxnInfo ( #23151 )
...
Fix a typo in #23078
2023-08-18 14:14:34 +08:00
795006ea3d
[fix](multi-catalog) conversion of compatible numerical types ( #23113 )
...
Hive support schema change, but doesn't rewrite the parquet file, so the physical type of parquet file may not equal the logical type of table schema.
2023-08-18 14:05:33 +08:00
4f7760a5f4
[bugfix](segment cache) Recycle the fds when drop table ( #23081 )
2023-08-18 13:31:34 +08:00
e6fe8c05d1
[fix](inverted index change) fix update delete bitmap incompletely when build inverted index on mow table ( #23047 )
2023-08-18 11:15:39 +08:00
a5ca6cadd6
[Improvement] Optimize count operation for iceberg ( #22923 )
...
Iceberg has its own metadata information, which includes count statistics for table data. If the table does not contain equli'ty delete, we can get the count data of the current table directly from the count statistics.
2023-08-18 09:57:51 +08:00
de98324ea7
[fix](inverted index change) make mutex for ALTER_INVERTED_INDEX task and STORAGE_MEDIUM_MIGRATE task ( #22995 )
2023-08-18 08:35:30 +08:00
314f5a5143
[Fix](orc-reader) Fix filling partition or missing column used incorrect row count. ( #23096 )
...
[Fix](orc-reader) Fix filling partition or missing column used incorrect row count.
`_row_reader->nextBatch` returns number of read rows. When orc lazy materialization is turned on, the number of read rows includes filtered rows, so caller must look at `numElements` in the row batch to determine how
many rows were not filtered which will to fill to the block.
In this case, filling partition or missing column used incorrect row count which will cause be crash by `filter.size() != offsets.size()` in filter column step.
When orc lazy materialization is turned off, add `_convert_dict_cols_to_string_cols(block, nullptr)` if `(block->rows() == 0)`.
2023-08-17 23:26:11 +08:00
57568ba472
[fix](be)shouldn't use arena to alloc memory for SingleValueDataString ( #23075 )
...
* [fix](be)shouldn't use arena to alloc memory for SingleValueDataString
* format code
2023-08-17 22:18:09 +08:00
29ff7b7964
[fix](merge-on-write) add sentinel mark when do compaction ( #23078 )
2023-08-17 20:08:01 +08:00
c5c984b79b
[refactor](bitmap) using template to reduce duplicate code ( #23060 )
...
* [refactor](bitmap) support for batch value insertion
* fix values was not filled for int8 and int16
2023-08-17 18:14:29 +08:00
330f369764
[enhancement](file-cache) limit the file cache handle num and init the file cache concurrently ( #22919 )
...
1. the real value of BE config `file_cache_max_file_reader_cache_size` will be the 1/3 of process's max open file number.
2. use thread pool to create or init the file cache concurrently.
To solve the issue that when there are lots of files in file cache dir, the starting time of BE will be very slow because
it will traverse all file cache dirs sequentially.
2023-08-17 16:52:08 +08:00
b252c49071
[fix](hash join) fix heap-use-after-free of HashJoinNode ( #23094 )
2023-08-17 16:29:47 +08:00
e289e03a1a
[fix](executor)fix no return with old type in time_round
2023-08-17 15:34:26 +08:00
cf1865a1c8
[Bug](scan) fix core dump due to store_path_map ( #23084 )
...
fix core dump due to store_path_map
2023-08-17 15:24:43 +08:00
8b51da0523
[Fix](load) fix partiotion Null pointer exception ( #22965 )
2023-08-17 14:09:47 +08:00
