0f0c0a266b
[opt](parquet)Skip page with offset index ( #33082 )
...
Make skip_page() in ColumnChunkReader more efficient. No more reading page headers if there are pagelocations in chunk.
2024-04-26 15:06:16 +08:00
c631f4f8a8
[fix](schema change) resolve the use count check of source logical column ( #33932 )
...
Fix error like:
```
8# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# doris::vectorized::Block::clear_column_data(int) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
10# doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:514
11# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vfile_scanner.cpp:333
12# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscanner.cpp:132
13# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscanner.cpp:99
```
Because source logical column is the destination logical column if logical converter is consistent. Previously, the reference of column was reset after the conversion was completed, but if an EOF occurred, it was returned in advance, but EOF is not a true error.
```
if (_logical_converter->is_consistent()) {
// If logical converter is consistent, _src_logical_column is the final destination column,
// other components will check the use count
_src_logical_column.reset();
}
```
2024-04-22 12:31:46 +08:00
ffd9da44a2
[fix](move-memtable) fix commit may fail due to duplicated reports ( #32403 )
2024-04-19 15:02:49 +08:00
9b7af4c0cf
[feature](schema change) unified schema change for parquet and orc reader ( #32873 )
...
Following #25138 , unified schema change interface for parquet and orc reader, and can be applied to other format readers as well.
Unified schema change interface for all format readers:
- First, read the data according to the column type of the file into source column;
- Second, convert source column to the destination column with type planned by FE.
2024-04-12 15:09:25 +08:00
5f30463bb3
[Chore](descriptors) remove unused codes for descriptors ( #33408 )
...
remove unused codes for descriptors
2024-04-12 15:09:25 +08:00
26d9082b9a
[Feature](function) Add function strcmp ( #33272 )
2024-04-12 15:09:25 +08:00
31984bb4f0
[feature](function) support quote string function #33055
2024-04-12 15:09:25 +08:00
5688c28364
[Bug](runtime-filter) try to fix heap use after free on runtime filter send filter size ( #33465 ) ( #33522 )
2024-04-11 13:10:24 +08:00
ef26479282
[improve](serde) support complex type in write/read pb serde ( #33124 )
...
support complex type and ip/jsonb in DataTypeSerDe::write_column_to_pb/read_column_from_pb function
2024-04-11 09:31:50 +08:00
8fd6d4c41b
[Chore](build) add -Wconversion and remove some unused code ( #33127 )
...
add -Wconversion and remove some unused code
2024-04-10 15:26:08 +08:00
c61d6ad1e2
[Feature] support function uuid_to_int and int_to_uuid #33005
2024-04-10 14:53:56 +08:00
bf022f9d8d
[enhancement](function truncate) truncate can use column as scale argument ( #32746 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-10 14:53:56 +08:00
cf7595d423
[opt](memory) Optimize mem tracker accuracy ( #32039 ) ( #33140 )
2024-04-10 11:42:19 +08:00
950ca68fac
[fix](move-memtable) fix timeout to get tablet schema ( #33256 ) ( #33260 )
2024-04-04 21:45:55 +08:00
df197c6a14
[fix](move-memtable) fix initial use count of streams for auto partition ( #33165 ) ( #33236 )
...
Co-authored-by: Kaijie Chen <ckj@apache.org >
2024-04-03 20:31:29 +08:00
ff0da8108b
[fix](RF) fix 'Invalid value' error of RF of decimal type ( #32749 )
2024-03-25 22:34:19 +08:00
d7a3ff1ddf
[Fix](Outfile) Fix the column type mapping in the orc/parquet file format ( #32281 )
...
| Doris Type | Orc Type | Parquet Type |
|---------------------|--------------------|------------------------|
| Date | Long (logical: DATE) | int32 (Logical: Date) |
| DateTime | TIMESTAMP (logical: TIMESTAMP) | int96 |
2024-03-22 08:52:16 +08:00
7486e96b12
[improve](function) add error msg if exceeded maximum default value in repeat function ( #32219 )
...
add some error msg from repeat function, so the user could know the count is greater than default value.
2024-03-21 14:07:49 +08:00
2196c534e8
[fix](group commit) Fix compatibility issues on serializing and deserializing wal file ( #32299 )
2024-03-21 14:07:24 +08:00
0990014e94
[fix](datetime) fix datetime rounding on BE ( #32075 )
2024-03-21 14:07:19 +08:00
ef2151ae66
[Feature-WIP](multi-catalog) Add Hive sink on BE side. ( #32306 ) ( #32364 )
...
bp #32306
Co-authored-by: Qi Chen <kaka11.chen@gmail.com >
2024-03-18 11:23:01 +08:00
0da010603e
[Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path ( #31141 )
...
Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse
2024-03-09 19:44:42 +08:00
8f77e6363a
[Feature](function) Support xxhash function like murmur hash function ( #31193 )
2024-02-23 19:03:28 +08:00
ad07dec0ed
[Improve](InPredict) enhance in predict with struct type ( #30840 )
2024-02-22 13:01:49 +08:00
b66583551c
[fix](group_commit)Fix bound checking problem when reading wal block ( #31112 )
2024-02-22 13:01:48 +08:00
f2a38e6345
[chore](columns) remove update_hashes_with_value for SipHash ( #31224 )
2024-02-22 13:01:48 +08:00
7a1bd6abb0
[improvment](group_commit) Refector scan wal function ( #30939 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2024-02-20 09:12:38 +08:00
bb4575a392
[Improvement](join) optimization for build_side_output_column ( #30826 )
...
optimization for build_side_output_column
2024-02-19 17:22:03 +08:00
6cf7468073
[enhancement](function) change some function nullable mode ( #30991 )
...
change some function nullable mode
2024-02-18 14:45:25 +08:00
68102fd531
[Fix](auto-partition) fix a concurrent bug of extremely long values ( #31005 )
2024-02-18 14:45:25 +08:00
45b4189bb6
[Refactor](opt) Opt rf and remove unless code ( #30900 )
...
Opt rf and remove unless code
2024-02-18 11:50:16 +08:00
8ff8d94697
[fix](ip) change IPv6 to little-endian byte order storage (like IPv4) ( #30730 )
2024-02-05 21:56:57 +08:00
3315c16383
[enhance](function) refactor from_format_str and support more format ( #30452 )
2024-02-01 19:08:37 +08:00
02a27a587a
remove some unused member function of IFunctionBase ( #30260 )
2024-01-24 09:59:45 +08:00
f7a340a2df
[improve](move-memtable) add cancel method to load stream stub ( #29994 )
2024-01-16 20:23:09 +08:00
d494674ff4
[opt](parquet-reader) Opt parquet decimal type reading. ( #29825 )
2024-01-12 13:58:19 +08:00
0d691c638b
[Feature](profile)Support report runtime workload statistics #29591
2024-01-12 11:59:27 +08:00
7c7dbf15bc
[feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance ( #29736 )
2024-01-12 11:57:16 +08:00
fc4ca712ed
[bugfix](core) using weak ptr in data stream receiver to avoid runtime state is deconstructed ( #29410 )
2024-01-12 11:48:39 +08:00
7287c0ca15
[Opt](exec)(multi-catalog) Opt date type reading. ( #29571 )
2024-01-12 11:48:39 +08:00
48f58510a8
[refactor](tabletwriter) make tablet writer's rpc callback safe, could exit any time ( #29684 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-12 11:46:29 +08:00
c58d18147b
[refactor](resultwriter) rename append_block to write method is more meaningful ( #29635 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-08 00:00:33 +08:00
0b731800a0
[enhancement](group_commit) refector wal manager code ( #29560 )
2024-01-07 18:54:41 +08:00
b0cac0014d
[enhance](FS) Improve FS error code ( #29432 )
2024-01-06 21:17:22 +08:00
e3c9f535dc
[refactor](wal) refactor some wal code ( #29434 )
2024-01-03 14:45:57 +08:00
69a01e0cf5
[improve](move-memtable) skip load stream stub close wait when cancel ( #29427 )
2024-01-02 23:35:50 +08:00
706463781c
[refactor](group commit) refactor group commit wal code ( #29375 )
2024-01-02 15:52:03 +08:00
b7487430da
Revert "[improve](move-memtable) cancel load rapidly when stream close wait ( #29322 )" ( #29371 )
...
This reverts commit bbf58c5aa42d40e66bc6ccc9ed91a4fcb4bdfff7.
2024-01-02 11:32:14 +08:00
bbf58c5aa4
[improve](move-memtable) cancel load rapidly when stream close wait ( #29322 )
2023-12-31 16:26:41 +08:00
7623b5cc31
[cleanup](move-memtable) remove namespace stream_load ( #27441 )
2023-12-30 20:08:23 +08:00
