Commit Graph

6278 Commits

Author SHA1 Message Date
5f66335e54 [feature](pipelineX) add local_shuffle in set_operation / assert_num operator (#28293) 2023-12-13 15:15:20 +08:00
cd6d75e518 [fix](memory) TabletSchema and Schema no longer track memory, only track columns count. (#28149)
TabletSchema and Schema no longer track memory, only track columns count. because cannot accurately track memory size.

TabletMeta MemTracker changed to track TabletSchema columns count.

Segment::_meta_mem_usage Unknown value overflow, causes the value of SegmentMeta MemTracker is similar to -2912341218700198079. So, temporarily put it in experimental type tracker.
2023-12-13 15:06:46 +08:00
613b0bd552 [fix](decimal) fix decimal overflow caused by null value (#28260) 2023-12-13 14:53:39 +08:00
862cb0912a [fix](load) count MemTable::to_block memory in flush tracker (#28312) 2023-12-13 14:49:25 +08:00
764d893cbf Remove unused const variables NUMBER, ZERO in vnumbers_tvf.cpp (#28317)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-12-13 13:16:48 +08:00
78b0fec33a [Fix](Outfile) Support export nested complex type data to orc file format (#28182) 2023-12-13 11:55:27 +08:00
a719d7a222 [fix](memory) Fix LRU Cache of type NUMBER charge (#28175) 2023-12-13 11:15:57 +08:00
b901800963 [Fix] Support BE log sys_log_level modify to take effect dynamically (apache#26060) (#28203) 2023-12-13 11:08:48 +08:00
78cf401c92 [enhancement](err-msg) expose real error msg for invalid delete conf (#28287) 2023-12-13 01:07:31 +08:00
13b9350aeb [Bug](scan)fix some case query timeout of not schedule scanner (#28243)
now in pipeline, when result block queue is empty, will be reschedule, and then choose a batch of scanner,
but sometimes, get_available_thread_slot_num() will return thread_slot_num <= 0, so it's will do nothing,
and then block queue will always empty.
have no chance to reschedule again until query timeout.
2023-12-12 21:00:22 +08:00
ace2b45c37 [fix](decimalv2) avoid crashing caused by decimalv2 arithmetic with check_overflow_for_decimal enabled (#28219) 2023-12-12 20:32:10 +08:00
15f5acf1ae [feature](pipelineX) add local_shuffle in sort partition sort analytic node (#28265) 2023-12-12 18:47:46 +08:00
b821c377b6 [Refact](inverted index) comment out block_column_predicate for inverted index and refact some code (#28281) 2023-12-12 18:12:01 +08:00
f401a9c7ec [fix](pipelineX) fix use-after-free in filter timer queue (#28236) 2023-12-12 17:25:14 +08:00
45b2dbab6a [improve](group commit) Group commit support max filter ratio when rows is less than value in config (#28139) 2023-12-12 16:33:36 +08:00
9861cfc4bc [Fix](Transactional-Hive) Fix transactional hive core dump when TransactionalHiveReader::init_row_filters(). (#28238)
Fix transactional hive core dump when TransactionalHiveReader::init_row_filters().
2023-12-12 14:17:26 +08:00
a5a7ab3c65 [pipelineX](profile) Add debug string if enable profile (#28262) 2023-12-12 11:39:08 +08:00
e49ed3d885 [regression test](memtable) add case for aggregation memtable (#28056)
1. create aggregation table
2. insert some data
3. drop the table and create again
4. modify some parameters for some branch
5. insert some data
6. change the parameters back to its default
2023-12-12 11:14:59 +08:00
5ff110e845 [exec](profile) only build expr debug string enable profile (#28261) 2023-12-12 09:13:37 +08:00
7fba3fcb91 [pipelineX](improvement) block local shuffle sink by mem usage (#28224) 2023-12-11 21:25:31 +08:00
c4e484916b [Fix](table property) Fix table property disable_auto_compaction (#27853) 2023-12-11 20:48:11 +08:00
d4f89389e3 [improve](group commit) Group commit support skip wal (#27957) 2023-12-11 19:38:32 +08:00
877935442f [feature](pipelineX)use markFragments instead of markInstances in pipelineX (#27829) 2023-12-11 17:59:53 +08:00
cff1de29ce [fix](group commit) Fix group commit memory calculation (#28242) 2023-12-11 17:05:26 +08:00
593cc92501 [chore] Change default max segment size to 1GB (#28201) 2023-12-11 14:30:57 +08:00
ac167f493b [fix](join) fix decimal overflow caused by left outer join (#28221)
For left outer join or full outer join, when build side data is empty, null data is output for build side, but nested column data of nullable column is not properly initialized, which may cause decimal arithmetic overflow
2023-12-11 11:51:05 +08:00
e1587537bc [Fix](status) fix unhandled status in exprs #28218
which marked static_cast<void> in https://github.com/apache/doris/pull/23395/files
partially fixed #28160
2023-12-11 11:04:58 +08:00
8f2202c89d [minor](log) Add debug info in operators (#28211) 2023-12-11 10:02:24 +08:00
1e5ff40e17 [refactor](group commit) remove future block (#27720)
Co-authored-by: huanghaibin <284824253@qq.com>
2023-12-11 08:41:51 +08:00
320ddf4987 [pipelineX](improvement) Support multiple instances execution on single tablet (#28178) 2023-12-10 20:18:41 +08:00
485d7db516 [fix](partial update) Fix missing rowsets during doing alignment when flushing memtable due to compaction (#28062) 2023-12-10 12:09:48 +08:00
a3cd36ce60 [bug](cooldown) Fix incorrect remote rowset dir after restarting BE (#28140) 2023-12-10 00:44:01 +08:00
4e86f9bab5 [improve](move-memtable) include and check offset when append data (#28159) 2023-12-09 16:21:36 +08:00
363721e066 [Bug](udf) java-udf function open failed cause BE core dump #28063
when the java-udf open function failed, and some JNI have not set,
so in close function can't call jni.
2023-12-09 11:00:30 +08:00
99be9d6ad3 [fix](memlimiter) refresh memtracker before flush active memtables (#28196)
Currently, _flush_active_memtables() is using stale memtracker data, especially when some other thread has just it.
Refresh memtrackers before flush to avoid this problem.
2023-12-09 01:40:51 +08:00
Pxl
027b06059a [Feature](materialized-view) support count(1) on materialized view (#28135)
support count(1) on materialized view
fix match failed like select k1, sum(k1) from t group by k1
2023-12-09 01:36:46 +08:00
055b3885c9 [Fix](inverted index) fix compound directory flush buffer error (#28191) 2023-12-09 00:57:35 +08:00
abc802b5ba [bugfix](core) child block is shared between operator and node, it should be shared ptr (#28106)
_child_block in nest loop join , table value function, repeat node will be shared between ExecNode and related operator, but it should not be a unique ptr in operator, it belongs to exec node.

It will double free the block, if operator's close method is not called correctly.

It should be a shared ptr, then it will not core even if the opeartor's close method is not called.
2023-12-09 00:18:14 +08:00
18ef131410 [fix](load) select more active memtables at once in memtable limiter (#28171) 2023-12-08 21:45:35 +08:00
573b594df3 [improvement](Variant Type) Support displaying subcolumns expanded for the variant column (#27764) 2023-12-08 20:34:58 +08:00
226a0c3b1d [chore](memory) Warning in log when turning on THP (#28122) 2023-12-08 17:38:38 +08:00
6da36e1077 [feature](merge-cloud) Refactor write path code by abstract base class (#26537)
Refactor write path code by abstract base class. Whether to use `StorageEngine` or `CloudStorageEngine` will be determined during compilation instead of runtime `config::cloud_mode` to avoid unexpected null pointer or undefined behavior issues caused by merging code.

Class that depend on `StorageEngine` but are shared by the cloud mode need to have an abstract base class. Common code should be extracted into the base class, while the code that depends on `StorageEngine` should be implemented in a `StorageEngine` mix-in class of the base class.
2023-12-08 14:50:36 +08:00
341822ec05 [regression-test](Variant) add compaction case for variant and fix bugs (#28066) 2023-12-08 12:18:46 +08:00
59ec3da899 open workload group in PR pipeline (#27744) 2023-12-08 11:56:03 +08:00
ebed055d2b [chore](clone) rename clone request field (#27591) 2023-12-08 11:53:57 +08:00
d534cdf027 [compile](BE) let arm gcc know some function no return (#28157)
let arm gcc know some function no return
2023-12-08 11:32:08 +08:00
0947bf4e97 [opt](mysql serde) Avoid core dump when converting invalid block to mysql result (#28069)
BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in #28030, so we add check branch to avoid BE core dump due to out of range related problem.
2023-12-08 10:21:09 +08:00
25b90eb782 [Feature](function) support random int from specific range (#28076)
mysql> select rand(-20, -10);
+------------------+
| random(-20, -10) |
+------------------+
|              -13 |
+------------------+
1 row in set (0.10 sec)
2023-12-08 10:15:25 +08:00
e75d91c91b [regression-test](Variant) Add more cases related to schema changes (#27958)
* [regression-test](Variant) Add more cases related to schema changes

And fix bugs about schema change for variant:
fix bug schema change crash on doing schema change with tablet schema that contains extracted columns
2023-12-08 10:15:12 +08:00
d8d8f15bf3 [improvement](vectorization) Use requires instead of specialization for doris::vectorized::Decimal (#28027)
Use requires instead of specialization for doris::vectorized::Decimal
2023-12-08 09:59:52 +08:00