Commit Graph

4533 Commits

Author SHA1 Message Date
da66a64e09 [fix](merge-on-write) return error st if check_pk_in_pre_segments failed (#19736) 2023-05-23 11:04:15 +08:00
fe111207a9 [Fix](lazy_open) Fix lazy open null point (#19829) 2023-05-23 09:17:46 +08:00
30417e06d4 [enhance](fs) use bvar to monitor s3 file reader& writer (#19607)
remove useless prefix
2023-05-22 23:01:21 +08:00
3dcdadcea6 [Improvement](function) support decimalv3 for function least and greatest (#19931) 2023-05-22 22:48:44 +08:00
53ba46e404 [Fix][Refactor] Fix 'not member call on null pointer of type 'doris::TextConverter' error in ubsan env and refactor text converter. (#19849)
Fix 'not member call on null pointer of type doris::TextConverter' error in ubsan env and refactor text converter.
2023-05-22 21:00:19 +08:00
6762af3c9b [Improve](struct)improve struct support into outfile (#19894)
support select into outfile for struct type
2023-05-22 18:45:56 +08:00
Pxl
9945067e3c [Bug](function) make VcompoundPred optimization work well (#19870)
make VcompoundPred optimization work well
#19818 this pr try to enable VcompoundPred optimization but get wrong result on tpcds q28.
The reason is some nullable logic on mysql need special handling.

mysql [regression_test_tpcds_sf1_p1]>select null and false;
+----------------+
| NULL AND FALSE |
+----------------+
|              0 |
+----------------+
1 row in set (0.00 sec)

mysql [regression_test_tpcds_sf1_p1]>select null and true;
+---------------+
| NULL AND TRUE |
+---------------+
| NULL          |
+---------------+
1 row in set (0.00 sec)

mysql [regression_test_tpcds_sf1_p1]>select null or false;
+---------------+
| NULL OR FALSE |
+---------------+
| NULL          |
+---------------+
1 row in set (0.00 sec)

mysql [regression_test_tpcds_sf1_p1]>select null or true;
+--------------+
| NULL OR TRUE |
+--------------+
|            1 |
+--------------+
1 row in set (0.00 sec)
2023-05-22 18:32:17 +08:00
Pxl
d64be9565d [Bug](function) fix function in get wrong result when input const column (#19791)
fix function in get wrong result when input const column
2023-05-22 10:58:29 +08:00
f57b6adba9 [improvement](libhdfs) Use keytab and principal to login kerberos (#19841)
User keytab and princpal to login kerberos.
And user does not need to execute kinit manually anymore.
2023-05-22 10:54:21 +08:00
a6bd014b8a [FIX](serde)pb ut is not stable #19907 2023-05-22 09:03:02 +08:00
1c950d6930 [fix](config) fix memory config enable_query_memroy_overcommit spell problem #19898 2023-05-22 00:32:20 +08:00
76c358b3e3 [revert](memory) revert page no use Allocator && default disable ChunkAllocator (#19905)
default chunk allocator reserve is 0. At this time, it is meaningless to enable chunk allocator, it will only waste memory.
2023-05-21 22:16:41 +08:00
33fd965b5c [feature-wip](resouce-group) Supports memory soft isolation of resource group (#19802)
create resource groups name properties(
    'enable_memory_overcommit' = 'true' // whether to enable memory soft isolation
)
2023-05-21 19:33:57 +08:00
512806f902 [fix](ubsan) UBSAN avoid thread local switch 2023-05-20 07:14:43 +08:00
47509e65c9 [fix](memory)page no use Allocator, avoid ckbench oom (#19877) 2023-05-19 21:23:31 +08:00
5547bbbaef [decimalv3](function) support function width_bucket (#19806) 2023-05-19 20:28:59 +08:00
abde8bf26a [chore](build) Fix the compilation errors on macOS (arm64) (#19859)
Some errors raise when building the codebase on macOS (arm64).
2023-05-19 18:50:47 +08:00
65807f888b [fix](memory) Remind log if vm/overcommit_memory=2 when be start (#19795)
Expect vm overcommit memory value to be 1, system will no longer throw bad_alloc, memory alloc are always accepted,
memory limit check is handed over to Doris Allocator, make sure throw exception position is controllable,
otherwise bad_alloc can be thrown anywhere and it will be difficult to achieve exception safety.
2023-05-19 15:01:08 +08:00
7d1844d380 [FIX](Array)fix be master comapitible with fe1.2 #19850
when upgrade be first , fe is 1.2 , array contains_nulls is set not in thrift ,so would has core in be
Issue Number: close #xxx
2023-05-19 14:10:29 +08:00
c4900eb658 [Bug](DecimalV3) fix decimalv3 functions (#19801) 2023-05-19 14:10:01 +08:00
609b20bd02 [Feature](planner) use partial update in update from & delete from (#19262) 2023-05-19 09:46:29 +08:00
3e010bbee7 [improvement](profile) add profile counter 'BytesSent' for VDataBufferSender (#19826) 2023-05-19 08:46:50 +08:00
1d01136b1b [Fix](parquet-reader) Fix partition field conjuncts not work. (#19837)
Fix partition field conjuncts not work.
Add predicate_partition_columns in _slot_id_to_filter_conjuncts(single slot conjuncts) to _filter_conjuncts, others should had been added from not_single_slot_filter_conjuncts.
2023-05-19 08:44:02 +08:00
f32deb18e9 [Update](build) change clucene from thirdparty to git module (#19352) 2023-05-19 08:25:51 +08:00
3d6a13605d [improvement](stacktrace) do not captute stack trace for txn error codes (#19817) 2023-05-18 23:58:56 +08:00
481e9aebdb [Refactor](spark load) remove parquet scanner (#19251) 2023-05-18 19:19:13 +08:00
ef0657c072 [Bug](pipeline) RegressionTest failed release resouce cause DCHECK failed (#19783)
RegressionTest failed release resouce cause DCHECK failed
2023-05-18 18:57:25 +08:00
e242d7dfcc [refactor-WIP](TaskWorkerPool) add DropTableTaskPool for DROP_TABLE task (#19793) 2023-05-18 18:25:13 +08:00
07bbf741fb [enhence](memory) gc inverted index cache when there is not enough memory (#19622)
Support to gc inverted index cache when there is not enough memory.
previous problem: The inverted index cache (InvertedIndexSearcherCache and InvertedIndexQueryCache) may use 20% memory which can't be released.
2023-05-18 16:41:51 +08:00
fd4fa5c64e [Optimize](row store) optimize serialization and deserialization (#19691)
1. Get DataTypeSerde in advance to avoid get temporary DataTypeSerde iterate each column
2. Iterate the original row once is enoungh for deserializing by introducing a map for record the index of each column's unique id
2023-05-18 16:22:38 +08:00
294599ee45 [feature](jsonb) rename JSONB type name and function name to JSON (#19774)
To be more compatible with MySQL, rename JSONB type name and function name to JSON.

The old JSONB type name and jsonb_xx function can still be used for backward compatibility.

There is a function jsonb_extract remained since json_extract is used by json string function and more work need to change it. It will be changed further.
2023-05-18 16:16:52 +08:00
068a32bc49 [Improvement](memory) faststring use Allocator #19762
After the outer catch exception, faststring resize reserve build may throw a memory alloc failure exception from the Allocator.

Currently page body compress will catch memory alloc failure exception
2023-05-18 15:00:49 +08:00
7c8b7878cd [fix](memory) Print all query/load memory before memory GC when memory_debug=true (#19720) 2023-05-18 14:55:47 +08:00
303bee6fa3 [Fix](single replica load) add inverted index copy for single replica load (#19663)
* [Fix](single replica load) add inverted index copy for single replica load
2023-05-18 14:13:41 +08:00
851886cc18 [minor](datev2) remove datev2 because datev2 is used by default (#19777) 2023-05-18 13:36:11 +08:00
943e5fb7e5 [improvement](MOW) use seperated cache for mow pk cache (#19686)
In mow, primary key cache have a big impact on load performance, so we add a new cache type to seperate
it from page cache to make it more flexible in some cases
2023-05-18 13:27:09 +08:00
62458ed0f4 [enhancement](compaction) not core when init failed (#19754) 2023-05-18 12:06:22 +08:00
6a5b590873 [refactor-WIP](TaskWorkerPool) add CreateTableTaskPool class for CREATE_TABLE task (#19734) 2023-05-18 11:43:09 +08:00
f412aec187 [improvement](load)disable shrink memory by default (#19714)
disable shrink memory by default, it becomes very slow when importing large amounts of data
you can turn on If you think it's necessary
2023-05-18 11:25:39 +08:00
fe42e52851 [pipeline](CTE) Support multi stream data sink in pipeline (#19519) 2023-05-18 10:34:37 +08:00
88ca4f3e6b [feature](like) make like regexp used as a sql function (#19755) 2023-05-18 10:03:12 +08:00
d5d47703fe [fix](memory) remove auto option in memory config and optimize memtracker logs #19706
fix mem_limit default value
memory_gc_sleep_time_s to memory_gc_sleep_time_ms
LoadChannelMgr::_handle_mem_exceed_limit process_mem_limit to process soft mem limit
fix query mem tracker print
2023-05-18 08:54:03 +08:00
cfab124ddd [Chore](inverted index) change Status::EndOfFile to just logging info, remove useless print (#19721) 2023-05-18 08:44:18 +08:00
6a6be52bc9 [enhancement](merge-on-write) Avoiding unnecessary primary key index traversal (#19746) 2023-05-18 08:41:49 +08:00
f04f181249 [Bug](pipeline) RegressionTest failed release resouce cause DCHECK failed #19772 2023-05-18 08:41:32 +08:00
5fa956b0d6 [Bug](pipeline) RegressionTest failed release resouce cause DCHECK failed #19773 2023-05-18 08:35:57 +08:00
4566281cc3 [fix](sink) disable lazy-open partition by default (#19769)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-05-18 07:28:04 +08:00
082b7cce41 [improvement](storage) let the storage_page_cache_shard_size conf be rounded up to a power of two (#19639) 2023-05-17 22:54:58 +08:00
79d30cfe46 [feature](compact) Duplicate with no keys tables compaction coredump (#19490)
Co-authored-by: yuxianbing <yuxianbing@yy.com>
2023-05-17 22:22:14 +08:00
49c6bbce84 [improvement](load) do not create pthread in tablet_sink (#19465)
add bvar stat for streamload.
2023-05-17 22:05:54 +08:00