a2c9ed7be8
[Chore](build) fix some undefined behavior about incomplete type vector #19753
2023-05-18 15:13:45 +08:00
068a32bc49
[Improvement](memory) faststring use Allocator #19762
...
After the outer catch exception, faststring resize reserve build may throw a memory alloc failure exception from the Allocator.
Currently page body compress will catch memory alloc failure exception
2023-05-18 15:00:49 +08:00
7c8b7878cd
[fix](memory) Print all query/load memory before memory GC when memory_debug=true ( #19720 )
2023-05-18 14:55:47 +08:00
160d2be0d8
[minimal](Nereids) add more comments for the rewriter ( #19788 )
...
Only add some comments to the rewriter. Because it is fewer comments before and it's hard to understand for the newbie.
2023-05-18 14:47:25 +08:00
303bee6fa3
[Fix](single replica load) add inverted index copy for single replica load ( #19663 )
...
* [Fix](single replica load) add inverted index copy for single replica load
2023-05-18 14:13:41 +08:00
851886cc18
[minor](datev2) remove datev2 because datev2 is used by default ( #19777 )
2023-05-18 13:36:11 +08:00
e45bc160c9
[fix](mtmv) fix bug that should not write edit log when replaying alter mv ( #19781 )
2023-05-18 13:34:05 +08:00
943e5fb7e5
[improvement](MOW) use seperated cache for mow pk cache ( #19686 )
...
In mow, primary key cache have a big impact on load performance, so we add a new cache type to seperate
it from page cache to make it more flexible in some cases
2023-05-18 13:27:09 +08:00
50370dead9
[fix](load) fix unified load converted failed when forwarding to master ( #19779 )
2023-05-18 12:28:32 +08:00
f43e8cc98f
[regressiontest](unionall) Regression_test_similar_query_boolean ( #19553 )
...
* regression_test_similar_query
* add the ORDER BY
* update ORDER BY to comfirm correctness
---------
Co-authored-by: ZI-MA <chime316@qq.com >
2023-05-18 12:21:32 +08:00
9a99d6cd7e
[improvement-#19739]statement error in data-model ( #19740 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com ’>
2023-05-18 12:07:01 +08:00
62458ed0f4
[enhancement](compaction) not core when init failed ( #19754 )
2023-05-18 12:06:22 +08:00
6a5b590873
[refactor-WIP](TaskWorkerPool) add CreateTableTaskPool class for CREATE_TABLE task ( #19734 )
2023-05-18 11:43:09 +08:00
18c1081659
[fix](nereids) fix some nereids bugs ( #19711 )
...
1. add json_unquote and json_extract functions
2. remove mv releated code in visitPhysicalOlapScan
3. forbid bitmap and hll type for topn node's sort exprs
4. HashDistributionInfo of olap scan node should use the slots from output not the full schema
5. SelectMaterializedIndexWithoutAggregate should use the filter node's output together with the predicate to get the correct mv
6. forbid SimplifyArithmeticRule for decimal type
7. make DecimalLiteral's type and value consistent with each other if the value is decimalv2
8. json_array need support empty argument
2023-05-18 11:33:56 +08:00
f412aec187
[improvement](load)disable shrink memory by default ( #19714 )
...
disable shrink memory by default, it becomes very slow when importing large amounts of data
you can turn on If you think it's necessary
2023-05-18 11:25:39 +08:00
a3f06e5fbd
[git](Nereids): ignore apache parquet. ( #19765 )
2023-05-18 10:54:46 +08:00
fe42e52851
[pipeline](CTE) Support multi stream data sink in pipeline ( #19519 )
2023-05-18 10:34:37 +08:00
88ca4f3e6b
[feature](like) make like regexp used as a sql function ( #19755 )
2023-05-18 10:03:12 +08:00
5707ecd0e5
[typo](docs) modify the dynamic table document ( #19750 )
2023-05-18 09:24:17 +08:00
51a12f4f11
[Doc](map-type) add map sql doc ( #19612 )
2023-05-18 09:09:41 +08:00
7f9cdc41f5
[typo](docs)add new describe for grant ( #19656 )
2023-05-18 09:00:39 +08:00
d5d47703fe
[fix](memory) remove auto option in memory config and optimize memtracker logs #19706
...
fix mem_limit default value
memory_gc_sleep_time_s to memory_gc_sleep_time_ms
LoadChannelMgr::_handle_mem_exceed_limit process_mem_limit to process soft mem limit
fix query mem tracker print
2023-05-18 08:54:03 +08:00
cfab124ddd
[Chore](inverted index) change Status::EndOfFile to just logging info, remove useless print ( #19721 )
2023-05-18 08:44:18 +08:00
6a6be52bc9
[enhancement](merge-on-write) Avoiding unnecessary primary key index traversal ( #19746 )
2023-05-18 08:41:49 +08:00
f04f181249
[Bug](pipeline) RegressionTest failed release resouce cause DCHECK failed #19772
2023-05-18 08:41:32 +08:00
fcd85cfc14
[typo](doc)fix typo in BROKER-LOAD.md #19737
2023-05-18 08:38:12 +08:00
ba1bccc518
[typo](docs)Fix typo in monitor-alert ( #19743 )
2023-05-18 08:37:55 +08:00
b32ee4facc
[typo](doc)fix audit plugin and spark load kerberos for yarn #19749
2023-05-18 08:37:41 +08:00
40909f49cb
[typo](docs) spark load example revise and some other revise ( #19756 )
...
Co-authored-by: zhuwei <zhuwei8421@gmail.com >
2023-05-18 08:37:23 +08:00
098dac20c2
[log](Nereids): add more debug info when check logicalproperties. ( #19763 )
2023-05-18 08:37:10 +08:00
35ad081784
[typo](doc)Example of adding stream load import to current time ( #19760 )
2023-05-18 08:36:39 +08:00
5fa956b0d6
[Bug](pipeline) RegressionTest failed release resouce cause DCHECK failed #19773
2023-05-18 08:35:57 +08:00
4566281cc3
[fix](sink) disable lazy-open partition by default ( #19769 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-05-18 07:28:04 +08:00
c80c4477cf
[Enhancement](broker-load) broker load show stmt support display cluster name if specified ( #19392 )
2023-05-18 00:10:15 +08:00
97d4778ecf
[enhancement](schema) dynamic_partition.time_unit support year ( #19551 )
...
dynamic_partition.time_unit support year
2023-05-17 23:49:15 +08:00
8aa7f0e188
[fix](catalog) fix the include_database_list not in effect ( #19589 )
2023-05-17 22:56:21 +08:00
60d5c82f44
[fix](tvf) fix the inconsistency between tvf backends function and show backends result ( #19697 )
2023-05-17 22:55:46 +08:00
082b7cce41
[improvement](storage) let the storage_page_cache_shard_size conf be rounded up to a power of two ( #19639 )
2023-05-17 22:54:58 +08:00
4ecd1c683d
[doc](fix) cold hot separation doc fix ( #19757 )
2023-05-17 22:39:31 +08:00
79d30cfe46
[feature](compact) Duplicate with no keys tables compaction coredump ( #19490 )
...
Co-authored-by: yuxianbing <yuxianbing@yy.com >
2023-05-17 22:22:14 +08:00
49c6bbce84
[improvement](load) do not create pthread in tablet_sink ( #19465 )
...
add bvar stat for streamload.
2023-05-17 22:05:54 +08:00
2536b57590
[doc](catalog) optimize catalog doc ( #19601 )
2023-05-17 21:45:08 +08:00
be47a27013
[Fix](multi catalog, nereids)Fix FileQueryScanNode couldn't filter partition in nereids planner bug ( #19564 )
...
Nereids planner add conjuncts to ScanNode after call finalize, this may cause external table scan node fail to filter
useless partition, because external table do the partition prune in the finalize method.
This pr is to fix this bug. In the rewrite stage, pass the conjuncts to LogicalFileScan object, and eventually pass to
ScanNode while creating it. So that the ScanNode could use the conjuncts while doing finalize.
Why not doing the partition prune in the LogicalFileScan like LogicalOlapScan doing?
Because Iceberg api doesn't have the partition concept, it just accept a list of Conjuncts,
so it's easier to pass the conjuncts to ScanNode (Hive, Icegerg, Hudi...) and doing the partition prune in there.
2023-05-17 21:39:59 +08:00
dc18da2ce4
[Log](expr) add DCHECK info for expr close DCHECK ( #19683 )
2023-05-17 21:37:38 +08:00
b5f60bde42
[fix](checkpoint)fix Checkpoint error when http server is not ready #19699
2023-05-17 21:33:56 +08:00
2993cdb36e
[fix](multi-catalog)fix iceberg catalog display type #19728
2023-05-17 21:33:18 +08:00
67668905d6
[Improve](complex-type)add complex type support unique table with regress test #19751
...
add complex type support unique table with regress test
struct / map / array now support unique table but no regress test
2023-05-17 21:32:46 +08:00
1d05feea1b
[Feature](Nereids) add executable function to support fold constant for functions ( #18209 )
...
1. Add date-time functions for fold constant for Nereids.
This is the list of executable date-time function nereids supports up to now:
- now()
- now(int)
- current_timestamp()
- current_timestamp(int)
- localtime()
- localtimestamp()
- curdate()
- current_date()
- curtime()
- current_time()
- date_{add/sub}(),{years/months/days/hours/minutes/seconds}_{add/sub}()
- datediff()
- {date/datev2}()
- {year/quarter/month/day/hour/minute/second}()
- dayof{year/month/week}()
- date_format()
- date_trunc()
- from_days()
- last_day()
- to_monday()
- from_unixtime()
- unix_timestamp()
- utc_timestamp()
- to_date()
- to_days()
- str_to_date()
- makedate()
2. solved problem:
- enable datev2/datetimev2 default.
- refactor Nereids foldConstantOnFE and support fold nested expression.
- separate the executable into multi-files for easily-reading and adding new functions
2023-05-17 21:26:31 +08:00
1eb929e1ca
[Bugfix](Jdbc Catalog) fix data type mapping of SQLServer Catalog ( #19525 )
...
We map `money/smallmoney` types of SQLSERVER into decimal type of doris.
2023-05-17 21:02:42 +08:00
30c4f25cb3
[fix](multi-catalog) verify the precision of datetime types for each data source ( #19544 )
...
Fix threes bugs of timestampv2 precision:
1. Hive catalog doesn't set the precision of timestampv2, and can't get the precision from hive metastore, so set the largest precision for timestampv2;
2. Jdbc catalog use datetimev1 to parse timestamp, and convert to timestampv2, so the precision is lost.
3. TVF doesn't use the precision from meta data of file format.
2023-05-17 20:50:15 +08:00
