Commit Graph

5595 Commits

Author SHA1 Message Date
927de33166 [config](log) disable StreamLoad log default and enable in regression pipeline (#24354)
disable StreamLoad log default and enable in regression pipeline
2023-09-14 20:47:26 +08:00
3ee89aea35 [Feature](merge-on-write)Support ignore mode for merge-on-write unique table (#21773) 2023-09-14 18:03:51 +08:00
Pxl
35c5d71549 [Improvement](join) some improvement of hash join (#23972)
some improvement of hash join
2023-09-14 17:55:35 +08:00
9c6734e68e [bugfix](index) Fix build index limitations (#24358)
1. skip existed index on column with different id on build index
2. allow build index for CANCELED or FINISHED state
2023-09-14 17:53:22 +08:00
4fbb25bc55 [Enhancement](function) Support date_trunc(date) and use it in auto partition (#24341)
Support date_trunc(date) and use it in auto partition
2023-09-14 16:53:09 +08:00
8e7f7c9566 [fix](profile) move probe time to pull and add LoopGenerateJoin time #24302 2023-09-14 16:41:01 +08:00
d8feca2530 [Enhancement]The page cache can be parameterized by the session variable of fe. (#23981) 2023-09-14 14:28:19 +08:00
1ef22d7f7c [Feature](variant) add variant type (#24170)
Add variant type for metadata Add persistent information for variant, including the path of variant sub-columns, persisting them to the segment footer and tablet schema of the rowset.
2023-09-14 14:21:53 +08:00
049032b4b3 Refactor Slice move/copy ctor && assignment to default (#24169)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-14 14:18:10 +08:00
268c867679 [Improve](serde)replace function_cast from_string to serde (#24087)
Now we can not support streamload with column which is map/array nested map/array
serde can do this now , so we can replace it
Notice. if item data in complex type data is empty we just return error, instead of makeup default value , because now we can not define right default for complex type
2023-09-14 13:53:16 +08:00
ed108d48fa [fix](invert index) fix query use char filter (#24268) 2023-09-14 11:42:47 +08:00
5dede120db [Enhancement](inverted index) fix compound predicates error (#24300) 2023-09-14 09:21:41 +08:00
9b7f041bea [Bug](function) fix explode_json_array_int can't handle min/max values (#24284)
the json str get value maybe beyond max/min of Int64,
so add some check to limit the value, and return the max/min of Int64
2023-09-14 09:20:59 +08:00
539a7c2975 [fix](agg) Add the unimplemented functions in 'AggregateFunctionCountNotNullUnaryOld' (#24310)
Similar to #24211
2023-09-14 09:19:02 +08:00
11afd321cb [fix](es catalog) fix issue with select and insert from es catalog core (#24318)
Issue Number: close #24315

The root cause of this issue is that Elasticsearch's long type allows inserting floats and strings. Doris did not handle these cases when doing type conversion. The current strategy is to take the integer before the decimal point if a float or string is found.
2023-09-13 23:07:31 +08:00
07dd6830e8 [pipelineX](refactor) add union node in pipelineX (#24286) 2023-09-13 20:39:58 +08:00
86aa3802cf [log](config) set streamload record default to enable 2023-09-13 16:32:30 +08:00
ccfc912ec0 [Fix](merge-on-write) Check the returned filtered rows from different replicas (#24191) 2023-09-13 16:03:17 +08:00
563c3f75ff [feature](move-memtable) share delta writer v2 among sinks (#24066) 2023-09-13 14:39:29 +08:00
335064f897 [feature](Nereids) add lambda argument and array_map function (#23598)
add array_map function

SELECT ARRAY_MAP(x->x+1, ARRAY(87, 33, -49))
+----------------------------------------------------------------------+
| array_map([x] -> (x + 1), x#1 of array(87, 33, -49))     |
+----------------------------------------------------------------------+
| [88, 34, -48]                                                                 |
+----------------------------------------------------------------------+
2023-09-13 14:24:16 +08:00
edd711105a [Feature](inverted index) add disjunction for inverted index query (#24263) 2023-09-13 14:19:02 +08:00
e30c3f3a65 [fix](csv_reader)fix bug that Read garbled files caused be crash. (#24164)
fix bug that read garbled files caused be crash.
2023-09-13 14:12:55 +08:00
9916324e9c [fix](FileCache) the logic of selecting the cache path is reversed (#24277)
Bug was introduced by https://github.com/apache/doris/pull/23881/files
The logic of selecting the cache path is reversed, and BE will be crashed when enable file cache.
2023-09-13 13:25:07 +08:00
2f74936382 [FIX](decimalv3) fix decimalv3 with precision cast (#24241)
now we use cast to decimalv3 may has error , because decimalv3 use type precision for translate string

mysql [test]>select cast("9999e-1" as decimal(2,1));
+------------------------------------+
| cast('9999e-1' as DECIMALV3(2, 1)) |
+------------------------------------+
|                              999.9 |
+------------------------------------+
1 row in set (0.01 sec)
this pr will fix this just keep reaction same with mysql

mysql> select cast('9999e-1' as decimalv3(2, 1));
+------------------------------------+
| cast('9999e-1' as DECIMALV3(2, 1)) |
+------------------------------------+
|                                9.9 |
+------------------------------------+
1 row in set (0.07 sec)
2023-09-13 11:35:33 +08:00
c7ae2a7d22 [Refactor & Bugfix](static variables) move some static vairables to exec_env (#24029) 2023-09-13 09:27:03 +08:00
9df72a96f3 [Feature](multi-catalog) Support hadoop viewfs. (#24168)
### Feature

Support hadoop viewfs.

### Test

- Regression tests: 
  - hive viewfs test.
  - tvf viewfs test.

- Broker load with broker and with hdfs tests manually.
2023-09-13 00:20:12 +08:00
c926e8ff9d [Enhancement](Status) use Status to expose the error info more explicitly in FlushToken (#24240) 2023-09-12 19:30:16 +08:00
d8ef9dda59 [feature](merge-cloud) Rewrite FS interface (#23953) 2023-09-12 19:20:25 +08:00
3f7a612e76 [fix](agg) Add the unimplemented functions in AggregateFunctionOldSum. (#24211) 2023-09-12 18:21:33 +08:00
6efeb12237 [chore](log) fix error log not aligned #24233 2023-09-12 18:18:07 +08:00
8d777e64e4 [Improvement](inverted index) return status error when create inverted index reader throwing error (#24223)
Doris will be core when index file error when initialize column reader's index reader, try to catch throwing error and return error status.
2023-09-12 16:34:44 +08:00
d3f1388717 [Feature](partitions) Support auto-partition (#24153)
Co-authored-by: zhangstar333 <2561612514@qq.com>
2023-09-12 15:23:15 +08:00
4bb9a12038 [function](bitmap) support bitmap_remove (#24190) 2023-09-12 14:52:04 +08:00
dbf509edc0 [Debug](scan) Add debug log for find p0 scan coredump in pipeline (#24202) 2023-09-12 12:17:44 +08:00
b602369482 [Refactor](merge-on-write) extract common logic in to Tablet::_get_segment_column_iterator (#24048) 2023-09-12 11:37:36 +08:00
bdacefa734 [Fix](status)Fix leaky abstraction and shield the status code END_OF_FILE from upper layers (#24165) 2023-09-12 11:10:52 +08:00
7aa8cdc1d4 [fix](load) check segment file size when finalize (#24196) 2023-09-12 11:04:10 +08:00
6913d68ba0 [Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist (#24011) 2023-09-12 08:56:46 +08:00
11e052c7a4 [fix](invert index) fix overloaded-virtual compiler warning (#24174) 2023-09-11 23:47:19 +08:00
1228995dec [improvement](segment) reduce memory footprint of column_reader and segment (#24140) 2023-09-11 21:54:00 +08:00
6e28d878b5 [fix](hudi) compatible with hudi spark configuration and support skip merge (#24067)
Fix three bugs:
1. Hudi slice maybe has log files only, so `new Path(filePath)`  will throw errors.
2. Hive column names are lowercase only, so match column names in ignore-case-mode.
3.  Compatible with [Spark Datasource Configs](https://hudi.apache.org/docs/configurations/#Read-Options), so users can add `hoodie.datasource.merge.type=skip_merge` in catalog properties to skip merge logs files.
2023-09-11 19:54:59 +08:00
86a064284e (improvement)[inverted-index] add and optimize checks when IO error occurs. (#24167)
When a disk io error occurs, errors may occur when reading and writing files in the inverted index. This PR adds error checking to prevent empty files from being generated.
2023-09-11 19:10:52 +08:00
dbb9365556 [Enhance](ip)optimize priority_ network matching logic for be (#23795)
Issue Number: close #xxx

If the user has configured the wrong priority_network, direct startup failure to avoid users mistakenly assuming that the configuration is correct
If the user has not configured p_ n. Select only the first IP from the IPv4 list, rather than selecting from all IPs, to avoid users' servers not supporting IPv4
extends #23784
2023-09-11 18:32:31 +08:00
9b4338f66a [refactor](pipelineX) Split init and open for local state (#24166) 2023-09-11 14:50:41 +08:00
8f7e7a7b31 [Fix](signal) fix signal handler (#24144) 2023-09-11 13:18:49 +08:00
134b210c03 [improvement](shutdown) not print thread pool error stack trace when shutdown (#24155)
* [improvement](shutdown) not print thread pool error stack trace when shutdown

when thread pool shutdown, should not print error stack trace, it is very confuse.
arrow flight server should not call shutdown, if it is not enabled, because it will print error stack.
remove service unavailable from thrift because it is useless.
Part of this PR need to pick to 2.0 branch.

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-09-11 12:20:07 +08:00
c94e47583c [fix](join) avoid DCHECK failed in '_filter_data_and_build_output' (#24162)
avoid DCHECK failed in '_filter_data_and_build_output'
2023-09-11 11:54:44 +08:00
cd13f9e8c6 [BUG](view) fix can't create view with lambda function (#23942)
before the lambda function Expr not implement toSqlImpl() function.
so it's call parent function, which is not suit for lambda function.
and will be have error when create view.
2023-09-11 10:04:00 +08:00
0896aefce3 [fix](local exchange) fix bug of accesssing released counter of local data stream receiver (#24148) 2023-09-11 09:52:31 +08:00
a0fcc30764 [Fix](Status) Handle status code correctly and add a new error code ENTRY_NOT_FOUND (#24139) 2023-09-11 09:32:11 +08:00