Commit Graph

2718 Commits

Author SHA1 Message Date
5e82aebd25 [fix](fe-meta) fix paimon file catalog meta issue and replication num analysis issue (#24681)
Fix 2 issues:

The replication num analysis will throw exception when there is no BE alive when FE restart.
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = errCode = 2, detailMessage = replication num should be less than the number of available backends. replication num is 1, available backend num is 0
        at org.apache.doris.common.util.PropertyAnalyzer.analyzeReplicaAllocation(PropertyAnalyzer.java:1018) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.TableProperty.buildReplicaAllocation(TableProperty.java:483) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.TableProperty.read(TableProperty.java:529) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.readFields(OlapTable.java:1370) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Table.read(Table.java:383) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Database.readFields(Database.java:631) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.InternalCatalog.loadDb(InternalCatalog.java:3075) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadDb(Env.java:1764) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_341]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_341]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_341]
        at org.apache.doris.persist.meta.MetaReader.read(MetaReader.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadImage(Env.java:1701) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:130) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
Paimon File Catalog meta issue
Fix #23366
2023-09-21 08:41:03 +08:00
2e85e0163d Revert "[feature](function) add json->operator convert to json_extract (#19899)" (#24679)
Revert "[feature](function) add json->operator convert to json_extract (#19899)"
because it conflict with lambda syntax
This reverts commit f54a068d82e88e8535f3ed55a4224886b752e46b.
2023-09-20 21:16:19 +08:00
e4b551e2ce [fix](Config): Remove unused config max_connection_scheduler_threads_num (#24597) 2023-09-20 18:11:56 +08:00
2a260be10c [improvement](jdbc catalog) when lower_case_table_names of jdbc catalog properties is set to true, use the real table name to query the jdbc data source (#24520) 2023-09-20 17:47:11 +08:00
81e65f4a12 [feature](function) Support SHA family functions (#24342) 2023-09-20 17:21:45 +08:00
fc12362a6d [feature-wip](arrow-flight)(step2) FE support Arrow Flight server (#24314)
This is a POC, the design documentation will be updated soon
2023-09-20 14:42:54 +08:00
ca56921481 [docs](partition) Auto partition docs (#24574) 2023-09-20 14:28:23 +08:00
075552ead4 [feature](partitions)support batch delete partition (#23986)
ALTER TABLE example_db.my_table
DROP PARTITION p1,
DROP PARTITION p2,
DROP PARTITION p3;
2023-09-20 11:45:52 +08:00
5a0ccd702c [typo](docs) fix error in routine load doc (#24623) 2023-09-20 11:13:14 +08:00
14bd290aec [feature](jsonb)support json_length and json_contains function (#24332) 2023-09-20 10:40:44 +08:00
e59aa49f28 [feature](datetime-func)support milliseconds_add/sub/diff and microseconds_diff (#24114) 2023-09-20 10:38:56 +08:00
3f0d9d182f [doc](optimizer) Rewrite docs of statistics according to the current master (#24438)
Remove verbsoe sample, stale grammar, deleted function
And add some use advices
2023-09-18 18:25:43 +08:00
b4432ce577 [Feature](statistics)Support external table analyze partition (#24154)
Enable collect partition level stats for hive external table.
2023-09-18 14:59:26 +08:00
ae0b58fcde [typo](doc)modify error word (#24456) 2023-09-18 11:27:52 +08:00
990d6c02ec [Feature](new function) Add a uuid-numeric function, returns uuid in largerint type, 20x faster than uuid (#24395) 2023-09-16 18:26:13 +08:00
0ccb032d79 [parameter](query timeout) change default query timeout to 15min (#24480)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-09-16 18:17:58 +08:00
ed8db3727c [feature](partial update) support MOW partial update for insert statement (#21597) 2023-09-16 17:11:59 +08:00
c270a2d89f [typo] (docs)Row storage mode can only be turned on when creating a table (#24257) 2023-09-15 17:40:46 +08:00
ddc0eb3508 [doc](flink-connector) add json string write (#24422) 2023-09-15 14:08:02 +08:00
29fe87982f [improve](outfile) add file_suffix options for outfile (#24334) 2023-09-15 12:58:41 +08:00
f57c75f935 [doc](agg_state) add agg_state in data-model (#24348)
add agg_state in data-model
2023-09-15 11:45:21 +08:00
320f1e9bbf [improve](routineload) improve show routine load output (#24264) 2023-09-15 10:22:47 +08:00
f61e6483bf [enhancement](broker-load) support compress type for old broker load, and split compress type from file format (#23882) 2023-09-14 21:42:28 +08:00
0488c87a38 [typo][doc]modify broken link. (#23988) 2023-09-14 21:41:40 +08:00
07720d3ff9 [feature](replica version) Add admin set replica version statement (#23706) 2023-09-14 21:12:00 +08:00
927de33166 [config](log) disable StreamLoad log default and enable in regression pipeline (#24354)
disable StreamLoad log default and enable in regression pipeline
2023-09-14 20:47:26 +08:00
3ee89aea35 [Feature](merge-on-write)Support ignore mode for merge-on-write unique table (#21773) 2023-09-14 18:03:51 +08:00
85e5b49d4c update structured streaming doc (#24016) 2023-09-14 17:47:27 +08:00
51a5895464 [Feature](RoutineLoad) Support max filter ratio for routine load (#24035) 2023-09-14 15:30:40 +08:00
51811f72b3 [doc](partial update) add complete doc for partial update (#24317) 2023-09-14 14:49:24 +08:00
25763d0ed9 [typo](doc) fix error description about load_to_single_tablet (#24270)
* fix error description about load_to_single_tablet

* fix review
2023-09-14 11:48:06 +08:00
64337a8698 [Improve](metadata)Start the script to set metadata_failure_recovery (#24308) 2023-09-14 10:02:35 +08:00
86aa3802cf [log](config) set streamload record default to enable 2023-09-13 16:32:30 +08:00
a6f05e89f5 [doc](catalog) update cache refresh doc (#24183)
Some cache refresh doc is missing
2023-09-13 11:45:26 +08:00
1f769291b5 [doc](invert index) add invert index char_filter doc (#24205) 2023-09-13 10:02:45 +08:00
9df72a96f3 [Feature](multi-catalog) Support hadoop viewfs. (#24168)
### Feature

Support hadoop viewfs.

### Test

- Regression tests: 
  - hive viewfs test.
  - tvf viewfs test.

- Broker load with broker and with hdfs tests manually.
2023-09-13 00:20:12 +08:00
4bb9a12038 [function](bitmap) support bitmap_remove (#24190) 2023-09-12 14:52:04 +08:00
aa850fc9c3 [doc](hive) add faq for multi delimiter config (#24179)
when a user sets `serde` as `org.apache.hadoop.hive.contrib.serde2.MultiDelimitserDe`, we should add a configuration in hive-site.xml
2023-09-12 10:34:23 +08:00
0c30fff811 add add navigator for vector distance functions (#24081) 2023-09-11 19:55:36 +08:00
6e28d878b5 [fix](hudi) compatible with hudi spark configuration and support skip merge (#24067)
Fix three bugs:
1. Hudi slice maybe has log files only, so `new Path(filePath)`  will throw errors.
2. Hive column names are lowercase only, so match column names in ignore-case-mode.
3.  Compatible with [Spark Datasource Configs](https://hudi.apache.org/docs/configurations/#Read-Options), so users can add `hoodie.datasource.merge.type=skip_merge` in catalog properties to skip merge logs files.
2023-09-11 19:54:59 +08:00
229ee50c93 [Docs](StreamLoad)Add partial columns docs (#24184) 2023-09-11 17:16:29 +08:00
8ae7e67623 [Docs](Ldap)Add Jdbc connect docs (#24181) 2023-09-11 15:53:37 +08:00
7abd88f1b4 remove editlogport in frontrnds disks (#24047) 2023-09-11 12:38:56 +08:00
cd13f9e8c6 [BUG](view) fix can't create view with lambda function (#23942)
before the lambda function Expr not implement toSqlImpl() function.
so it's call parent function, which is not suit for lambda function.
and will be have error when create view.
2023-09-11 10:04:00 +08:00
a27349c83a [fix](Export) Concatenation the outfile sql for Export (#23635)
In the original logic, the `Export` statement generates `Selectstmt` for execution. But there is no way to make the `SelectStmt` use the new optimizer.

Now, we change the `Export` statement to generate the `outfile SQL`, and then use the new optimizer to parse the SQL so that outfile can use the new optimizer.
2023-09-08 10:20:18 +08:00
20b3e5eafe [feature](Datetime) add from_microsecond / from_millisecond function (#23902) 2023-09-07 19:03:49 +08:00
7adab1138c [improvement](index) support CANCEL BUILD INDEX (#23760)
Currently, BUILD INDEX can not be canceled. 
This pr add CANCEL BUILD INDEX as follows:
`CANCEL BUILD INDEX index_name ON table_name [(jobid1,jobid2)]`
2023-09-07 14:48:13 +08:00
xy
45dfbeb8b3 [feature](agg) add the aggregation function 'array_agg' #23474
This function requires one arguments just as ARRAY_AGG(col) and col means the column whose values you want to aggregate.
This function Aggregates the values including NULL in a column into an array and returns a value of the ARRAY data type.
2023-09-07 10:21:55 +08:00
Pxl
a96adc01aa [Chore](function) refactor of quantile_state (#23862)
refactor of quantile_state
2023-09-06 15:39:19 +08:00
728ee90462 [improvement](deploy) Forbid LocalDeployManager drop node (#23875)
Forbid LocalDeployManager drop nodes to prevent errors in the cluster.info file from causing nodes to be dropped.
2023-09-06 08:58:25 +08:00