Commit Graph

13599 Commits

Author SHA1 Message Date
4ca650f306 [improvement](jdbc catalog) Adjust function replacement order and add new function support (#24685) 2023-09-21 08:45:27 +08:00
5e82aebd25 [fix](fe-meta) fix paimon file catalog meta issue and replication num analysis issue (#24681)
Fix 2 issues:

The replication num analysis will throw exception when there is no BE alive when FE restart.
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = errCode = 2, detailMessage = replication num should be less than the number of available backends. replication num is 1, available backend num is 0
        at org.apache.doris.common.util.PropertyAnalyzer.analyzeReplicaAllocation(PropertyAnalyzer.java:1018) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.TableProperty.buildReplicaAllocation(TableProperty.java:483) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.TableProperty.read(TableProperty.java:529) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.readFields(OlapTable.java:1370) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Table.read(Table.java:383) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Database.readFields(Database.java:631) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.InternalCatalog.loadDb(InternalCatalog.java:3075) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadDb(Env.java:1764) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_341]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_341]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_341]
        at org.apache.doris.persist.meta.MetaReader.read(MetaReader.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadImage(Env.java:1701) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:130) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
Paimon File Catalog meta issue
Fix #23366
2023-09-21 08:41:03 +08:00
b25c3e2197 [Fix](merge-on-write) enable_unique_key_partial_update should be forwarded to master #24697 2023-09-21 08:31:02 +08:00
3650d870d4 [fix](spill) avoid printing too much status stack trace if gc dir does not exist (#24688) 2023-09-21 08:24:59 +08:00
8b1e5897c8 [fix](security): Use SecureRandom instead of Random, because it provides better security #24483 2023-09-21 08:22:16 +08:00
xyf
9ccce61836 [chore](thirdparty)We need to issue an error when starting FE without setting the Java home environment (#23943)
---------

Co-authored-by: yiguolei <676222867@qq.com>
2023-09-21 08:10:36 +08:00
73722ad1cc [fix](tools) tpch-tools and tpcds-tools update #24650
move analyze from run-query script to load-data script
fix some errors in scripts
2023-09-20 23:59:11 +08:00
49d6627e31 [test](pipelinex) disable pipelinex case temporaraly #24699 2023-09-20 21:34:30 +08:00
2e85e0163d Revert "[feature](function) add json->operator convert to json_extract (#19899)" (#24679)
Revert "[feature](function) add json->operator convert to json_extract (#19899)"
because it conflict with lambda syntax
This reverts commit f54a068d82e88e8535f3ed55a4224886b752e46b.
2023-09-20 21:16:19 +08:00
677b3424e3 [feature](Load)(step1)support nereids load, add load grammar (#23485)
support nereids load grammar.

we will convert the broker load stmt to insert into clause:
1. rename broker load to bulk load.
2. add load grammar to nereids optimizer.
3. convert to insert into clause with table value function.

https://github.com/apache/doris/issues/24221
2023-09-20 21:12:23 +08:00
294062d519 [bug](function) fix width bucket function return wrong result (#24673) 2023-09-20 20:37:22 +08:00
8aa39137fc [Bugfix](multi-catalog)If es hosts ends with "/", the index cannot be obtained. (#24272) 2023-09-20 20:35:14 +08:00
e4b551e2ce [fix](Config): Remove unused config max_connection_scheduler_threads_num (#24597) 2023-09-20 18:11:56 +08:00
67e789e025 [Fix](point query) Fix point query unstable (#24570) 2023-09-20 18:07:07 +08:00
2a260be10c [improvement](jdbc catalog) when lower_case_table_names of jdbc catalog properties is set to true, use the real table name to query the jdbc data source (#24520) 2023-09-20 17:47:11 +08:00
Pxl
8c8c563c3c [Chore](status) adjust some error status print log (#24660)
adjust some error status print log
2023-09-20 17:42:51 +08:00
ca5bdc0482 [feature](restore) Add thrift RestoreSnapshotRequest support table alias (#24547)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-09-20 17:33:35 +08:00
81e65f4a12 [feature](function) Support SHA family functions (#24342) 2023-09-20 17:21:45 +08:00
5eb8fe3d6e [improvement](type) modify the inner type display of the Array/Map/Struct type (#24459)
In the old code, when using desc command to view the table schema
It will display as follows
```
ARRAY<TINYINT(4)>
ARRAY<SMALLINT(6)>
ARRAY<INT(11)>
ARRAY<BIGINT(20)>
ARRAY<LARGEINT(40)>
```
However, for normal integer type displays, the width is not displayed
So, I changed it to the following
```
ARRAY<TINYINT>
ARRAY<SMALLINT>
ARRAY<INT>
ARRAY<BIGINT>
ARRAY<LARGEINT>
```
2023-09-20 17:03:03 +08:00
1405b7ca82 [improve](scan) support lower the thread priority of scan thread (#24526)
The configuration item is used to lower the priority of the scanner thread,
typically employed to ensure CPU scheduling for write operations.
2023-09-20 17:00:24 +08:00
862c939ca2 [Bug](pipelineX) fix AnalyticDependency concurrent problem (#24671) 2023-09-20 16:39:49 +08:00
a3fd37da02 [opt](Nereids): speedup performance of Nereids (#24600)
- remove redundant ImmuableList.copyOf()
- GroupExpr equals() don't compare LogicalProperties.
- fix project forgot except
- project constructor use List as children instead of Plan
2023-09-20 16:38:07 +08:00
a34757422c [regression](pipelineX) enable pipelineX test cases (#24670) 2023-09-20 16:32:03 +08:00
a946f99b8c [Fix](regression-test) fix regression-test of export parquet file format (#24450) 2023-09-20 15:41:49 +08:00
4d2a92e47c [community](collaborator) add collaborator wm1581066 for issue and pr label #24500 2023-09-20 15:16:01 +08:00
cf4143f3d1 [testcase](map) update regress test for array/map nested insert into nagetive cases (#24534) 2023-09-20 15:07:29 +08:00
7c32b2b4ed [Fix](broker load) broker load with or predicate error fix #24157
Co-authored-by: wangqingtao6 <wangqingtao6@jd.com>
2023-09-20 14:56:32 +08:00
dc9fa1a4f1 [Refactor](Sink) convert to tablet sink to tablet writer (#24474) 2023-09-20 14:47:18 +08:00
fc12362a6d [feature-wip](arrow-flight)(step2) FE support Arrow Flight server (#24314)
This is a POC, the design documentation will be updated soon
2023-09-20 14:42:54 +08:00
a3361df7b9 [Feat](Nereids) support json and jsonb datatype (#24156)
Feature:
support jsonb and json type in nereids

Document:
this feature supports these two datatype in nereids optimizer like original planner, the sql reference is same as before
[JSON - Apache Doris](https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-reference/Data-Types/JSON)
2023-09-20 14:32:22 +08:00
e9435c14f8 [Improve](array-func)improve array union support multi params (#24327) 2023-09-20 14:29:48 +08:00
ca56921481 [docs](partition) Auto partition docs (#24574) 2023-09-20 14:28:23 +08:00
8aea31e383 [fix](timezone) fix timezone parse when there is no tzfile (#24578) 2023-09-20 14:28:12 +08:00
aa9f2260ea [fix](multi-catalog)Es catalog needs to verify whether it is a valid configuration. (#24309) 2023-09-20 14:20:57 +08:00
df66922bc0 [Chore](sonar)sonar (C++) configuration file name error (#24662)
FYI https://community.sonarsource.com/t/project-root-configuration-file-none/99389
2023-09-20 13:58:30 +08:00
26ca0b2780 Add some block counter (#24465)
Add some block counter (#24465)
2023-09-20 13:23:01 +08:00
deafa2dd88 [fix](Nereids) fix row count unconsistent when join ordering (#24589)
In the context of reorder join, when a new plan is generated, it may include a project operation. In this case, the newly generated join root and the original join root will no longer be in the same group. To avoid inconsistencies in the statistics between these two groups, we keep the child group's row count unchanged when the parent group expression is a project operation.
2023-09-20 13:11:35 +08:00
901ee7a8d3 [regression](pipelineX) disable pipelineX test cases (#24654) 2023-09-20 13:01:08 +08:00
c0df8fca20 [pipelineX](fix) Fix potential concurrent problem (#24651) 2023-09-20 13:00:58 +08:00
c704497d02 [fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)
Fixed bug when parsing multi-character delimiters.
2023-09-20 12:41:35 +08:00
075552ead4 [feature](partitions)support batch delete partition (#23986)
ALTER TABLE example_db.my_table
DROP PARTITION p1,
DROP PARTITION p2,
DROP PARTITION p3;
2023-09-20 11:45:52 +08:00
0fb79e4011 [fix](broker-load) fix file offset for compressed file #24564
Co-authored-by: Kang <kxiao.tiger@gmail.com>
2023-09-20 11:41:52 +08:00
a2e29d171a [enhancement](be-meta) sync rocksdb by default to protect data (#24571)
If performance of user's disks is low, users can change the config
to false, this way users know what would happen if a kernel panic.
2023-09-20 11:41:26 +08:00
b7ca4fcc8d [fix](io): use try with resource make io stream close automatically to avoid resource leak (#24605) 2023-09-20 11:39:03 +08:00
848290d8a8 [Fix](nereids) Support partial update for insert into table (#24594) 2023-09-20 11:35:09 +08:00
b02398ba85 [fix](planner) statement run successful but log error msg in audit log (#24628)
legacy planner will set error msg when throw AnalysisException.
However, in some place, we catch these exception and muted them.
So, we should reset back error msg and error code.
2023-09-20 11:32:47 +08:00
5a0ccd702c [typo](docs) fix error in routine load doc (#24623) 2023-09-20 11:13:14 +08:00
8316aad417 [chore](macOS) Fix linkage errors (#24642)
Issue Number: close #24643
2023-09-20 10:50:10 +08:00
9a4a4c0760 [opt](Nereids)skip unknown col stats check on __internal_scheam and information_schema (#24625)
columns in __internal_scheam and information_schema do not have column stats
2023-09-20 10:48:05 +08:00
c41cadb64d [fix](broker) fix broker read issue (#24635)
The given "length" of broker's pread() method is the buffer length, not the length required from file.
So it may larger than the file length.
So we should return all read data, instead of return EOF when `read()` method return -1

I will add regression test case later when the framework support broker process.
2023-09-20 10:43:16 +08:00