Commit Graph

18263 Commits

Author SHA1 Message Date
12f936558e [fix](doc) spell errors fixes for debug-point-action (#28152) 2024-01-16 18:30:23 +08:00
1998735432 [Improvement](function) enable ipv6_num_to_string function to support handling of IPv6 type (#29886)
Enable ipv6_num_to_string function to handle IPv6 type normally in addition to handling 16 byte string types
2024-01-16 18:30:23 +08:00
e7b221ba66 [fix](be-ut) Fix unstable test cases (#29896)
The following cases are unstable.

1. LoadStreamMgrTest
2. TaskWorkerPoolTest.PriorTaskWorkerPool

Rationales

1. LoadStreamMgrTest
It is related to timeout. If we investigate the examples in BRPC, we will find the timeout is usually set to 0 rather than a specific number.
2. TaskWorkerPoolTest.PriorTaskWorkerPool
The order of the threads for the lock contentions is undetermined.
2024-01-16 18:30:23 +08:00
88eab1b4b9 [doc](hight-concurrent-point-query) Improve and supplement hight-concurrent-point-query documentation (#29396) 2024-01-16 18:30:23 +08:00
41875a0bf5 [fix](move-memtable) check segment id in add_segment (#29898) 2024-01-16 18:30:23 +08:00
ee66f1563e [fix](Nereids) fix rf push down union (#29847)
Current union rf push down only support rf from parent join, but not support ancestor join.
The pr fixes this problem on project/distribute node's rf pushing down checking.
2024-01-16 18:30:22 +08:00
fd4795dace [opt](Nereids) add graph sql function and one arg truncate (#29864) 2024-01-16 18:30:22 +08:00
f79ec8ea7e [test](regression-test) fix case bug suites/export/test_array_export.groovy (#29783) 2024-01-16 18:30:22 +08:00
06a6477275 [test](regression-test) move test_alter_user.groovy to run nonConcurrent, for it has set global operation (#29772) 2024-01-16 18:30:22 +08:00
5f1b888a24 add 2.1.0-rc04 tag 2024-01-16 14:28:38 +08:00
a34ac7f73a [asf] remove some collaborators 2024-01-15 11:46:31 +08:00
de3fdc7d08 [chore](Fix) Fix uninitilized buffer in read_cluster_id() (#29949) 2024-01-14 15:56:19 +08:00
12af86176a [fix](class-loader) fix class loader conflict on BE side (#29942)
1. make `hadoop-common` in be java extension as `provided`.
2. must load be java extension jars before hadoop jars
2024-01-14 15:53:33 +08:00
7a6475eeee [deps](hadoop) update hadoop on BE side to 3.3.6 (#29939)
Same as on FE side
2024-01-14 15:52:04 +08:00
60f6436f26 [fix](schema cache) adjust the destruction order of _tablet_schema_cache and storage engine (#29923) 2024-01-13 23:36:15 +08:00
e4e57e9b05 [chore](removelogs) remove debug query timeout logs 2024-01-12 14:37:20 +08:00
62064a86bf [test](ut) added UT cases for show create load stmt (#29564) 2024-01-12 13:58:20 +08:00
99024ad7bd [fix](move-memtable) check eos for already closed streams (#29734) 2024-01-12 13:58:20 +08:00
3b25e69311 [bug](rf) fix invalid type for runtime filters when result column is const (#29851) 2024-01-12 13:58:20 +08:00
2a77858845 [fix](move-memtable) check all streams for failed reason (#29877) 2024-01-12 13:58:20 +08:00
a314491535 [Fix](inverted index) fix array inverted index builder error (#29869) 2024-01-12 13:58:19 +08:00
5ef8428345 [Refactor](executor)refactor workload group log fron WARNING to INFO #29878 2024-01-12 13:58:19 +08:00
1718341051 [pipelineX](fix) Fix correctness problem due to local hash shuffle (#29881) 2024-01-12 13:58:19 +08:00
acda8d2129 [feature](profile )merge of profiles can be disabled by profile level. #29861
The merging of profiles requires ensuring the correctness of the profiles themselves. However, if merging is intended for troubleshooting correctness issues through profiles, errors may occur.

Moreover, the 'try-catch' does not catch exceptions related to profile merging. If merging fails, even the normal profile cannot be obtained.
2024-01-12 13:58:19 +08:00
3ef1229635 [docs](query-accel) refine several statements in docs (#29716) 2024-01-12 13:58:19 +08:00
2a51750abd [fix](dynamic partition) fix dynamic partition storage medium not working (#29490) 2024-01-12 13:58:19 +08:00
ad2c13e009 [Optimize](kill-query)Support the scanners exits as soon as possible when kill query #29803 2024-01-12 13:58:19 +08:00
d494674ff4 [opt](parquet-reader) Opt parquet decimal type reading. (#29825) 2024-01-12 13:58:19 +08:00
ad986a78ae [Fix](executor)Fix Grayscale upgrade be code dump when report statistics #29843 2024-01-12 13:58:19 +08:00
d525f576e1 [improve] Use lru cache to count the number of column in tablet schema to control memory (#29668) 2024-01-12 13:58:19 +08:00
0d6ab3c68c [chore](regression test) check disk is good (#29740) 2024-01-12 13:58:19 +08:00
89995402dd [regression](conf) enable load_stream_fault_injection (#29829)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-01-12 13:58:19 +08:00
53639a01fe [Fix] (schema change) fix the bug that non light schema change tables can rename column (#29850) 2024-01-12 13:58:19 +08:00
42c21f7eda [regression](conf) enable test_stream_stub_fault_injection (#29830) 2024-01-12 13:58:19 +08:00
fc5dc1c285 [config](move-memtable) set default load_stream_per_node to 20 (#29822) 2024-01-12 13:58:19 +08:00
cbffdbb8bf [bug](group_commit) fix relay wal problem on materialized-view (#29848) 2024-01-12 13:58:19 +08:00
a4f29193f6 [pipelineX](fix) Fix incorrect runtime filter (#29860) 2024-01-12 13:58:19 +08:00
407a4a285d [improve](load) reduce logs from memtable memory limiter (#29840) 2024-01-12 13:58:19 +08:00
f94ff21eef [ci](perf) adjust threshold (#29856)
Co-authored-by: stephen <hello-stephen@qq.com>
2024-01-12 13:58:19 +08:00
ebfbe0c8dd [opt](information_schema) support information_schema in external catalog (#28919)
Add `information_schema` database for all catalog.
This is useful when using BI tools to connect to Doris,
the tools can get meta info from `information_schema`.

This PR mainly changes:

1. There will be a `information_schema` db in each catalog.
2. Each `information_schema` db only store the meta info of the catalog it belongs to.
3. For `information_schema`, the `TABLE_SCHEMA` column's value is the database name.
4. There is a new global variable `show_full_dbname_in_info_schema_db`, default is false, if set to true,
    The `TABLE_SCHEMA` column's value is the like `ctl.db`, because:

	When connect to Doris, the `database` info in connection url will be: `xxx?db=ctl.db`.
	
	And then some BI will try to query `information_schema` with sql like:
	
	`select * from information_schema.columns where TABLE_SCHEMA = "ctl.db"`
	
	So it has to be format as `ctl.db`
	
	eg, the `information_schema.columns` table in external catalog `doris` is like:
	
	```
	mysql> select * from information_schema.columns limit 1\G
	*************************** 1. row ***************************
	           TABLE_CATALOG: doris
	            TABLE_SCHEMA: doris.__internal_schema
	              TABLE_NAME: column_statistics
	             COLUMN_NAME: id
	        ORDINAL_POSITION: 1
	          COLUMN_DEFAULT: NULL
	             IS_NULLABLE: NO
	               DATA_TYPE: varchar
	CHARACTER_MAXIMUM_LENGTH: 4096
	  CHARACTER_OCTET_LENGTH: 16384
	       NUMERIC_PRECISION: NULL
	           NUMERIC_SCALE: NULL
	      DATETIME_PRECISION: NULL
	      CHARACTER_SET_NAME: NULL
	          COLLATION_NAME: NULL
	             COLUMN_TYPE: varchar(4096)
	              COLUMN_KEY:
	                   EXTRA:
	              PRIVILEGES:
	          COLUMN_COMMENT:
	             COLUMN_SIZE: 4096
	          DECIMAL_DIGITS: NULL
	   GENERATION_EXPRESSION: NULL
	                  SRS_ID: NULL
	```
	
6. Modify the behavior of

	- show tables
	- shwo databases
	- show columns
	- show table status

	The above statements may query the `information_schema` db if there is `where` predicate after them
2024-01-12 13:58:19 +08:00
d3721455b0 [Session](rf) Change the default min size of bf runtime filter (#29837) 2024-01-12 13:58:19 +08:00
f67a00ffbb [opt](nereids) prune runtime redundant filters (#29828)
1. expand_runtime_filter_by_inner_join will create some redundant rfs,e.g., tpch q5 and q9, we need to remove one
2. hive: prune rf if target only used as probe
2024-01-12 13:58:19 +08:00
ed3c8bba87 [fix](auth)remove the key when priv is empty (#29522)
- remove the key when priv is empty
- check priv when create mv
2024-01-12 13:58:19 +08:00
8ba1eb0b02 [feature](mtmv) task tvf add queryId (#29671)
To better locate abnormal situations, add queryId
2024-01-12 12:00:32 +08:00
4d97f8ea75 [enhance](function) support two special format for str_to_date (#29823) 2024-01-12 12:00:32 +08:00
eed72a101e [fix](Nereids) decimalv3 cast in fe produce wrong data (#29808)
case:
```
MySQL root@127.0.0.1:test> select cast(12 as decimalv3(2,1))
+-----------------------------+
| cast(12 as DECIMALV3(2, 1)) |
+-----------------------------+
| 12.0                        |
+-----------------------------+
```

decimalv2 literal will generate wrong result too. But it is not only
bugs in planner, but also have bugs in executor. So we need fix executor
bug in another PR.
2024-01-12 12:00:13 +08:00
22c134fa0a [fix](rowset-reader) direct mode shouldn't use merge iterator (#29678) 2024-01-12 11:59:52 +08:00
885d8b28ba [fix](Nerids): fix function deps when check unique and not null #29797 2024-01-12 11:59:52 +08:00
f02fb5d49e [fix](vec) wrong implementation of operator <=> of Field (#29743) 2024-01-12 11:59:52 +08:00
18f850c94f [enhance](auto-partition) forbid null column for auto partition (#29749) 2024-01-12 11:59:52 +08:00