Commit Graph

17549 Commits

Author SHA1 Message Date
60f6436f26 [fix](schema cache) adjust the destruction order of _tablet_schema_cache and storage engine (#29923) 2024-01-13 23:36:15 +08:00
e4e57e9b05 [chore](removelogs) remove debug query timeout logs 2024-01-12 14:37:20 +08:00
62064a86bf [test](ut) added UT cases for show create load stmt (#29564) 2024-01-12 13:58:20 +08:00
99024ad7bd [fix](move-memtable) check eos for already closed streams (#29734) 2024-01-12 13:58:20 +08:00
3b25e69311 [bug](rf) fix invalid type for runtime filters when result column is const (#29851) 2024-01-12 13:58:20 +08:00
2a77858845 [fix](move-memtable) check all streams for failed reason (#29877) 2024-01-12 13:58:20 +08:00
a314491535 [Fix](inverted index) fix array inverted index builder error (#29869) 2024-01-12 13:58:19 +08:00
5ef8428345 [Refactor](executor)refactor workload group log fron WARNING to INFO #29878 2024-01-12 13:58:19 +08:00
1718341051 [pipelineX](fix) Fix correctness problem due to local hash shuffle (#29881) 2024-01-12 13:58:19 +08:00
acda8d2129 [feature](profile )merge of profiles can be disabled by profile level. #29861
The merging of profiles requires ensuring the correctness of the profiles themselves. However, if merging is intended for troubleshooting correctness issues through profiles, errors may occur.

Moreover, the 'try-catch' does not catch exceptions related to profile merging. If merging fails, even the normal profile cannot be obtained.
2024-01-12 13:58:19 +08:00
3ef1229635 [docs](query-accel) refine several statements in docs (#29716) 2024-01-12 13:58:19 +08:00
2a51750abd [fix](dynamic partition) fix dynamic partition storage medium not working (#29490) 2024-01-12 13:58:19 +08:00
ad2c13e009 [Optimize](kill-query)Support the scanners exits as soon as possible when kill query #29803 2024-01-12 13:58:19 +08:00
d494674ff4 [opt](parquet-reader) Opt parquet decimal type reading. (#29825) 2024-01-12 13:58:19 +08:00
ad986a78ae [Fix](executor)Fix Grayscale upgrade be code dump when report statistics #29843 2024-01-12 13:58:19 +08:00
d525f576e1 [improve] Use lru cache to count the number of column in tablet schema to control memory (#29668) 2024-01-12 13:58:19 +08:00
0d6ab3c68c [chore](regression test) check disk is good (#29740) 2024-01-12 13:58:19 +08:00
89995402dd [regression](conf) enable load_stream_fault_injection (#29829)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-01-12 13:58:19 +08:00
53639a01fe [Fix] (schema change) fix the bug that non light schema change tables can rename column (#29850) 2024-01-12 13:58:19 +08:00
42c21f7eda [regression](conf) enable test_stream_stub_fault_injection (#29830) 2024-01-12 13:58:19 +08:00
fc5dc1c285 [config](move-memtable) set default load_stream_per_node to 20 (#29822) 2024-01-12 13:58:19 +08:00
cbffdbb8bf [bug](group_commit) fix relay wal problem on materialized-view (#29848) 2024-01-12 13:58:19 +08:00
a4f29193f6 [pipelineX](fix) Fix incorrect runtime filter (#29860) 2024-01-12 13:58:19 +08:00
407a4a285d [improve](load) reduce logs from memtable memory limiter (#29840) 2024-01-12 13:58:19 +08:00
f94ff21eef [ci](perf) adjust threshold (#29856)
Co-authored-by: stephen <hello-stephen@qq.com>
2024-01-12 13:58:19 +08:00
ebfbe0c8dd [opt](information_schema) support information_schema in external catalog (#28919)
Add `information_schema` database for all catalog.
This is useful when using BI tools to connect to Doris,
the tools can get meta info from `information_schema`.

This PR mainly changes:

1. There will be a `information_schema` db in each catalog.
2. Each `information_schema` db only store the meta info of the catalog it belongs to.
3. For `information_schema`, the `TABLE_SCHEMA` column's value is the database name.
4. There is a new global variable `show_full_dbname_in_info_schema_db`, default is false, if set to true,
    The `TABLE_SCHEMA` column's value is the like `ctl.db`, because:

	When connect to Doris, the `database` info in connection url will be: `xxx?db=ctl.db`.
	
	And then some BI will try to query `information_schema` with sql like:
	
	`select * from information_schema.columns where TABLE_SCHEMA = "ctl.db"`
	
	So it has to be format as `ctl.db`
	
	eg, the `information_schema.columns` table in external catalog `doris` is like:
	
	```
	mysql> select * from information_schema.columns limit 1\G
	*************************** 1. row ***************************
	           TABLE_CATALOG: doris
	            TABLE_SCHEMA: doris.__internal_schema
	              TABLE_NAME: column_statistics
	             COLUMN_NAME: id
	        ORDINAL_POSITION: 1
	          COLUMN_DEFAULT: NULL
	             IS_NULLABLE: NO
	               DATA_TYPE: varchar
	CHARACTER_MAXIMUM_LENGTH: 4096
	  CHARACTER_OCTET_LENGTH: 16384
	       NUMERIC_PRECISION: NULL
	           NUMERIC_SCALE: NULL
	      DATETIME_PRECISION: NULL
	      CHARACTER_SET_NAME: NULL
	          COLLATION_NAME: NULL
	             COLUMN_TYPE: varchar(4096)
	              COLUMN_KEY:
	                   EXTRA:
	              PRIVILEGES:
	          COLUMN_COMMENT:
	             COLUMN_SIZE: 4096
	          DECIMAL_DIGITS: NULL
	   GENERATION_EXPRESSION: NULL
	                  SRS_ID: NULL
	```
	
6. Modify the behavior of

	- show tables
	- shwo databases
	- show columns
	- show table status

	The above statements may query the `information_schema` db if there is `where` predicate after them
2024-01-12 13:58:19 +08:00
d3721455b0 [Session](rf) Change the default min size of bf runtime filter (#29837) 2024-01-12 13:58:19 +08:00
f67a00ffbb [opt](nereids) prune runtime redundant filters (#29828)
1. expand_runtime_filter_by_inner_join will create some redundant rfs,e.g., tpch q5 and q9, we need to remove one
2. hive: prune rf if target only used as probe
2024-01-12 13:58:19 +08:00
ed3c8bba87 [fix](auth)remove the key when priv is empty (#29522)
- remove the key when priv is empty
- check priv when create mv
2024-01-12 13:58:19 +08:00
8ba1eb0b02 [feature](mtmv) task tvf add queryId (#29671)
To better locate abnormal situations, add queryId
2024-01-12 12:00:32 +08:00
4d97f8ea75 [enhance](function) support two special format for str_to_date (#29823) 2024-01-12 12:00:32 +08:00
eed72a101e [fix](Nereids) decimalv3 cast in fe produce wrong data (#29808)
case:
```
MySQL root@127.0.0.1:test> select cast(12 as decimalv3(2,1))
+-----------------------------+
| cast(12 as DECIMALV3(2, 1)) |
+-----------------------------+
| 12.0                        |
+-----------------------------+
```

decimalv2 literal will generate wrong result too. But it is not only
bugs in planner, but also have bugs in executor. So we need fix executor
bug in another PR.
2024-01-12 12:00:13 +08:00
22c134fa0a [fix](rowset-reader) direct mode shouldn't use merge iterator (#29678) 2024-01-12 11:59:52 +08:00
885d8b28ba [fix](Nerids): fix function deps when check unique and not null #29797 2024-01-12 11:59:52 +08:00
f02fb5d49e [fix](vec) wrong implementation of operator <=> of Field (#29743) 2024-01-12 11:59:52 +08:00
18f850c94f [enhance](auto-partition) forbid null column for auto partition (#29749) 2024-01-12 11:59:52 +08:00
c9a949130b [Case](wal) Add wal group commit sink case with low disk space fault injection (#29731) 2024-01-12 11:59:52 +08:00
8c0b046ad4 [case](wal)Add wal backpressure case (#29725) 2024-01-12 11:59:52 +08:00
Pxl
068367063f [Improvement](function) optimization for substr with ascii string (#29799) 2024-01-12 11:59:52 +08:00
Pxl
3cf95d0fdf [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec (#28788)
optimize for ColumnNullable's serialize_vec/deserialize_vec
2024-01-12 11:59:52 +08:00
e93a16ac6e [fix](Nereids) support complex literal cast in fe (#29599) 2024-01-12 11:59:52 +08:00
771c66c034 [docs](typo)Update flink-doris-connector.md (#29834) 2024-01-12 11:59:52 +08:00
542a801041 [typo](doc) optimization show processlist (#28342) 2024-01-12 11:59:52 +08:00
Pxl
33b8311d5f [Improvement](runtime-filter) build runtime filter before build hash table on join build probe (#29727)
build runtime filter before build hash table on join build probe
2024-01-12 11:59:52 +08:00
58f8994f5d [Fix](core) Fix initializing the WalManager could prevent the BE from starting (#29688) 2024-01-12 11:59:27 +08:00
ba1ecf4390 [pipelineX](profile) refine scan profile (#29801) 2024-01-12 11:59:27 +08:00
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
17a2b89945 [runtimeFilter](nereids) expand runtime filter by join condition by default (#29633)
1. expand rf by join condition 
2. fix ignore_shape_nodes bug
2024-01-12 11:59:27 +08:00
a94343c5f9 [fix](planner) Fix table sample not take effect if exist conjunct #29814 2024-01-12 11:59:27 +08:00
4a98db2d47 [fix](Nereids): add def in test (#29796)
missing `def` will cause variable is global, it's easily to cause concurrent problem
2024-01-12 11:59:27 +08:00