Commit Graph

16328 Commits

Author SHA1 Message Date
cbffdbb8bf [bug](group_commit) fix relay wal problem on materialized-view (#29848) 2024-01-12 13:58:19 +08:00
a4f29193f6 [pipelineX](fix) Fix incorrect runtime filter (#29860) 2024-01-12 13:58:19 +08:00
407a4a285d [improve](load) reduce logs from memtable memory limiter (#29840) 2024-01-12 13:58:19 +08:00
f94ff21eef [ci](perf) adjust threshold (#29856)
Co-authored-by: stephen <hello-stephen@qq.com>
2024-01-12 13:58:19 +08:00
ebfbe0c8dd [opt](information_schema) support information_schema in external catalog (#28919)
Add `information_schema` database for all catalog.
This is useful when using BI tools to connect to Doris,
the tools can get meta info from `information_schema`.

This PR mainly changes:

1. There will be a `information_schema` db in each catalog.
2. Each `information_schema` db only store the meta info of the catalog it belongs to.
3. For `information_schema`, the `TABLE_SCHEMA` column's value is the database name.
4. There is a new global variable `show_full_dbname_in_info_schema_db`, default is false, if set to true,
    The `TABLE_SCHEMA` column's value is the like `ctl.db`, because:

	When connect to Doris, the `database` info in connection url will be: `xxx?db=ctl.db`.
	
	And then some BI will try to query `information_schema` with sql like:
	
	`select * from information_schema.columns where TABLE_SCHEMA = "ctl.db"`
	
	So it has to be format as `ctl.db`
	
	eg, the `information_schema.columns` table in external catalog `doris` is like:
	
	```
	mysql> select * from information_schema.columns limit 1\G
	*************************** 1. row ***************************
	           TABLE_CATALOG: doris
	            TABLE_SCHEMA: doris.__internal_schema
	              TABLE_NAME: column_statistics
	             COLUMN_NAME: id
	        ORDINAL_POSITION: 1
	          COLUMN_DEFAULT: NULL
	             IS_NULLABLE: NO
	               DATA_TYPE: varchar
	CHARACTER_MAXIMUM_LENGTH: 4096
	  CHARACTER_OCTET_LENGTH: 16384
	       NUMERIC_PRECISION: NULL
	           NUMERIC_SCALE: NULL
	      DATETIME_PRECISION: NULL
	      CHARACTER_SET_NAME: NULL
	          COLLATION_NAME: NULL
	             COLUMN_TYPE: varchar(4096)
	              COLUMN_KEY:
	                   EXTRA:
	              PRIVILEGES:
	          COLUMN_COMMENT:
	             COLUMN_SIZE: 4096
	          DECIMAL_DIGITS: NULL
	   GENERATION_EXPRESSION: NULL
	                  SRS_ID: NULL
	```
	
6. Modify the behavior of

	- show tables
	- shwo databases
	- show columns
	- show table status

	The above statements may query the `information_schema` db if there is `where` predicate after them
2024-01-12 13:58:19 +08:00
d3721455b0 [Session](rf) Change the default min size of bf runtime filter (#29837) 2024-01-12 13:58:19 +08:00
f67a00ffbb [opt](nereids) prune runtime redundant filters (#29828)
1. expand_runtime_filter_by_inner_join will create some redundant rfs,e.g., tpch q5 and q9, we need to remove one
2. hive: prune rf if target only used as probe
2024-01-12 13:58:19 +08:00
ed3c8bba87 [fix](auth)remove the key when priv is empty (#29522)
- remove the key when priv is empty
- check priv when create mv
2024-01-12 13:58:19 +08:00
8ba1eb0b02 [feature](mtmv) task tvf add queryId (#29671)
To better locate abnormal situations, add queryId
2024-01-12 12:00:32 +08:00
4d97f8ea75 [enhance](function) support two special format for str_to_date (#29823) 2024-01-12 12:00:32 +08:00
eed72a101e [fix](Nereids) decimalv3 cast in fe produce wrong data (#29808)
case:
```
MySQL root@127.0.0.1:test> select cast(12 as decimalv3(2,1))
+-----------------------------+
| cast(12 as DECIMALV3(2, 1)) |
+-----------------------------+
| 12.0                        |
+-----------------------------+
```

decimalv2 literal will generate wrong result too. But it is not only
bugs in planner, but also have bugs in executor. So we need fix executor
bug in another PR.
2024-01-12 12:00:13 +08:00
22c134fa0a [fix](rowset-reader) direct mode shouldn't use merge iterator (#29678) 2024-01-12 11:59:52 +08:00
885d8b28ba [fix](Nerids): fix function deps when check unique and not null #29797 2024-01-12 11:59:52 +08:00
f02fb5d49e [fix](vec) wrong implementation of operator <=> of Field (#29743) 2024-01-12 11:59:52 +08:00
18f850c94f [enhance](auto-partition) forbid null column for auto partition (#29749) 2024-01-12 11:59:52 +08:00
c9a949130b [Case](wal) Add wal group commit sink case with low disk space fault injection (#29731) 2024-01-12 11:59:52 +08:00
8c0b046ad4 [case](wal)Add wal backpressure case (#29725) 2024-01-12 11:59:52 +08:00
Pxl
068367063f [Improvement](function) optimization for substr with ascii string (#29799) 2024-01-12 11:59:52 +08:00
Pxl
3cf95d0fdf [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec (#28788)
optimize for ColumnNullable's serialize_vec/deserialize_vec
2024-01-12 11:59:52 +08:00
e93a16ac6e [fix](Nereids) support complex literal cast in fe (#29599) 2024-01-12 11:59:52 +08:00
771c66c034 [docs](typo)Update flink-doris-connector.md (#29834) 2024-01-12 11:59:52 +08:00
542a801041 [typo](doc) optimization show processlist (#28342) 2024-01-12 11:59:52 +08:00
Pxl
33b8311d5f [Improvement](runtime-filter) build runtime filter before build hash table on join build probe (#29727)
build runtime filter before build hash table on join build probe
2024-01-12 11:59:52 +08:00
58f8994f5d [Fix](core) Fix initializing the WalManager could prevent the BE from starting (#29688) 2024-01-12 11:59:27 +08:00
ba1ecf4390 [pipelineX](profile) refine scan profile (#29801) 2024-01-12 11:59:27 +08:00
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
17a2b89945 [runtimeFilter](nereids) expand runtime filter by join condition by default (#29633)
1. expand rf by join condition 
2. fix ignore_shape_nodes bug
2024-01-12 11:59:27 +08:00
a94343c5f9 [fix](planner) Fix table sample not take effect if exist conjunct #29814 2024-01-12 11:59:27 +08:00
4a98db2d47 [fix](Nereids): add def in test (#29796)
missing `def` will cause variable is global, it's easily to cause concurrent problem
2024-01-12 11:59:27 +08:00
f0eb23a496 [fix](insert-into) fix insert into lose data (#29802) 2024-01-12 11:58:55 +08:00
c7968480ac [fix](move-memtable) fix use-after-free in LoadStreamReplyHandler (#29791) 2024-01-12 11:57:16 +08:00
81680383e6 [UT](wal) Add wal dirs info be ut (#29759) 2024-01-12 11:57:16 +08:00
0d16ec7345 [improvement](cooldown) do not cooldown tablet without cold data (#29690) 2024-01-12 11:57:16 +08:00
7c7dbf15bc [feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance (#29736) 2024-01-12 11:57:16 +08:00
81d6775b7b [Cleanup](Nereids): delete useless ddlSql to avoid wrong usage (#29788)
ddlSql is useless and some code use getDdlSql() wrong, so delete those code
2024-01-12 11:57:16 +08:00
2e90c457b6 [fix](move-memtable) don't fail if success tablets to one BE is less than quorum (#29754) 2024-01-12 11:57:16 +08:00
8fd1ce4a2e [Enhancement](inverted index) refact some inverted index writer code to make more effieciency (#29602) 2024-01-12 11:53:58 +08:00
99c8e47518 [fix](nereids) fix regression case "nereids_p0/runtimefilter" (#29776) 2024-01-12 11:53:58 +08:00
a244f11da2 [fix](statistics)Fix alter column stats not forward to master bug (#29786)
Alter column stats operation need to write bdbje, so it should be forwarded to master to execute. Otherwise, the operation on follower/observer will cause the FE crash.
2024-01-12 11:53:57 +08:00
f8d3b20911 [improve](fmt) use format_to and FMT_COMPILE to speed up write data #29682 2024-01-12 11:53:57 +08:00
697a6a4ba2 [Refactor](admin-stmt) rename some admin-show statestmt (#29492)
The `ADMIN SHOW` statement can not be executed with high version of mysql 8.x jdbc driver.
So I rename these statement, remove the `ADMIN` keywords.

1. ADMIN SHOW CONFIG -> SHOW CONFIG
2. ADMIN SHOW REPLICA -> SHOW REPLICA
3. ADMIN DIAGNOSE TABLET -> SHOW TABLET DIAGNOSIS
4. ADMIN SHOW TABLET -> SHOW TABLET

for compatibility, the old statements are still supported, but not recommend to use.
They will be removed in later version
2024-01-12 11:53:57 +08:00
5d0732fcf4 [typo](docker thirdparties) Modify script prompt information error (#29770) 2024-01-12 11:53:57 +08:00
ddf2e8d3dd [feature](Nereids): merge topNs (#28246)
merge topNs like 
```
TopN
|
TopN

merge ->

TopN
```
2024-01-12 11:53:42 +08:00
3e9cd3a8b9 [minor](jdbc) fix wrong log and add more info (#29557) 2024-01-12 11:53:21 +08:00
bd4ef638f1 [fix](invert index) fix match_phrase_prefix memory leak (#29774) 2024-01-12 11:53:12 +08:00
7db65f3f6c [tools] update tpcds tools query (#29745)
Co-authored-by: zhongjian.xzj <zhongjian.xzj@192.168.2.27>
2024-01-12 11:52:56 +08:00
542cfb37b0 [ci](perf) adjust performance pipeline (#29622)
remove old tpch trigger
branch-2.0 also run performance
2024-01-12 11:52:47 +08:00
38c225f76c [fix](nereids) generate correct order for runtime filter when contains NullSafeEquals hash condition (#29726)
Be do not support RF for NullSafeEquals, so fe not generate RF for them.
However, after we support NullSafeEquals as Hash join condition,
the order of RF is wrong when generating RF in FE. this PR fix it.
2024-01-12 11:49:09 +08:00
e17809a684 [fix](nereids)logicalhaving is in wrong place after logicalagg and logicalwindow (#29463) 2024-01-12 11:48:39 +08:00
883d6dfc73 [fix](planner)strip trailing zeros for decimal literal if the precision larger than max decimal precision in doris (#29737) 2024-01-12 11:48:39 +08:00