Commit Graph

6797 Commits

Author SHA1 Message Date
c599cf311d [fix](migrate) migrate check old tablet had deleted (#29909) 2024-01-16 18:33:51 +08:00
e3a1138da7 [fix](migrate disk) fix tablet disk migration timeout too large (#29895) 2024-01-16 18:33:51 +08:00
e417128fb9 [bug](bitmap) should return error status when execute failed (#29841) 2024-01-16 18:30:23 +08:00
e7b221ba66 [fix](be-ut) Fix unstable test cases (#29896)
The following cases are unstable.

1. LoadStreamMgrTest
2. TaskWorkerPoolTest.PriorTaskWorkerPool

Rationales

1. LoadStreamMgrTest
It is related to timeout. If we investigate the examples in BRPC, we will find the timeout is usually set to 0 rather than a specific number.
2. TaskWorkerPoolTest.PriorTaskWorkerPool
The order of the threads for the lock contentions is undetermined.
2024-01-16 18:30:23 +08:00
41875a0bf5 [fix](move-memtable) check segment id in add_segment (#29898) 2024-01-16 18:30:23 +08:00
de3fdc7d08 [chore](Fix) Fix uninitilized buffer in read_cluster_id() (#29949) 2024-01-14 15:56:19 +08:00
60f6436f26 [fix](schema cache) adjust the destruction order of _tablet_schema_cache and storage engine (#29923) 2024-01-13 23:36:15 +08:00
e4e57e9b05 [chore](removelogs) remove debug query timeout logs 2024-01-12 14:37:20 +08:00
99024ad7bd [fix](move-memtable) check eos for already closed streams (#29734) 2024-01-12 13:58:20 +08:00
3b25e69311 [bug](rf) fix invalid type for runtime filters when result column is const (#29851) 2024-01-12 13:58:20 +08:00
2a77858845 [fix](move-memtable) check all streams for failed reason (#29877) 2024-01-12 13:58:20 +08:00
a314491535 [Fix](inverted index) fix array inverted index builder error (#29869) 2024-01-12 13:58:19 +08:00
5ef8428345 [Refactor](executor)refactor workload group log fron WARNING to INFO #29878 2024-01-12 13:58:19 +08:00
1718341051 [pipelineX](fix) Fix correctness problem due to local hash shuffle (#29881) 2024-01-12 13:58:19 +08:00
ad2c13e009 [Optimize](kill-query)Support the scanners exits as soon as possible when kill query #29803 2024-01-12 13:58:19 +08:00
d494674ff4 [opt](parquet-reader) Opt parquet decimal type reading. (#29825) 2024-01-12 13:58:19 +08:00
ad986a78ae [Fix](executor)Fix Grayscale upgrade be code dump when report statistics #29843 2024-01-12 13:58:19 +08:00
d525f576e1 [improve] Use lru cache to count the number of column in tablet schema to control memory (#29668) 2024-01-12 13:58:19 +08:00
cbffdbb8bf [bug](group_commit) fix relay wal problem on materialized-view (#29848) 2024-01-12 13:58:19 +08:00
a4f29193f6 [pipelineX](fix) Fix incorrect runtime filter (#29860) 2024-01-12 13:58:19 +08:00
407a4a285d [improve](load) reduce logs from memtable memory limiter (#29840) 2024-01-12 13:58:19 +08:00
ebfbe0c8dd [opt](information_schema) support information_schema in external catalog (#28919)
Add `information_schema` database for all catalog.
This is useful when using BI tools to connect to Doris,
the tools can get meta info from `information_schema`.

This PR mainly changes:

1. There will be a `information_schema` db in each catalog.
2. Each `information_schema` db only store the meta info of the catalog it belongs to.
3. For `information_schema`, the `TABLE_SCHEMA` column's value is the database name.
4. There is a new global variable `show_full_dbname_in_info_schema_db`, default is false, if set to true,
    The `TABLE_SCHEMA` column's value is the like `ctl.db`, because:

	When connect to Doris, the `database` info in connection url will be: `xxx?db=ctl.db`.
	
	And then some BI will try to query `information_schema` with sql like:
	
	`select * from information_schema.columns where TABLE_SCHEMA = "ctl.db"`
	
	So it has to be format as `ctl.db`
	
	eg, the `information_schema.columns` table in external catalog `doris` is like:
	
	```
	mysql> select * from information_schema.columns limit 1\G
	*************************** 1. row ***************************
	           TABLE_CATALOG: doris
	            TABLE_SCHEMA: doris.__internal_schema
	              TABLE_NAME: column_statistics
	             COLUMN_NAME: id
	        ORDINAL_POSITION: 1
	          COLUMN_DEFAULT: NULL
	             IS_NULLABLE: NO
	               DATA_TYPE: varchar
	CHARACTER_MAXIMUM_LENGTH: 4096
	  CHARACTER_OCTET_LENGTH: 16384
	       NUMERIC_PRECISION: NULL
	           NUMERIC_SCALE: NULL
	      DATETIME_PRECISION: NULL
	      CHARACTER_SET_NAME: NULL
	          COLLATION_NAME: NULL
	             COLUMN_TYPE: varchar(4096)
	              COLUMN_KEY:
	                   EXTRA:
	              PRIVILEGES:
	          COLUMN_COMMENT:
	             COLUMN_SIZE: 4096
	          DECIMAL_DIGITS: NULL
	   GENERATION_EXPRESSION: NULL
	                  SRS_ID: NULL
	```
	
6. Modify the behavior of

	- show tables
	- shwo databases
	- show columns
	- show table status

	The above statements may query the `information_schema` db if there is `where` predicate after them
2024-01-12 13:58:19 +08:00
4d97f8ea75 [enhance](function) support two special format for str_to_date (#29823) 2024-01-12 12:00:32 +08:00
22c134fa0a [fix](rowset-reader) direct mode shouldn't use merge iterator (#29678) 2024-01-12 11:59:52 +08:00
f02fb5d49e [fix](vec) wrong implementation of operator <=> of Field (#29743) 2024-01-12 11:59:52 +08:00
c9a949130b [Case](wal) Add wal group commit sink case with low disk space fault injection (#29731) 2024-01-12 11:59:52 +08:00
8c0b046ad4 [case](wal)Add wal backpressure case (#29725) 2024-01-12 11:59:52 +08:00
Pxl
068367063f [Improvement](function) optimization for substr with ascii string (#29799) 2024-01-12 11:59:52 +08:00
Pxl
3cf95d0fdf [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec (#28788)
optimize for ColumnNullable's serialize_vec/deserialize_vec
2024-01-12 11:59:52 +08:00
Pxl
33b8311d5f [Improvement](runtime-filter) build runtime filter before build hash table on join build probe (#29727)
build runtime filter before build hash table on join build probe
2024-01-12 11:59:52 +08:00
58f8994f5d [Fix](core) Fix initializing the WalManager could prevent the BE from starting (#29688) 2024-01-12 11:59:27 +08:00
ba1ecf4390 [pipelineX](profile) refine scan profile (#29801) 2024-01-12 11:59:27 +08:00
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
c7968480ac [fix](move-memtable) fix use-after-free in LoadStreamReplyHandler (#29791) 2024-01-12 11:57:16 +08:00
81680383e6 [UT](wal) Add wal dirs info be ut (#29759) 2024-01-12 11:57:16 +08:00
0d16ec7345 [improvement](cooldown) do not cooldown tablet without cold data (#29690) 2024-01-12 11:57:16 +08:00
7c7dbf15bc [feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance (#29736) 2024-01-12 11:57:16 +08:00
2e90c457b6 [fix](move-memtable) don't fail if success tablets to one BE is less than quorum (#29754) 2024-01-12 11:57:16 +08:00
8fd1ce4a2e [Enhancement](inverted index) refact some inverted index writer code to make more effieciency (#29602) 2024-01-12 11:53:58 +08:00
f8d3b20911 [improve](fmt) use format_to and FMT_COMPILE to speed up write data #29682 2024-01-12 11:53:57 +08:00
bd4ef638f1 [fix](invert index) fix match_phrase_prefix memory leak (#29774) 2024-01-12 11:53:12 +08:00
a94f2564e3 [fix] (schema change) ignore the error of repeatedly writing rowset during schema change (#29733) 2024-01-12 11:48:39 +08:00
ca75c9b8ab add more logs to debug timeout
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-12 11:48:39 +08:00
fc4ca712ed [bugfix](core) using weak ptr in data stream receiver to avoid runtime state is deconstructed (#29410) 2024-01-12 11:48:39 +08:00
7287c0ca15 [Opt](exec)(multi-catalog) Opt date type reading. (#29571) 2024-01-12 11:48:39 +08:00
463a7ab212 [Performance](exec) opt the exchange performance (#29579) 2024-01-12 11:46:29 +08:00
48f58510a8 [refactor](tabletwriter) make tablet writer's rpc callback safe, could exit any time (#29684)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-12 11:46:29 +08:00
f8aba3faae [Enhancement](index tool) refine inverted index tool code (#29717) 2024-01-12 11:46:29 +08:00
aec373e2a0 [fix](regression) remove a buggy fault-injection case which make BE crash (#29673)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2024-01-12 11:46:29 +08:00
abb7640d37 [debug](timeout) add more log in scanner ctx to find timeout problem #29704
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-12 11:44:21 +08:00