database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
16,324 Commits 1 Branch 0 Tags
ebfbe0c8dd2879a462c8a604242fccc11d897e4c
Commit Graph

6776 Commits

Author SHA1 Message Date
Mingyu Chen
ebfbe0c8dd [opt](information_schema) support information_schema in external catalog (#28919) 
Add `information_schema` database for all catalog.
This is useful when using BI tools to connect to Doris,
the tools can get meta info from `information_schema`.

This PR mainly changes:

1. There will be a `information_schema` db in each catalog.
2. Each `information_schema` db only store the meta info of the catalog it belongs to.
3. For `information_schema`, the `TABLE_SCHEMA` column's value is the database name.
4. There is a new global variable `show_full_dbname_in_info_schema_db`, default is false, if set to true,
    The `TABLE_SCHEMA` column's value is the like `ctl.db`, because:

	When connect to Doris, the `database` info in connection url will be: `xxx?db=ctl.db`.
	
	And then some BI will try to query `information_schema` with sql like:
	
	`select * from information_schema.columns where TABLE_SCHEMA = "ctl.db"`
	
	So it has to be format as `ctl.db`
	
	eg, the `information_schema.columns` table in external catalog `doris` is like:
	
	```
	mysql> select * from information_schema.columns limit 1\G
	*************************** 1. row ***************************
	           TABLE_CATALOG: doris
	            TABLE_SCHEMA: doris.__internal_schema
	              TABLE_NAME: column_statistics
	             COLUMN_NAME: id
	        ORDINAL_POSITION: 1
	          COLUMN_DEFAULT: NULL
	             IS_NULLABLE: NO
	               DATA_TYPE: varchar
	CHARACTER_MAXIMUM_LENGTH: 4096
	  CHARACTER_OCTET_LENGTH: 16384
	       NUMERIC_PRECISION: NULL
	           NUMERIC_SCALE: NULL
	      DATETIME_PRECISION: NULL
	      CHARACTER_SET_NAME: NULL
	          COLLATION_NAME: NULL
	             COLUMN_TYPE: varchar(4096)
	              COLUMN_KEY:
	                   EXTRA:
	              PRIVILEGES:
	          COLUMN_COMMENT:
	             COLUMN_SIZE: 4096
	          DECIMAL_DIGITS: NULL
	   GENERATION_EXPRESSION: NULL
	                  SRS_ID: NULL
	```
	
6. Modify the behavior of

	- show tables
	- shwo databases
	- show columns
	- show table status

	The above statements may query the `information_schema` db if there is `where` predicate after them
 2024-01-12 13:58:19 +08:00
zclllyybb
4d97f8ea75 [enhance](function) support two special format for str_to_date (#29823) 2024-01-12 12:00:32 +08:00
Xin Liao
22c134fa0a [fix](rowset-reader) direct mode shouldn't use merge iterator (#29678) 2024-01-12 11:59:52 +08:00
Jerry Hu
f02fb5d49e [fix](vec) wrong implementation of operator <=> of Field (#29743) 2024-01-12 11:59:52 +08:00
abmdocrt
c9a949130b [Case](wal) Add wal group commit sink case with low disk space fault injection (#29731) 2024-01-12 11:59:52 +08:00
abmdocrt
8c0b046ad4 [case](wal)Add wal backpressure case (#29725) 2024-01-12 11:59:52 +08:00
Pxl
068367063f [Improvement](function) optimization for substr with ascii string (#29799) 2024-01-12 11:59:52 +08:00
Pxl
3cf95d0fdf [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec (#28788) 
optimize for ColumnNullable's serialize_vec/deserialize_vec
 2024-01-12 11:59:52 +08:00
Pxl
33b8311d5f [Improvement](runtime-filter) build runtime filter before build hash table on join build probe (#29727) 
build runtime filter before build hash table on join build probe
 2024-01-12 11:59:52 +08:00
abmdocrt
58f8994f5d [Fix](core) Fix initializing the WalManager could prevent the BE from starting (#29688) 2024-01-12 11:59:27 +08:00
Gabriel
ba1ecf4390 [pipelineX](profile) refine scan profile (#29801) 2024-01-12 11:59:27 +08:00
wangbo
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
Kaijie Chen
c7968480ac [fix](move-memtable) fix use-after-free in LoadStreamReplyHandler (#29791) 2024-01-12 11:57:16 +08:00
abmdocrt
81680383e6 [UT](wal) Add wal dirs info be ut (#29759) 2024-01-12 11:57:16 +08:00
Yongqiang YANG
0d16ec7345 [improvement](cooldown) do not cooldown tablet without cold data (#29690) 2024-01-12 11:57:16 +08:00
plat1ko
7c7dbf15bc [feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance (#29736) 2024-01-12 11:57:16 +08:00
Kaijie Chen
2e90c457b6 [fix](move-memtable) don't fail if success tablets to one BE is less than quorum (#29754) 2024-01-12 11:57:16 +08:00
airborne12
8fd1ce4a2e [Enhancement](inverted index) refact some inverted index writer code to make more effieciency (#29602) 2024-01-12 11:53:58 +08:00
zhangstar333
f8d3b20911 [improve](fmt) use format_to and FMT_COMPILE to speed up write data #29682 2024-01-12 11:53:57 +08:00
zzzxl
bd4ef638f1 [fix](invert index) fix match_phrase_prefix memory leak (#29774) 2024-01-12 11:53:12 +08:00
Luwei
a94f2564e3 [fix] (schema change) ignore the error of repeatedly writing rowset during schema change (#29733) 2024-01-12 11:48:39 +08:00
yiguolei
ca75c9b8ab add more logs to debug timeout 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:48:39 +08:00
yiguolei
fc4ca712ed [bugfix](core) using weak ptr in data stream receiver to avoid runtime state is deconstructed (#29410) 2024-01-12 11:48:39 +08:00
Qi Chen
7287c0ca15 [Opt](exec)(multi-catalog) Opt date type reading. (#29571) 2024-01-12 11:48:39 +08:00
HappenLee
463a7ab212 [Performance](exec) opt the exchange performance (#29579) 2024-01-12 11:46:29 +08:00
yiguolei
48f58510a8 [refactor](tabletwriter) make tablet writer's rpc callback safe, could exit any time (#29684) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:46:29 +08:00
airborne12
f8aba3faae [Enhancement](index tool) refine inverted index tool code (#29717) 2024-01-12 11:46:29 +08:00
zhengyu
aec373e2a0 [fix](regression) remove a buggy fault-injection case which make BE crash (#29673) 
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2024-01-12 11:46:29 +08:00
yiguolei
abb7640d37 [debug](timeout) add more log in scanner ctx to find timeout problem #29704 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:44:21 +08:00
yangshijie
be56bf06cf [feature](function) support ip function named is_ip_address_in_range(addr, cidr) (#29681) 2024-01-12 11:44:21 +08:00
xy
374c638788 [improvement](executor) improve readability and efficiency of the checks (#29696) 
Co-authored-by: xingying01 <xingying01@corp.netease.com>
 2024-01-12 11:44:21 +08:00
abmdocrt
2959219b4d [Enhancement](wal) Add fault injection case for wal back pressure (#29689) 2024-01-12 11:44:21 +08:00
airborne12
87a6def862 [Fix](index tool) fix inverted index tool memory leak (#29701) 2024-01-12 11:44:21 +08:00
airborne12
87023d3b7a [Fix](inverted index) fix memory leak in inverted index when encountering fault (#29676) 2024-01-12 11:44:21 +08:00
Jerry Hu
9f73f77d12 [fix](pipeline) incorrect result caused by missing output block of union operator (#29677) 2024-01-12 11:44:21 +08:00
Mryange
aa4de6f39a (feature)[pipelineX]Make operator_id negative in pipelineX (#29649) 
"operator_id" should be invisible, but the local shuffle is a planned operator in the BE (Backend), without a plan node ID. We use it in profiles and other places, and there might be duplicates. Therefore, we switch it to a negative number here to distinguish it as a plan node ID.
 2024-01-12 11:44:21 +08:00
plat1ko
28dca66c06 Fix BE UT macOS compile (#29663) 2024-01-12 11:44:20 +08:00
zy-kkk
8fc9c18c85 [improvement](jdbc catalog) Put the jdbc connection pool parameters into catalog properties (#29195) 2024-01-12 11:40:28 +08:00
zzzxl
c9e2f1934a [opt](file cache) make the cache queue percentage configurable (#29537) 2024-01-12 11:36:58 +08:00
zzzxl
9791a63212 [opt](match_phrase) Optimizing match_phrase with a new algorithm (#29444) 
Based on the latest Lucene algorithm
 2024-01-12 11:33:44 +08:00
qiye
53f1521308 [feature](inverted index)Support failover when index compaction failed (#29553) 2024-01-12 11:33:15 +08:00
yiguolei
9ef4e49307 [bugfix](scannerdeadloop) there is a dead loop in scanner ctx (#29794) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-11 16:47:54 +08:00
Gabriel
01a91a482e [pipelineX](fix) fix StreamingAggSource crash due to empty data block (#29769) 2024-01-10 17:13:24 +08:00
plat1ko
fae7a395de [Bug](FS) Fix S3FileSystem::file_size_impl #29720 2024-01-10 10:34:10 +08:00
Gabriel
e2ccca6290 [pipelineX](bug) Fix runtime filter timeout (#29738) 2024-01-10 10:32:57 +08:00
Gabriel
767de7afe8 Revert "[feature](pipelineX) control exchange sink by memory usage (#28814)" (#29652) 
This reverts commit e326ebb63e4e07d8ee6595561ab19dc5d411f592.
 2024-01-08 21:48:51 +08:00
Gabriel
59d7f64360 [Fix](Nereids) fix pipelineX distribute expr list with child output expr ids (#29621) 2024-01-08 10:46:27 +08:00
Pxl
e556536de1 [Refactor](join) split SetHashTableVariants out from HashTableVariants (#29519) 
split SetHashTableVariants out from HashTableVariants
 2024-01-08 10:37:00 +08:00
yiguolei
c58d18147b [refactor](resultwriter) rename append_block to write method is more meaningful (#29635) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-08 00:00:33 +08:00
yiguolei
c497f749ce [debug](timeout) debug select timeout (#29627) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-07 19:54:02 +08:00