Commit Graph

18263 Commits

Author SHA1 Message Date
bd74890cf7 [fix](multi-catalog) JDBC Catalog Unknown UNSIGNED type of mysql, type: [DOUBLE] (#19912) 2023-05-23 09:29:57 +08:00
fe111207a9 [Fix](lazy_open) Fix lazy open null point (#19829) 2023-05-23 09:17:46 +08:00
30417e06d4 [enhance](fs) use bvar to monitor s3 file reader& writer (#19607)
remove useless prefix
2023-05-22 23:01:21 +08:00
3dcdadcea6 [Improvement](function) support decimalv3 for function least and greatest (#19931) 2023-05-22 22:48:44 +08:00
53ba46e404 [Fix][Refactor] Fix 'not member call on null pointer of type 'doris::TextConverter' error in ubsan env and refactor text converter. (#19849)
Fix 'not member call on null pointer of type doris::TextConverter' error in ubsan env and refactor text converter.
2023-05-22 21:00:19 +08:00
6762af3c9b [Improve](struct)improve struct support into outfile (#19894)
support select into outfile for struct type
2023-05-22 18:45:56 +08:00
Pxl
9945067e3c [Bug](function) make VcompoundPred optimization work well (#19870)
make VcompoundPred optimization work well
#19818 this pr try to enable VcompoundPred optimization but get wrong result on tpcds q28.
The reason is some nullable logic on mysql need special handling.

mysql [regression_test_tpcds_sf1_p1]>select null and false;
+----------------+
| NULL AND FALSE |
+----------------+
|              0 |
+----------------+
1 row in set (0.00 sec)

mysql [regression_test_tpcds_sf1_p1]>select null and true;
+---------------+
| NULL AND TRUE |
+---------------+
| NULL          |
+---------------+
1 row in set (0.00 sec)

mysql [regression_test_tpcds_sf1_p1]>select null or false;
+---------------+
| NULL OR FALSE |
+---------------+
| NULL          |
+---------------+
1 row in set (0.00 sec)

mysql [regression_test_tpcds_sf1_p1]>select null or true;
+--------------+
| NULL OR TRUE |
+--------------+
|            1 |
+--------------+
1 row in set (0.00 sec)
2023-05-22 18:32:17 +08:00
7ae3724acd [typo](doc)fix typo in readme #19895
miliseconds -> milliseconds
2023-05-22 18:31:19 +08:00
750a3ea1b4 [doc](fqdn)broker fqdn doc #19910 2023-05-22 18:30:57 +08:00
Pxl
e9223f6a19 [Feature](aggregation) add agg_state define and ddl support (#19824)
add agg_state define and ddl support
2023-05-22 11:45:53 +08:00
Pxl
d64be9565d [Bug](function) fix function in get wrong result when input const column (#19791)
fix function in get wrong result when input const column
2023-05-22 10:58:29 +08:00
f57b6adba9 [improvement](libhdfs) Use keytab and principal to login kerberos (#19841)
User keytab and princpal to login kerberos.
And user does not need to execute kinit manually anymore.
2023-05-22 10:54:21 +08:00
76dc5841dc [opt](nereids)compute runtime filter size by the ndv of build side (#18803) 2023-05-22 10:38:11 +08:00
060fbe2b1f [typo](docs) replace the wrong field "json_paths" in s3 tvf related docs with the correct one "jsonpaths" #19900 2023-05-22 09:03:56 +08:00
a6bd014b8a [FIX](serde)pb ut is not stable #19907 2023-05-22 09:03:02 +08:00
7c539575c7 [refactor](hudi) remove hudi external table (#19908)
Hudi external table is deprecated since 1.2.
We should remove it now.
Recommend to use "multi-catalog" feature to connect to Hudi.

User can not create Hudi external table.
When restarting FE, all hudi external table will still be replayed but can not be read. And when doing checkpoint, all these tables will be discarded.
2023-05-22 09:02:34 +08:00
1c950d6930 [fix](config) fix memory config enable_query_memroy_overcommit spell problem #19898 2023-05-22 00:32:20 +08:00
76c358b3e3 [revert](memory) revert page no use Allocator && default disable ChunkAllocator (#19905)
default chunk allocator reserve is 0. At this time, it is meaningless to enable chunk allocator, it will only waste memory.
2023-05-21 22:16:41 +08:00
33fd965b5c [feature-wip](resouce-group) Supports memory soft isolation of resource group (#19802)
create resource groups name properties(
    'enable_memory_overcommit' = 'true' // whether to enable memory soft isolation
)
2023-05-21 19:33:57 +08:00
a7f3bfec89 [refactor](cluster)(step-2) remove cluster related to Backend (#19842) 2023-05-21 09:00:35 +08:00
777bdce5a5 [minor](clone) add more debug log for tablet scheduler (#19892)
Sometimes I find that the tablet scheduler can not schedule tablet, and with no more info for debugging.
So I add some debug log for this process.
No logic is changed.
2023-05-20 15:59:26 +08:00
8b9813663d [test](executor)add crud regression test for resource group (#19659)
dd crud regression test for resource group (#19659)
2023-05-20 13:49:02 +08:00
499f443779 [feature](iceberg) Support read iceberg data on gcs (#19815) 2023-05-20 12:40:03 +08:00
HB
1b119704f8 [Enhancement] show total transactions in show proc "/transactions" (#19492)
In a scenario where multiple DBs are simultaneously imported with high concurrency, a significant number of transactions will be generated. Without a summary field, we cannot clearly see how many transactions there are in the current cluster. Therefore, I have enhanced this point.

```
mysql> show proc "/transactions";
+-------+-----------------------------------+-----------------------+
| DbId     | DbName                                             | RunningTransactionNum |
+-------+-----------------------------------+-----------------------+
| 10002   | default_cluster:xxxx                         | 0                                     |
| 14005   | default_cluster:__internal_schema | 0                                     |
| Total     |  2                                                         | 0                                     |
+-------+-----------------------------------+-----------------------+
3 rows in set (0.02 sec)
```
2023-05-20 11:26:28 +08:00
a81db3e984 [improvement](FQDN) broker support fqdn (#19821)
1.broker support fqdn
2.change 'master_only' attr of 'enable_fqdn_mode'
2023-05-20 11:25:58 +08:00
178d6cc529 [improvement](multi-catalog)hms sync event log more info #19887 2023-05-20 08:25:14 +08:00
512806f902 [fix](ubsan) UBSAN avoid thread local switch 2023-05-20 07:14:43 +08:00
3e02b0e7ea [Doc](fix) Fixed two documentation errors (#19879) 2023-05-20 00:25:01 +08:00
ca737c37ee add testcases for inverted index on different datatypes (#19843) 2023-05-20 00:21:34 +08:00
6f21cd8a5f update create table doc (#19871) 2023-05-20 00:18:38 +08:00
e13e96473f Modify the jdk version requirements in the documentation (#19874) 2023-05-20 00:18:24 +08:00
0b367888f8 Modify the usage of Doriswriter parameters (#19875) 2023-05-20 00:18:10 +08:00
0d956da927 [doc](multi-catalog)change hms event doc (#19886)
change hms event doc
need restart more service
2023-05-19 22:48:41 +08:00
77dfdfdd50 [Bug][pipeline] Fix regression tpcds failed in nereid planner (#19885) 2023-05-19 22:30:48 +08:00
24b2fab943 [fix](Nereids): BuildAggForUnion forgot to convert Qualifier Type. (#19883) 2023-05-19 22:18:38 +08:00
88d09ec263 [fix](community): remove some collaborators. (#19884) 2023-05-19 22:08:01 +08:00
47509e65c9 [fix](memory)page no use Allocator, avoid ckbench oom (#19877) 2023-05-19 21:23:31 +08:00
5547bbbaef [decimalv3](function) support function width_bucket (#19806) 2023-05-19 20:28:59 +08:00
78bcc68ab8 [Fix](Nereids) fix serialize colocate table index concurrent bug (#19862)
When doing serialization of minidump input, we can find that when serializing colocate table index, the size and entry get by the hash map always unmatched when concurrent occur. So a write lock be added to ensure concurrency.
2023-05-19 19:51:22 +08:00
ae1577e95c [improvement](jdbc catalog) set oceanbase mysql mode jdbc param useCursorFetch default true (#19856) 2023-05-19 19:45:22 +08:00
68be81363b [enhance](Nereids): Pushdown Filter Through Project in Post Processor. (#19873)
Originally, PushdownFilterThroughProject is in CBO phase, but it will increase Memo size.
So, we move it into PostProcessor
2023-05-19 19:27:52 +08:00
abde8bf26a [chore](build) Fix the compilation errors on macOS (arm64) (#19859)
Some errors raise when building the codebase on macOS (arm64).
2023-05-19 18:50:47 +08:00
67dc68630b [Improve](complex-type)improve array/map/struct creating and function with decimalv3 (#19830) 2023-05-19 17:43:36 +08:00
d8d6880ef1 [Doc](point query) Modify the document to add constraints and limitations for optimizing high-concurrency point queries. (#19852) 2023-05-19 17:29:10 +08:00
2ab844550f [feature-wip](MTMV) support multi catalog (#19854)
* mtmv support multi catalog

* mtmv support multi catalog
2023-05-19 16:44:55 +08:00
0fc8d2e029 [Bug](decimal) fix variance_samp and avg_weighted #19861 2023-05-19 16:44:36 +08:00
24de75757d [fix](Nereids) drop table before create table in window function test (#19857) 2023-05-19 16:44:13 +08:00
514be8def1 [improvement](conf)Add an example of directly specifying an IP address #19860 2023-05-19 16:43:47 +08:00
65807f888b [fix](memory) Remind log if vm/overcommit_memory=2 when be start (#19795)
Expect vm overcommit memory value to be 1, system will no longer throw bad_alloc, memory alloc are always accepted,
memory limit check is handed over to Doris Allocator, make sure throw exception position is controllable,
otherwise bad_alloc can be thrown anywhere and it will be difficult to achieve exception safety.
2023-05-19 15:01:08 +08:00
9d54545bac [Fix](inverted index) add datev2/datetimev2 for inverted index column type (#19845)
When we try to query array of datetimev2 column by inverted index, it returns an error like this:

CREATE TABLE `nested` (
 `qid` bigint(20) NULL,
 `tag` array<text> NULL,
 `creationDate` datetime NULL,
 `title` text NULL,
 `user` text NULL,
 `answers.user` array<text> NULL,
 `answers.date` array<datetimev2(0)> NULL,
 INDEX tag_idx (`tag`) USING INVERTED PROPERTIES("parser" = "english") COMMENT '',
 INDEX creation_date_idx (`creationDate`) USING INVERTED COMMENT '',
 INDEX title_idx (`title`) USING INVERTED COMMENT '',
 INDEX user_idx (`user`) USING INVERTED COMMENT '',
 INDEX answers_user_idx (`answers.user`) USING INVERTED COMMENT '',
 INDEX answers_date_idx (`answers.date`) USING INVERTED COMMENT ''
) ENGINE=OLAP
DUPLICATE KEY(`qid`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`qid`) BUCKETS 18
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"storage_format" = "V2",
"compression" = "ZSTD",
"light_schema_change" = "true",
"dynamic_schema" = "true",
"disable_auto_compaction" = "false"
); 

mysql> select * from nested.nested where tag match 'java' and `answers.date` element_le '2012-04-08T21:15:33.873Z' limit 10;
ERROR 1105 (HY000): errCode = 2, detailMessage = no function found for MATCH_ELEMENT_LE,`answers.date` MA
2023-05-19 14:57:01 +08:00