Commit Graph

10647 Commits

Author SHA1 Message Date
1c950d6930 [fix](config) fix memory config enable_query_memroy_overcommit spell problem #19898 2023-05-22 00:32:20 +08:00
76c358b3e3 [revert](memory) revert page no use Allocator && default disable ChunkAllocator (#19905)
default chunk allocator reserve is 0. At this time, it is meaningless to enable chunk allocator, it will only waste memory.
2023-05-21 22:16:41 +08:00
33fd965b5c [feature-wip](resouce-group) Supports memory soft isolation of resource group (#19802)
create resource groups name properties(
    'enable_memory_overcommit' = 'true' // whether to enable memory soft isolation
)
2023-05-21 19:33:57 +08:00
a7f3bfec89 [refactor](cluster)(step-2) remove cluster related to Backend (#19842) 2023-05-21 09:00:35 +08:00
777bdce5a5 [minor](clone) add more debug log for tablet scheduler (#19892)
Sometimes I find that the tablet scheduler can not schedule tablet, and with no more info for debugging.
So I add some debug log for this process.
No logic is changed.
2023-05-20 15:59:26 +08:00
8b9813663d [test](executor)add crud regression test for resource group (#19659)
dd crud regression test for resource group (#19659)
2023-05-20 13:49:02 +08:00
499f443779 [feature](iceberg) Support read iceberg data on gcs (#19815) 2023-05-20 12:40:03 +08:00
HB
1b119704f8 [Enhancement] show total transactions in show proc "/transactions" (#19492)
In a scenario where multiple DBs are simultaneously imported with high concurrency, a significant number of transactions will be generated. Without a summary field, we cannot clearly see how many transactions there are in the current cluster. Therefore, I have enhanced this point.

```
mysql> show proc "/transactions";
+-------+-----------------------------------+-----------------------+
| DbId     | DbName                                             | RunningTransactionNum |
+-------+-----------------------------------+-----------------------+
| 10002   | default_cluster:xxxx                         | 0                                     |
| 14005   | default_cluster:__internal_schema | 0                                     |
| Total     |  2                                                         | 0                                     |
+-------+-----------------------------------+-----------------------+
3 rows in set (0.02 sec)
```
2023-05-20 11:26:28 +08:00
a81db3e984 [improvement](FQDN) broker support fqdn (#19821)
1.broker support fqdn
2.change 'master_only' attr of 'enable_fqdn_mode'
2023-05-20 11:25:58 +08:00
178d6cc529 [improvement](multi-catalog)hms sync event log more info #19887 2023-05-20 08:25:14 +08:00
512806f902 [fix](ubsan) UBSAN avoid thread local switch 2023-05-20 07:14:43 +08:00
3e02b0e7ea [Doc](fix) Fixed two documentation errors (#19879) 2023-05-20 00:25:01 +08:00
ca737c37ee add testcases for inverted index on different datatypes (#19843) 2023-05-20 00:21:34 +08:00
6f21cd8a5f update create table doc (#19871) 2023-05-20 00:18:38 +08:00
e13e96473f Modify the jdk version requirements in the documentation (#19874) 2023-05-20 00:18:24 +08:00
0b367888f8 Modify the usage of Doriswriter parameters (#19875) 2023-05-20 00:18:10 +08:00
0d956da927 [doc](multi-catalog)change hms event doc (#19886)
change hms event doc
need restart more service
2023-05-19 22:48:41 +08:00
77dfdfdd50 [Bug][pipeline] Fix regression tpcds failed in nereid planner (#19885) 2023-05-19 22:30:48 +08:00
24b2fab943 [fix](Nereids): BuildAggForUnion forgot to convert Qualifier Type. (#19883) 2023-05-19 22:18:38 +08:00
88d09ec263 [fix](community): remove some collaborators. (#19884) 2023-05-19 22:08:01 +08:00
47509e65c9 [fix](memory)page no use Allocator, avoid ckbench oom (#19877) 2023-05-19 21:23:31 +08:00
5547bbbaef [decimalv3](function) support function width_bucket (#19806) 2023-05-19 20:28:59 +08:00
78bcc68ab8 [Fix](Nereids) fix serialize colocate table index concurrent bug (#19862)
When doing serialization of minidump input, we can find that when serializing colocate table index, the size and entry get by the hash map always unmatched when concurrent occur. So a write lock be added to ensure concurrency.
2023-05-19 19:51:22 +08:00
ae1577e95c [improvement](jdbc catalog) set oceanbase mysql mode jdbc param useCursorFetch default true (#19856) 2023-05-19 19:45:22 +08:00
68be81363b [enhance](Nereids): Pushdown Filter Through Project in Post Processor. (#19873)
Originally, PushdownFilterThroughProject is in CBO phase, but it will increase Memo size.
So, we move it into PostProcessor
2023-05-19 19:27:52 +08:00
abde8bf26a [chore](build) Fix the compilation errors on macOS (arm64) (#19859)
Some errors raise when building the codebase on macOS (arm64).
2023-05-19 18:50:47 +08:00
67dc68630b [Improve](complex-type)improve array/map/struct creating and function with decimalv3 (#19830) 2023-05-19 17:43:36 +08:00
d8d6880ef1 [Doc](point query) Modify the document to add constraints and limitations for optimizing high-concurrency point queries. (#19852) 2023-05-19 17:29:10 +08:00
2ab844550f [feature-wip](MTMV) support multi catalog (#19854)
* mtmv support multi catalog

* mtmv support multi catalog
2023-05-19 16:44:55 +08:00
0fc8d2e029 [Bug](decimal) fix variance_samp and avg_weighted #19861 2023-05-19 16:44:36 +08:00
24de75757d [fix](Nereids) drop table before create table in window function test (#19857) 2023-05-19 16:44:13 +08:00
514be8def1 [improvement](conf)Add an example of directly specifying an IP address #19860 2023-05-19 16:43:47 +08:00
65807f888b [fix](memory) Remind log if vm/overcommit_memory=2 when be start (#19795)
Expect vm overcommit memory value to be 1, system will no longer throw bad_alloc, memory alloc are always accepted,
memory limit check is handed over to Doris Allocator, make sure throw exception position is controllable,
otherwise bad_alloc can be thrown anywhere and it will be difficult to achieve exception safety.
2023-05-19 15:01:08 +08:00
9d54545bac [Fix](inverted index) add datev2/datetimev2 for inverted index column type (#19845)
When we try to query array of datetimev2 column by inverted index, it returns an error like this:

CREATE TABLE `nested` (
 `qid` bigint(20) NULL,
 `tag` array<text> NULL,
 `creationDate` datetime NULL,
 `title` text NULL,
 `user` text NULL,
 `answers.user` array<text> NULL,
 `answers.date` array<datetimev2(0)> NULL,
 INDEX tag_idx (`tag`) USING INVERTED PROPERTIES("parser" = "english") COMMENT '',
 INDEX creation_date_idx (`creationDate`) USING INVERTED COMMENT '',
 INDEX title_idx (`title`) USING INVERTED COMMENT '',
 INDEX user_idx (`user`) USING INVERTED COMMENT '',
 INDEX answers_user_idx (`answers.user`) USING INVERTED COMMENT '',
 INDEX answers_date_idx (`answers.date`) USING INVERTED COMMENT ''
) ENGINE=OLAP
DUPLICATE KEY(`qid`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`qid`) BUCKETS 18
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"storage_format" = "V2",
"compression" = "ZSTD",
"light_schema_change" = "true",
"dynamic_schema" = "true",
"disable_auto_compaction" = "false"
); 

mysql> select * from nested.nested where tag match 'java' and `answers.date` element_le '2012-04-08T21:15:33.873Z' limit 10;
ERROR 1105 (HY000): errCode = 2, detailMessage = no function found for MATCH_ELEMENT_LE,`answers.date` MA
2023-05-19 14:57:01 +08:00
f46f0c84b2 [Enhancement](meta) Show remote data usage via SHOW DATA #19533 (#19752)
* [Enhancement](meta) Show remote data usage via SHOW DATA #19533

* [fix] correct some unit test results
2023-05-19 14:23:50 +08:00
a7376bf109 [typo](doc)flink and spark connector remove thrift compiled documents (#19794)
* [typo](doc)flink and spark connector remove thrift compiled documents

* delete enable_http_server_v2
2023-05-19 14:12:07 +08:00
7d1844d380 [FIX](Array)fix be master comapitible with fe1.2 #19850
when upgrade be first , fe is 1.2 , array contains_nulls is set not in thrift ,so would has core in be
Issue Number: close #xxx
2023-05-19 14:10:29 +08:00
c4900eb658 [Bug](DecimalV3) fix decimalv3 functions (#19801) 2023-05-19 14:10:01 +08:00
fcffb1d3de [minor](Nereids): add toString() for LogicalProperties (#19851) 2023-05-19 13:46:47 +08:00
92c6a3c53b [fix](Nereids) normalize repeat generate push down project with error nullable (#19831) 2023-05-19 13:15:42 +08:00
9c86cad4ec [improvement](session variable) add max execution time session variabe like mysql and add setter attributes in variables (#19759)
1. add session variable max_execution_time to an alias of query timeout, if user set max_execution_time, the query timeout will be modified too.
2. add a setter attribute to session variable, so that we could add some logic in setter method instead of field reflection.
2023-05-19 12:42:47 +08:00
cf7083d58b [explain](point query) modify explain for SHORT-CIRCUIT query (#19820) 2023-05-19 11:50:08 +08:00
a0c3ebeeab [Chore](build) fix typo in build.sh (#19846) 2023-05-19 11:49:20 +08:00
609b20bd02 [Feature](planner) use partial update in update from & delete from (#19262) 2023-05-19 09:46:29 +08:00
cbe12cfb69 [chore](build) Support specifying output path #19669
now we could

./build.sh --be --fe --output PATH_TO_BINARY_YOU_LIKE
did not modify the default value
2023-05-19 09:22:36 +08:00
84bad03ccb [feature](nereids) set proper min/max value for column stats when minExpr/maxExpr is not avialable #19673 2023-05-19 09:02:40 +08:00
0dd361dbf7 [fix](tracing) fix the issue that a trace may track multiple queries (#19804) 2023-05-19 08:58:53 +08:00
3e0b661267 [fix](test) Comment unstable stats test #19729 2023-05-19 08:55:28 +08:00
6f6d744a2a [fix](nereids) avoid 0 row count in stats derive #19640
row count of join estimation is at least 1 to make less error propagation.
2023-05-19 08:54:24 +08:00
14620a6766 [minor](log) add details for unqueryable replicas (#19792)
Add a new FE config: show_details_for_unaccessible_tablet.
Default is false, when set to true, if a query is unable to select a healthy replica,
the detailed information of all the replicas of the tablet including the specific reason why they are unqueryable,
will be printed out.
2023-05-19 08:53:57 +08:00