Commit Graph

3542 Commits

Author SHA1 Message Date
33fd965b5c [feature-wip](resouce-group) Supports memory soft isolation of resource group (#19802)
create resource groups name properties(
    'enable_memory_overcommit' = 'true' // whether to enable memory soft isolation
)
2023-05-21 19:33:57 +08:00
a7f3bfec89 [refactor](cluster)(step-2) remove cluster related to Backend (#19842) 2023-05-21 09:00:35 +08:00
777bdce5a5 [minor](clone) add more debug log for tablet scheduler (#19892)
Sometimes I find that the tablet scheduler can not schedule tablet, and with no more info for debugging.
So I add some debug log for this process.
No logic is changed.
2023-05-20 15:59:26 +08:00
499f443779 [feature](iceberg) Support read iceberg data on gcs (#19815) 2023-05-20 12:40:03 +08:00
HB
1b119704f8 [Enhancement] show total transactions in show proc "/transactions" (#19492)
In a scenario where multiple DBs are simultaneously imported with high concurrency, a significant number of transactions will be generated. Without a summary field, we cannot clearly see how many transactions there are in the current cluster. Therefore, I have enhanced this point.

```
mysql> show proc "/transactions";
+-------+-----------------------------------+-----------------------+
| DbId     | DbName                                             | RunningTransactionNum |
+-------+-----------------------------------+-----------------------+
| 10002   | default_cluster:xxxx                         | 0                                     |
| 14005   | default_cluster:__internal_schema | 0                                     |
| Total     |  2                                                         | 0                                     |
+-------+-----------------------------------+-----------------------+
3 rows in set (0.02 sec)
```
2023-05-20 11:26:28 +08:00
a81db3e984 [improvement](FQDN) broker support fqdn (#19821)
1.broker support fqdn
2.change 'master_only' attr of 'enable_fqdn_mode'
2023-05-20 11:25:58 +08:00
178d6cc529 [improvement](multi-catalog)hms sync event log more info #19887 2023-05-20 08:25:14 +08:00
77dfdfdd50 [Bug][pipeline] Fix regression tpcds failed in nereid planner (#19885) 2023-05-19 22:30:48 +08:00
24b2fab943 [fix](Nereids): BuildAggForUnion forgot to convert Qualifier Type. (#19883) 2023-05-19 22:18:38 +08:00
5547bbbaef [decimalv3](function) support function width_bucket (#19806) 2023-05-19 20:28:59 +08:00
78bcc68ab8 [Fix](Nereids) fix serialize colocate table index concurrent bug (#19862)
When doing serialization of minidump input, we can find that when serializing colocate table index, the size and entry get by the hash map always unmatched when concurrent occur. So a write lock be added to ensure concurrency.
2023-05-19 19:51:22 +08:00
ae1577e95c [improvement](jdbc catalog) set oceanbase mysql mode jdbc param useCursorFetch default true (#19856) 2023-05-19 19:45:22 +08:00
68be81363b [enhance](Nereids): Pushdown Filter Through Project in Post Processor. (#19873)
Originally, PushdownFilterThroughProject is in CBO phase, but it will increase Memo size.
So, we move it into PostProcessor
2023-05-19 19:27:52 +08:00
67dc68630b [Improve](complex-type)improve array/map/struct creating and function with decimalv3 (#19830) 2023-05-19 17:43:36 +08:00
2ab844550f [feature-wip](MTMV) support multi catalog (#19854)
* mtmv support multi catalog

* mtmv support multi catalog
2023-05-19 16:44:55 +08:00
0fc8d2e029 [Bug](decimal) fix variance_samp and avg_weighted #19861 2023-05-19 16:44:36 +08:00
f46f0c84b2 [Enhancement](meta) Show remote data usage via SHOW DATA #19533 (#19752)
* [Enhancement](meta) Show remote data usage via SHOW DATA #19533

* [fix] correct some unit test results
2023-05-19 14:23:50 +08:00
c4900eb658 [Bug](DecimalV3) fix decimalv3 functions (#19801) 2023-05-19 14:10:01 +08:00
fcffb1d3de [minor](Nereids): add toString() for LogicalProperties (#19851) 2023-05-19 13:46:47 +08:00
92c6a3c53b [fix](Nereids) normalize repeat generate push down project with error nullable (#19831) 2023-05-19 13:15:42 +08:00
9c86cad4ec [improvement](session variable) add max execution time session variabe like mysql and add setter attributes in variables (#19759)
1. add session variable max_execution_time to an alias of query timeout, if user set max_execution_time, the query timeout will be modified too.
2. add a setter attribute to session variable, so that we could add some logic in setter method instead of field reflection.
2023-05-19 12:42:47 +08:00
cf7083d58b [explain](point query) modify explain for SHORT-CIRCUIT query (#19820) 2023-05-19 11:50:08 +08:00
609b20bd02 [Feature](planner) use partial update in update from & delete from (#19262) 2023-05-19 09:46:29 +08:00
84bad03ccb [feature](nereids) set proper min/max value for column stats when minExpr/maxExpr is not avialable #19673 2023-05-19 09:02:40 +08:00
0dd361dbf7 [fix](tracing) fix the issue that a trace may track multiple queries (#19804) 2023-05-19 08:58:53 +08:00
6f6d744a2a [fix](nereids) avoid 0 row count in stats derive #19640
row count of join estimation is at least 1 to make less error propagation.
2023-05-19 08:54:24 +08:00
14620a6766 [minor](log) add details for unqueryable replicas (#19792)
Add a new FE config: show_details_for_unaccessible_tablet.
Default is false, when set to true, if a query is unable to select a healthy replica,
the detailed information of all the replicas of the tablet including the specific reason why they are unqueryable,
will be printed out.
2023-05-19 08:53:57 +08:00
dc8a992bba [improve](nereids) check be status when column stats is unknown #19742
when forbid_unknown_col_stats is open and some column stats is unknown,
we will check the be status by StatisticsUtil.statsTblAvailable(), and report error according to be status.
2023-05-19 08:53:34 +08:00
1e8eb1c756 [fix](profile) Fix pipeline load channel profile #19828 2023-05-19 08:51:02 +08:00
adc5522c9b [bug](MTMV) Fix the wrong interpretation for NEVER REFRESH (#19800) 2023-05-18 23:56:56 +08:00
dfc4432e83 [improvement](jdbc catalog) Add adaptation to Oracle special character / table names (#19809) 2023-05-18 22:58:33 +08:00
f2b2a568de [fix](jdbc catalog)fixed oceanbase catalog row limit bug (#19796) 2023-05-18 22:05:51 +08:00
40ab4ce305 fix select resource groups bug (#19808) 2023-05-18 21:54:31 +08:00
f68d3a660e [improvement](opentelemetry) upgrade opentelemetry jar to v1.26.0 and opentelemetry-cpp to v1.8.3 (#19733)
why upgrade? anything wrong?

Try to fix the problem about opentelemetry::v1::ext::http::client::curl::HttpOperation::Send(), I have updated the pr info.
2023-05-18 18:46:20 +08:00
ed85a10a70 [Fix](multi-catalog) Fix sync hms event failed. (#19555)
A similar situation with #19344 , because sometimes hms meta info is newer than hms events, if we try to invoke org.apache.doris.datasource.hive.PooledHiveMetaStoreClient#getTable and this table is not exists, some error will throws and this event can not be handled.
2023-05-18 18:34:18 +08:00
e67872d391 [fix](Nereids) fallback not work when cannot parse after forward (#19790) 2023-05-18 18:24:59 +08:00
294599ee45 [feature](jsonb) rename JSONB type name and function name to JSON (#19774)
To be more compatible with MySQL, rename JSONB type name and function name to JSON.

The old JSONB type name and jsonb_xx function can still be used for backward compatibility.

There is a function jsonb_extract remained since json_extract is used by json string function and more work need to change it. It will be changed further.
2023-05-18 16:16:52 +08:00
160d2be0d8 [minimal](Nereids) add more comments for the rewriter (#19788)
Only add some comments to the rewriter. Because it is fewer comments before and it's hard to understand for the newbie.
2023-05-18 14:47:25 +08:00
e45bc160c9 [fix](mtmv) fix bug that should not write edit log when replaying alter mv (#19781) 2023-05-18 13:34:05 +08:00
50370dead9 [fix](load) fix unified load converted failed when forwarding to master (#19779) 2023-05-18 12:28:32 +08:00
18c1081659 [fix](nereids) fix some nereids bugs (#19711)
1. add json_unquote and json_extract functions
2. remove mv releated code in visitPhysicalOlapScan
3. forbid bitmap and hll type for topn node's sort exprs
4. HashDistributionInfo of olap scan node should use the slots from output not the full schema
5. SelectMaterializedIndexWithoutAggregate should use the filter node's output together with the predicate to get the correct mv
6. forbid SimplifyArithmeticRule for decimal type
7. make DecimalLiteral's type and value consistent with each other if the value is decimalv2
8. json_array need support empty argument
2023-05-18 11:33:56 +08:00
a3f06e5fbd [git](Nereids): ignore apache parquet. (#19765) 2023-05-18 10:54:46 +08:00
88ca4f3e6b [feature](like) make like regexp used as a sql function (#19755) 2023-05-18 10:03:12 +08:00
098dac20c2 [log](Nereids): add more debug info when check logicalproperties. (#19763) 2023-05-18 08:37:10 +08:00
c80c4477cf [Enhancement](broker-load) broker load show stmt support display cluster name if specified (#19392) 2023-05-18 00:10:15 +08:00
97d4778ecf [enhancement](schema) dynamic_partition.time_unit support year (#19551)
dynamic_partition.time_unit support year
2023-05-17 23:49:15 +08:00
8aa7f0e188 [fix](catalog) fix the include_database_list not in effect (#19589) 2023-05-17 22:56:21 +08:00
60d5c82f44 [fix](tvf) fix the inconsistency between tvf backends function and show backends result (#19697) 2023-05-17 22:55:46 +08:00
be47a27013 [Fix](multi catalog, nereids)Fix FileQueryScanNode couldn't filter partition in nereids planner bug (#19564)
Nereids planner add conjuncts to ScanNode after call finalize, this may cause external table scan node fail to filter 
useless partition, because external table do the partition prune in the finalize method.

This pr is to fix this bug. In the rewrite stage, pass the conjuncts to LogicalFileScan object, and eventually pass to 
ScanNode while creating it. So that the ScanNode could use the conjuncts while doing finalize. 

Why not doing the partition prune in the LogicalFileScan like LogicalOlapScan doing? 

Because Iceberg api doesn't have the partition concept, it just accept a list of Conjuncts, 
so it's easier to pass the conjuncts to ScanNode (Hive, Icegerg, Hudi...) and doing the partition prune in there.
2023-05-17 21:39:59 +08:00
b5f60bde42 [fix](checkpoint)fix Checkpoint error when http server is not ready #19699 2023-05-17 21:33:56 +08:00