1. Delete Invalid Counter In Data_Stream_Sender. (#3598)
2. Add Counter For PartitionHashTable of PartitionAggregationNode:
* Hash Probe Method
* Row processed by Aggregation
* HashFilledBuckets: Counter How Many FilledBuckets in Aggragation
* HTResize: Counter How Many Resize of HashTable
* HashProbe: Counter Probe of HashTable
* HashFailedProbe: Counter Failed Probe of HashTable
* HashTravelLength: Total TravelLength for Probe
* HashCollisions: Counter of HashCollision
3. Del some unecessary code in PartitionHashTable by template
The doris rewrite the subquery in case-when to inline view.
So it the result is different between subquery in case-when and inline view.
We could not support the empty set of subquery in case-when.
This commit forbidden this case.
**1. phenomenon：**
The following two statements are the same, but a query has results and the other query has no results
mysql> select * from (select '积极' as kk1, sum(k2) from table_range where k1 = '2013-01-01' group by kk1)tt where kk1 = '积极';
+--------+-----------+
| kk1 | sum(`k2`) |
+--------+-----------+
| 积极 | 1 |
+--------+-----------+
1 row in set (0.01 sec)
mysql> select * from (select '积极' as kk1, sum(k2) from table_range where k1 = '2013-01-01' group by kk1)tt where kk1 in ('积极');
Empty set (0.01 sec)
**2. reason：**
In partition prune, constant in predicate（‘积极’ in ‘积极’） is mistakenly considered to meet partition prune conditions, and mistakenly regarded as partition prune column. Then in partition prune , no partition is considered to meet the requirements, so it is planned to be 0 partition in query planning
This PR is the first step to make Doris stream load more robust with higher concurrent
performance(#3368)，the main work is to support txn management in db level isolation
and use ArrayDeque to stored final status txns.
#3479
Here I try to explain the cause of the problem and how to fix it.
**The Cause of The problem**
Take the case in issue(#3479 ) as an example:
The general results are as follows:
```
GET table/_doc/_search
{"query":{"match_all":{}},"stored_fields":"_none_","docvalue_fields":["k1"],"sort":["_doc"],"size":100}
{
"took": 6,
"timed_out": false,
"_shards": {
……
},
"hits": {
"total": 3,
"max_score": null,
"hits": [
{
"_index": "table",
"_score": null,
"sort": [
0
]
},
{
"_index": "table",
"_score": null,
"fields": {
"k1": [
"kkk1"
]
},
"sort": [
0
]
},
{
"_index": "table",
"_score": null,
"sort": [
0
]
}
]
}
}
```
But in Doris on ES，Be fetched data parallelly on all shards, and use `filter_path` to reduce the network cost. The process will be as follows:
```
GET table/_doc/_search?preference=_shards:1&filter_path=_scroll_id,hits.hits._source,hits.total,_id,hits.hits._source.fields,hits.hits.fields
{"query":{"match_all":{}},"stored_fields":"_none_","docvalue_fields":["k1"],"sort":["_doc"],"size":100}
{
"hits": {
"total": 0
}
}
GET table/_doc/_search?preference=_shards:2&filter_path=_scroll_id,hits.hits._source,hits.total,_id,hits.hits._source.fields,hits.hits.fields
{"query":{"match_all":{}},"stored_fields":"_none_","docvalue_fields":["k1"],"sort":["_doc"],"size":100}
{
"hits": {
"total": 1
}
}
GET table/_doc/_search?preference=_shards:3&filter_path=_scroll_id,hits.hits._source,hits.total,_id,hits.hits._source.fields,hits.hits.fields
{"query":{"match_all":{}},"stored_fields":"_none_","docvalue_fields":["k1"],"sort":["_doc"],"size":100}
{
"hits": {
"total": 1,
"hits": [
{
"fields": {
"k1": [
"kkk1"
]
}
}
]
}
}
```
*Scan-Worker On BE which processed result of shard2 will failed.*
**The reasons are as follows:**
1. "filter_path" causes the hits.hits object not exist.
2. In the current implementation, if there are some data rows（total > 0）, the hits.hits. object must be an array
**How To Fix it**
Two Method:
1. modify "filter_path" to contain the hits.
Pros: Fixed Code is very simple
Cons: More network cost
2. Deal with the case where fields are missing in a batch.
Pros: No loss of performance
Cons: Code is more complex
Performance first, I use Method2.
**Design**
1. Add a variable "_doc_value_mode" into Class "EsScrollParser" to =indicate whether the data processed by this parser is doc_value_mode or not.
2. "_doc_value_mode" is passed from ESScollReader <- ESScanner <- ScrollQueryBuilder::build() that determines whether DSL is enable doc_value_mode
3. When hits.hits of response from ES is empty and total > 0. We know there are data lines, but the corresponding fields do not exist. EsScrollParser will use "_doc_value_mode" and _total to construct _total lines which fields are assigned with 'NULL'
LSAN detected errors have been fixed by a prior pathch (#3326), but
there are still some ASAN detected errors.
This patch try to fix these errors to make Doris BE more robustness.
And then we can add CI run in LSAN/ASAN mode to detect memory errors
as early as possible.
Each test case in ExternalScanContextMgrTest may cost 1 minitue
which is too long, we'd better disable backgrounp scan context
gc to speed up unit test.
The callback added to the CallbackFactory should not be removed until the
transaction is aborted or visible. Otherwise, some callback method may failed
to be called.
When BE sets `ignore_broken_disk` to true, it's expected that non-exist path in storage_root_path won't prevent BE from launching, but in 0.12 BE fails to launch in such scenario.
```
W0506 14:46:11.039953 17040 options.cpp:64] path can not be canonicalized. may be not exist. path=/data11/olap
W0506 14:46:11.040014 17040 options.cpp:141] failed to parse store path /data11/olap, res=-203
```
The reason is that #2861 adds a path existence check in `parse_root_path` which precedes the usage of `ignore_broken_disk` in the main method.
