Add log to record the compaction point changing.
When OLAP_ERR_BE_SEGMENTS_OVERLAPPING happens,
it will be used to track the bugs. Add issue #4134 link.
Using slice->data to create HyperLogLog, it will exec HyperLogLog(Slice(const char*)). Then Slice(const char*) will use strlen(data) to calc the size. But the slice in this unit test isn't a C-string. Need to use Slice.
This commit mainly supports creating bitmap_union, hll_union, and count materialized views.
* The main changes are as follows:
1. When creating a materialized view, doris judge the semantic analysis of the newly supported aggregate function.
Only bitmap_union(to_bitmap(column)), hll_union(hll_hash(column)) and count(column) are supported.
2. Match the correct materialized view when querying.
After the user sends the query, if there is a possibility of matching the materialized view, the query will be rewritten firstly.
Such as:
Table: k1 int, k2 int
MV: k1 int, mv_bitmap_union_k2 bitmap mv_bitmap_union
mv_bitmap_union = to_bitmap(k2)
Query: select k1, count(distinct k2) from Table
Found that there is a match between the materialized view column and the query column, the query is rewritten as:
Rewritten query: select k1, bitmap_union_count(mv_bitmap_union_k2) from table
Then when the materialized view is matched, it can be matched to the query materialized view table.
Sometimes the rewritten query may not match any materialized view, which means that the rewriting failed. The query needs to be re-parsed and executed again.
Related issue #4017, main changes as follows:
1. Add expired_snapshot_rs_version_map，_expired_snapshot_rs_metas，
2. Add VersionedRowsetTracker record compacted path version
3. Record path version when rowsets compact
4. In gc process, add expired snapshot rowsets to unused set to remove.
* [Bug] Fix bug that tablet meta lock twice
The tablet meta load may already be hold before calling
generate_tablet_meta_copy(), so we need provide a unlocked
version of generate_tablet_meta_copy()
* fix typo
Co-authored-by: chenmingyu <chenmingyu@baidu.com>
This commit mainly supports load bitmap_union, hll_union, and count materialized views.
The main changes are as follows:
1、insert stmt support load extend column
2、load stmt support load extend column
Issue : #3344
Co-authored-by: HangyuanLiu <460660956@qq.com>
cpp-mustache is a C++ implementation of a Mustache template engine
with support for RapidJSON, and in order to simplify RapidJSON object
building, we introduce class EasyJson from Apache Kudu.
Result may error when ORC load negative decimal value
When load negative decimal which has pre zero , the result is wrong.
eg -0.0014, the orc result is -14(precision ... 0)
When set `disable_colocate_balance` to false and set some BE to decommission,
`Coloratebalancer#balanceGroup` will choose decommissioned BE to locate tablets,
which is not right
Fix#4102
+ Building the materialized view function for schema_change here based on defineExpr.
+ This is a trick because the current storage layer does not support expression evaluation.
+ count distinct materialized view will set mv_expr with to_bitmap or hll_hash.
+ count materialized view will set mv_expr with count.
+ Support to regenerate historical data when a new materialized view is created in BE。
+ Support to_bitmap function
+ Support hll_hash function
+ Support count(field) function
For #3344
The output columns of query should be collected by all of tupleIds
in BaseTableRef rather than the top tupleIds of query.
The top tupleIds of count(*) is Agg tuple which does not expand the star.
Fixed#4065
Resource release should be done by dest RowBatch.
When we call method transfer_resource_ownership.
if we don't clear the corresponding resources,
which will cause the core problem of double delete.
[Bug]Fix some schema change not work right
This CL mainly fix some schema change to varchar type not work right
because forget to logic check && Add ConvertTypeResolver to add
supported convert type in order to avoid forget logic check
[Bug] Fix the problem of mem exec, when analytic eval node need to spill to disk with a low mem limit.
And clear_reservations of Analytic node reservation of block manager.
[Running Profile] Add Spilled flag in Running Profile, when Analytic eval node and sort node spill to Disk.
This PR is mainly to add `thrift_client_retry_interval_ms` config in be for thrift client
to avoid avalanche disaster in fe thrift server and fix some typo and some rpc
setting problems at the same time.
fix#4047#3886 has certain relevance to this case。
the sql : `bigtable t1 join mysqltable t2 join mysqltable t3 on t1.k1 = t3.k1`
1. after reorder:
t1, t2, t3
2. choose join t1 with t2:
t1 join t2 with no conditions, and Doris choose cross join
3. choose join (t1 join on t2) with t3:
in old code, the t2 is mysqlTable, so the cardinality is zero,
and "the cross join t1 with t2" 's cardinality is t1.cardinality multiply t2.cardinality,
for t2 is mysql, so t2.cardinality is zero, and "the cross join t1 with t2" is zero.
t3 is mysqltable, t3's cardinality is zero.
**If two tables need to be joined both are zero，we will choose the shuffle join**
So I change the mysql table ‘s cardinality from 0 to 1, the cross join's cardinality is not zero.
1. Doris support modify char to varchar. There is a bug when use two-level pointer when converting the date.
2. Boolean can be used as metric value with REPLACE and REPLACE_IF_NOT_NULL aggregation function. The aggregation function should be added into aggregation map.
Its time to enable some features by default.
1. Enable FE plugins by setting `plugin_enable=true`
2. Enable dynamic partition by setting `dynamic_partition_enable=true`
3. Enable nio mysql server by setting `mysql_service_nio_enabled=true`
Also modify installation doc, add download link of MySQL client.
This CL mainly changes:
1. Reorganized the code logic to limit the supported json format to two, and the import behavior is more consistent.
2. Modified the statistical behavior of the number of error rows when loading in json format, so that the error rows can be counted correctly.
3. See `load-json-format.md` to get details of loading json format.
Fixes#3995
## Why does it happen
When SetOperations encounters that the previous node needs Aggregate, the timing of add AggregationNode is wrong. You should add AggregationNode first before add other children.
## Why doesn't intersect and union have this problem
intersect and union conform to the commutation law, so it doesn't matter if the order is wrong
## Why this problem has not been tested before
In the previous test case, not cover the previous node was not AggregationNode
