In some scenarios, when a user creates an olap table that is range partition by time, the user needs to periodically add and remove partitions to ensure that the data is valid. As a result, adding and removing partitions dynamically can be very useful for users.
* [Alter Table] No need to check whether table is stable when doing some kinds of alter operation.
Not all alter table operation require table to be stable. Such as rename, modify meta data.
ISSUE #1553
This commit will remove function count_distinct().
We already have function multi_distinct_count as an alternative to help us calculate "count distinct" of any type value.
Besides, the count_distinct() function is with the the same symbol as count() function, which fails to express the meaning.
So I suggest to remove count_distinct() function.
This commit adds a new statement named alter view, like
ALTER VIEW view_name
(
col_1,
col_2,
col_3,
)
AS SELECT k1, k2, SUM(v1) FROM exampleDb.testTbl GROUP BY k1,k2
Support compaction operation to compact only one rowset.
After the modification, the last rowset of the tablet will
also be compacted.
At the same time, we added a `segments_overlap_pb` field to
the rowset meta. Used to describe whether the segment data
in the rowset overlaps. This field is set by `rowset_writer`.
Initially UNKNOWN for compatibility with existing data.
In addition, the version hash of the rowset generated after
compaction is directly set to the version hash of last rowset
participating in compaction, to ensure that the tablet's
version hash remains unchanged after compaction.
When processing tablet reports, some tablets carry transaction information.
This information is used by the FE to determine whether to publish these
transactions or clear these transactions.
During this process, Doris may try to obtain the commit information of some
deleted partitions, resulting in a null pointer exception.
Support time zone variable like "-8:00","+8:00","8:00"
Time zone variable like "-8:00" is illegal in time-zone ID ,so we mush transfer it to standard format
Some export job from old version of Doris may not has timeout property,
which will cause NPE.
2 more changes:
1. Change the default BE config "max_runnings_transactions" to 2000.
2. Add a new metric to FE to show the master ip:port.
Those query of issue could not be supported. #2483#2493
Those query is forbidden:
query1: select * from t1 where k1=(select k1 from t2 where t1.k2=t2.k2);
query2: select * from t1 where k1=(select distinct k1 from t2 where t1.k2=t2.k2);
Only sum, max, min, avg and count function could appear on select clause for correlated subquery. #2420
Those query is legal:
query1: select * from t1 where k1=(select avg(k1) from t2 where t1.k2=t2.k2);
to solve the issue #2246.
scheme is as following:
add a optional preferred_rowset_type in TabletMeta for V2 format rollup index tablet
add a boolean session variable use_v2_rollup, if set true, the query will v2 storage format rollup index to process the query.
test queries will be sent to online service to verify the correctness of segment-v2 by send the the same queries to fe with use_v2_rollup set or not to check whether the returned results are the same.
Support to create materialized view
This commit support to create materiliazed view.
The syntax of stmt is following:
CREATE Materialized View [MV name] AS
SELECT select_expr[, select_expr ...]
FROM [Base table name]
GROUP BY column_name[, column_name ...]
ORDER BY column_name[, column_name ...]
The CreateMaterializedViewClause is used to check the semantic of stmt in the first step.
Now, the where, having, limit clause is forbidden in CREATE MATERIALIZED VIEW.
Also the aggregation function is restricted in SUM/MIN/MAX.
The second step is to validate stmt according to metadata of base table.
For example, the aggregate type of mv column must be same as the aggregate type of base column in aggregate table.
The last step is to prepare index of mv and add this new mvJob in Handler.
The handler will asynchronous process this new mvJob.
This commit will promote the priority of the || operator to the front of the + - * / mod operators.
It solves the problems 2.1 that mentioned at issue #2396 .
For problem at 2.2 in issue #2396 , it is actually the same problem mentioned in issue #2142 . As it said in pr #2398 before, the influence of modifying that logic will cause semantic errors in insert and load, so this commit will left the bug unsolved temporary.
appendix:
In Mysql 5.7.27
|| and |
select 23|1||7;
23
select (23|1)||7
237
select 23|(1||7)
23
Priority : || > |
|| and &
select 10&1||7;
0
select (10&1)||7
7
select 10&(1||7)
0
Priority : || > &
|| and ^
select 10^1||7
27
select (10^1)||7
117
select 10^(1||7)
27
Priority : || > ^
|| and ~
select ~1||7
184467440737095516147
select ~(1||7)
18446744073709551598
priority : || < ~
[Tag System]
This CL includes 2 parts:
Add classes related to "tag"
Resource: is the collective name of the nodes that provide various service capabilities in Doris cluster.
Tag: A Tag consists of type and name.
TagSet: TagSet represents a set of tags.
TagManager: maintains 2 indexes:
one is from tag to resource.
one is from resource to tags
ISSUE #1723
Using JSON as serialization methods of metadata
Introduce GSON library to serialize the new classes mentioned above.
ISSUE #2415#2389
GSON's version is updated to 2.8.6
1 Because we don't support array type currently, so I use variable arguments instead.
2 intersect_count directly return final count, not bitmap like bitmap_union, because intersect_count return bitmap is more complex and need more serialize. If we really need bitmap format from intersect_count, we could do that in another PR and which won't have compatibility problems.
The multi cluster feature will be deprecated soon.
Add a FE config "disable_cluster_feature", and default is true, to
forbid any cluster related operations, include:
* create/drop cluster
* add free backend/add backend to cluster/decommission cluster balance
* change the backends num of cluster
* link/migration db
* fix ut
