Support replication_num setting for table level, so There is no need for user to set replication_num for every alter table add partition statement.
eg:
`alter table tbl set ("default.replication_num" = "2");`
Note that the methods in path_util are only related to path processing,
and do not involve any file and IO operations
The upcoming patch will use these util methods, used to extract operations
such as concatenation of directory strings from processing logic.
If we connect to a non-master FE and execute `show routine load;`. It may sometimes
throw Unknown Exception, because some of fields in thrift result is not set.
From January 15th, 2020, Requests to http://repo1.maven.org/maven2/ return a 501 HTTPS Required status.
So switch central repository url from http to https
Support Grouping Sets, Rollup and Cube to extend group by statement
support GROUPING SETS syntax
```
SELECT a, b, SUM( c ) FROM tab1 GROUP BY GROUPING SETS ( (a, b), (a), (b), ( ) );
```
cube or rollup like
```
SELECT a, b,c, SUM( d ) FROM tab1 GROUP BY ROLLUP|CUBE(a,b,c)
```
[ADD] support grouping functions in expr like grouping(a) + grouping(b) (#2039)
[FIX] fix analyzer error in window function(#2039)
Fixes#2771
Main changes in this CL
* RoaringBitmap is renamed to BitmapValue and moved into bitmap_value.h
* leveraging Roaring64Map to support unsigned BIGINT for BITMAP type
* introduces two new format (SINGLE64 and BITMAP64) for BITMAP type
So far we have three storage format for BITMAP type
```
EMPTY := TypeCode(0x00)
SINGLE32 := TypeCode(0x01), UInt32LittleEndian
BITMAP32 := TypeCode(0x02), RoaringBitmap(defined by https://github.com/RoaringBitmap/RoaringFormatSpec/)
```
In order to support BIGINT element and keep backward compatibility, introduce two new format
```
SINGLE64 := TypeCode(0x03), UInt64LittleEndian
BITMAP64 := TypeCode(0x04), CustomRoaringBitmap64
```
Please note that SINGLE64/BITMAP64 doesn't replace SINGLE32/BITMAP32. Doris will choose the smaller (in terms of space) type automatically during serializing. For example, BITMAP32 is preferred over BITMAP64 when the maximum element is <= UINT32_MAX. This will also make BE rollback possible as long as user didn't write element larger than UINT32_MAX into bitmap column.
Another important design decision is that we fork and maintain our own version of Roaring64Map instead of using the one in "roaring/roaring64map.hh". The reasons are
1. RoaringBitmap doesn't define a standard for the binary format of 64-bits bitmap. As a result, different implementations of Roaring64Map use different format. For example the [C++ version](https://github.com/RoaringBitmap/CRoaring/blob/v0.2.60/cpp/roaring64map.hh#L545) is different from the [Java version](35104c564e/src/main/java/org/roaringbitmap/longlong/Roaring64NavigableMap.java (L1097)). Even for CRoaring, the format may change in future releases. However Doris require the serialized format to be stable across versions. Fork is a safe way to achieve this.
2. We may want to make some code changes to Roaring64Map according to our needs. For example, in order to use the BITMAP32 format when the maximum element can be represented in 32 bits, we may want to access the private member of Roaring64Map. Another example is we want to further customize and optimize the format for BITMAP64 case, such as using vint64 instead of uint64 for map size.
The `TResourceInfo` was used to help `cgruops` to isolate resources,
but it is no longer used.
In fact, the `TResourceInfo` information is no longer carried in
the requests from FE to BE.
Standardize the return results of INSERT operations,
which is convenient for users to use and locate problems.
More details can be found in insert-into-manual.md
This commit replaces the PowerMock/EasyMock in our unit tests, But not all.
PS.(The tests relevant to DescribeStmt are ignored until I find a way to fix it)
This CL changes:
1. add function bitmap_to_string and bitmap_from_string, which will
convert a bitmap to/from string which contains all bit in bitmap
2. add function murmur_hash3_32, which will compute murmur hash for
input strings
3. make the function cast float to string the same with user result
logic
The problem of inconsistence style in Doris code is too big, it's hard to minimize modification when reformatting code.
So here, our aim is to make the style rules, tune the config in .clang-format.
Note: I choose clang-format-8.0+ to support richer sytle options.
In previous versions, if the children of the IN predicate included NULL, all child types would be converted to DOUBLE for calculation.
For example:
select * from t1 where k1 in ('TABLE', NULL);
But children like varchar cannot be converted to double, so the query cannot be executed.
The error is "TABLE is not a number"
The current version, if null exists in the child, it will not enter the calculation of compatibility type.
For the above query, the compatibility type is varchar, so the 'TABLE' is not converted to double, and the query could be executed.
Also, for JDBC. It will convert 'show tables;' to :
```
SELECT
TABLE_SCHEMA AS TABLE_CAT, NULL AS TABLE_SCHEM, TABLE_NAME,
CASE WHEN TABLE_TYPE='BASE TABLE'
THEN CASE WHEN TABLE_SCHEMA = 'mysql' OR TABLE_SCHEMA = 'performance_schema'
THEN 'SYSTEM TABLE' ELSE 'TABLE'END WHEN TABLE_TYPE='TEMPORARY'
THEN 'LOCAL_TEMPORARY' ELSE TABLE_TYPE END AS TABLE_TYPE, TABLE_COMMENT AS REMARKS, NULL AS TYPE_CAT, NULL AS TYPE_SCHEM, NULL AS TYPE_NAME, NULL AS SELF_REFERENCING_COL_NAME, NULL AS REF_GENERATION
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA LIKE 'test_db'
AND TABLE_NAME LIKE '%'
HAVING TABLE_TYPE IN ('TABLE','VIEW',null,null,null)
ORDER BY TABLE_TYPE, TABLE_SCHEMA, TABLE_NAME
```
In previous version, Doris could not return the correct tables to JDBC. It will thrown the error "'TABLE' is not a number".
After this commit, #2729 is fixed. Doris could return the tables schema by JDBC.
This CL make bitmap_count, bitmap_union, and bitmap_union_count accept any expression whose return type is bitmap as input so that we can support flexible bitmap expression such as bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))).
This CL also create separate documentation for each bitmap UDF to conform with other functions.
For #2722
In our test environment, Doris cluster used 1 fe and 7 be (32C+128G). When using spakr-doris connecter to query a table containing 67 columns, it took about 1 hour for the query to return 69 million rows of data. After the improvement, the same query condition took 2.5 minutes and the query performance was significantly improved
Row will be scanned mistakenly after cumulative compaction on singleton rowset.
If I have (1, 1), (2, 2), (3, 3) three records.
Now I have read (1, 1), this bug will make return row is (2, 2)
instead of (1, 1).
This commit contains the following changes:
1. Let create/alter view statement support cte sql. (Issue #2625 )
e.g.
```
Alter view test_tbl_view (h1, h2)
as
with testTbl_cte (w1, w2) as
(
select col1, col2 from testDb.testTbl
)
select w1 as c1, sum(w2) as c2 from testTbl_cte
where w1 > 10
group by w1 order by w1
```
2. Fix the bug that view's schema remains unchanged after replaying alter view. (Issue #2624 )
