fix a bug when using grouping sets without all column in a grouping set item will produce wrong value.
fix grouping function check will not work in group by clause
This CL implements a simulated FE process and a simulated BE service.
You can view their specific usage methods at
`fe/src/test/java/org/apache/doris/utframe/DemoTest.java`
At the same time, I modified the configuration of the maven-surefire-plugin plugin,
so that each unit test runs in a separate JVM, which can avoid conflicts caused by
various singleton classes in FE.
Starting a separate jvm for each unit test will bring about 30% extra time overhead.
However, you can control the number of concurrency of unit tests by setting the `forkCount`
configuration of the maven-surefire-plugin plugin in `fe/pom.xml`. The default configuration
is still 1 for easy viewing of the output log. If set to 3, the entire FE unit test run time is about
4 minutes.
**Describe the bug**
**First**, In the broker load, we allow users to add multiple data descriptions. Each data description
represents a description of a file (or set of files). Including file path, delimiter, table and
partitions to be loaded, and other information.
When the user specifies multiple data descriptions, Doris currently aggregates the data
descriptions belonging to the same table and generates a unified load task.
The problem here is that although different data descriptions point to the same table,
they may specify different partitions. Therefore, the aggregation of data description
should not only consider the table level, but also the partition level.
Examples are as follows:
data description 1 is:
```
DATA INFILE("hdfs://hdfs_host:hdfs_port/input/file1")
INTO TABLE `tbl1`
PARTITION (p1, p2)
```
data description 2 is:
```
DATA INFILE("hdfs://hdfs_host:hdfs_port/input/file2")
INTO TABLE `tbl1`
PARTITION (p3, p4)
```
What user expects is to load file1 into partition p1 and p2 of tbl1, and load file2 into paritition
p3 and p4 of same table. But currently, it will be aggregated together, which result in loading
file1 and file2 into all partitions p1, p2, p3 and p4.
**Second**, the following 2 data descriptions are not allowed:
```
DATA INFILE("hdfs://hdfs_host:hdfs_port/input/file1")
INTO TABLE `tbl1`
PARTITION (p1, p2)
DATA INFILE("hdfs://hdfs_host:hdfs_port/input/file2")
INTO TABLE `tbl1`
PARTITION (p2, p3)
```
They have overlapping partition(p2), which is not support yet. And we should throw an Exception
to cancel this load job.
**Third**, there is a problem with the code implementation. In the constructor of
`OlapTableSink.java`, we pass in a string of partition names separated by commas.
But at the `OlapTableSink` level, we should be able to pass in a list of partition ids directly,
instead of names.
ISSUE: #2823
Support replication_num setting for table level, so There is no need for user to set replication_num for every alter table add partition statement.
eg:
`alter table tbl set ("default.replication_num" = "2");`
If we connect to a non-master FE and execute `show routine load;`. It may sometimes
throw Unknown Exception, because some of fields in thrift result is not set.
From January 15th, 2020, Requests to http://repo1.maven.org/maven2/ return a 501 HTTPS Required status.
So switch central repository url from http to https
Support Grouping Sets, Rollup and Cube to extend group by statement
support GROUPING SETS syntax
```
SELECT a, b, SUM( c ) FROM tab1 GROUP BY GROUPING SETS ( (a, b), (a), (b), ( ) );
```
cube or rollup like
```
SELECT a, b,c, SUM( d ) FROM tab1 GROUP BY ROLLUP|CUBE(a,b,c)
```
[ADD] support grouping functions in expr like grouping(a) + grouping(b) (#2039)
[FIX] fix analyzer error in window function(#2039)
Standardize the return results of INSERT operations,
which is convenient for users to use and locate problems.
More details can be found in insert-into-manual.md
This commit replaces the PowerMock/EasyMock in our unit tests, But not all.
PS.(The tests relevant to DescribeStmt are ignored until I find a way to fix it)
This CL changes:
1. add function bitmap_to_string and bitmap_from_string, which will
convert a bitmap to/from string which contains all bit in bitmap
2. add function murmur_hash3_32, which will compute murmur hash for
input strings
3. make the function cast float to string the same with user result
logic
In previous versions, if the children of the IN predicate included NULL, all child types would be converted to DOUBLE for calculation.
For example:
select * from t1 where k1 in ('TABLE', NULL);
But children like varchar cannot be converted to double, so the query cannot be executed.
The error is "TABLE is not a number"
The current version, if null exists in the child, it will not enter the calculation of compatibility type.
For the above query, the compatibility type is varchar, so the 'TABLE' is not converted to double, and the query could be executed.
Also, for JDBC. It will convert 'show tables;' to :
```
SELECT
TABLE_SCHEMA AS TABLE_CAT, NULL AS TABLE_SCHEM, TABLE_NAME,
CASE WHEN TABLE_TYPE='BASE TABLE'
THEN CASE WHEN TABLE_SCHEMA = 'mysql' OR TABLE_SCHEMA = 'performance_schema'
THEN 'SYSTEM TABLE' ELSE 'TABLE'END WHEN TABLE_TYPE='TEMPORARY'
THEN 'LOCAL_TEMPORARY' ELSE TABLE_TYPE END AS TABLE_TYPE, TABLE_COMMENT AS REMARKS, NULL AS TYPE_CAT, NULL AS TYPE_SCHEM, NULL AS TYPE_NAME, NULL AS SELF_REFERENCING_COL_NAME, NULL AS REF_GENERATION
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA LIKE 'test_db'
AND TABLE_NAME LIKE '%'
HAVING TABLE_TYPE IN ('TABLE','VIEW',null,null,null)
ORDER BY TABLE_TYPE, TABLE_SCHEMA, TABLE_NAME
```
In previous version, Doris could not return the correct tables to JDBC. It will thrown the error "'TABLE' is not a number".
After this commit, #2729 is fixed. Doris could return the tables schema by JDBC.
This CL make bitmap_count, bitmap_union, and bitmap_union_count accept any expression whose return type is bitmap as input so that we can support flexible bitmap expression such as bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))).
This CL also create separate documentation for each bitmap UDF to conform with other functions.
This commit contains the following changes:
1. Let create/alter view statement support cte sql. (Issue #2625 )
e.g.
```
Alter view test_tbl_view (h1, h2)
as
with testTbl_cte (w1, w2) as
(
select col1, col2 from testDb.testTbl
)
select w1 as c1, sum(w2) as c2 from testTbl_cte
where w1 > 10
group by w1 order by w1
```
2. Fix the bug that view's schema remains unchanged after replaying alter view. (Issue #2624 )
"Type" is a abstract class, it has 4 sub classes:
1. ScalarType
2. ArrayType
3. MapType
4. StructType
This CL only support ScalarType. Other types can be added later.
the num segments should be read from rowset meta pb.
But the previous code error caused this value not to be set in some cases.
So when init the rowset meta and find that the num_segments is 0(not set),
we will try to calculate the num segments from AlphaRowsetExtraMetaPB,
and then set the num_segments field.
This should only happen in some rowsets converted from old version.
and for all newly created rowsets, the num_segments field must be set.
