Commit Graph

192 Commits

Author SHA1 Message Date
70cc6df415 [Doc] Fix some typo (#3024) 2020-03-02 22:13:47 +08:00
511c5eed50 [Doc] Modify format of some docs (#3021)
Format of some docs are incorrect for building the doc website.
* fix a bug that `gensrc` dir can not be built with -j.
* fix ut bug of CreateFunctionTest
2020-03-02 19:07:52 +08:00
ef4bb0c011 [RoutineLoad] Auto Resume RoutineLoadJob (#2958)
When all backends restart, the routine load job can be resumed.
2020-03-02 13:27:35 +08:00
df56588bb5 [Temp Partition] Support add/drop/replace temp partitions (#2828)
This CL implements 3 new operations:

```
ALTER TABLE tbl ADD TEMPORARY PARTITION ...;
ALTER TABLE tbl DROP TEMPORARY PARTITION ...;
ALTER TABLE tbl REPLACE TEMPORARY PARTITION (p1, p2, ...);
```

User manual can be found in document:
`docs/documentation/cn/administrator-guide/alter-table/alter-table-temp-partition.md`

I did not update the grammar manual of `alter-table.md`.
This manual is too confusing and too big, I will reorganize this manual after.

This is the first part to implement the "overwrite load" feature mentioned in issue #2663.
I will implement the "load to temp partition" feature in next PR.

This CL also add GSON serialization method for the following classes (But not used):

```
Partition.java
MaterializedIndex.java
Tablet.java
Replica.java
```
2020-03-01 21:30:34 +08:00
0d1e28746e [Function] Support null_or_empty function (#2977)
It returns true if the string is empty or NULL. Otherwise it returns false.
2020-03-01 17:35:45 +08:00
078e35a62e Support Amazon S3 data source in Broker Load (#3004) 2020-03-01 12:53:50 +08:00
2ac07a8c07 [Doc] Fix docs mixed Chinese and English (#3017) 2020-02-28 16:36:37 +08:00
54b7828c3f [Doc] The doc of max_running_txn_num_per_db config (#3007) 2020-02-27 14:57:46 +08:00
57483ade00 [Doc] Fix typo in Chinese document (#2963)
Fix some errors in Chinese document
2020-02-25 22:30:21 +08:00
wyb
fc2d92d68a Update spark load doc (#2973) 2020-02-22 12:00:50 +08:00
96248058a1 [Doc] Modify the default port of restore meta instance in document (#2971) 2020-02-21 22:05:30 +08:00
70d2ccf384 Add spark load design (#2856) 2020-02-21 14:32:18 +08:00
cc0d41277c [Alter] Add more schema change to varchar type (#2777) 2020-02-19 23:14:43 +08:00
3994b52f34 [Alter] Change max create replicas timeout configurable (#2945) 2020-02-19 17:47:27 +08:00
87a84a793e Add more doc content about hdfs broker auth and config detail (#2935) 2020-02-18 21:15:33 +08:00
625411bd28 Doris support in memory olap table (#2847) 2020-02-18 10:45:54 +08:00
3e160aeb66 [GroupingSet] fix a bug when using grouping set without all column in a grouping set item (#2877)
fix a bug when using grouping sets without all column in a grouping set item will produce wrong value.
fix grouping function check will not work in group by clause
2020-02-12 21:50:12 +08:00
59dd2a0d7f Fix some typo in bitmap index docs. (#2879) 2020-02-11 19:03:42 +08:00
1f001481ae Support batch add and drop rollup indexes #2671 (#2781) 2020-02-11 12:58:01 +08:00
d8a47f9a7f [Doc] Fix CANCEL DECOMMISSION syntax bug (#2860)
Fix CANCEL DECOMMISSION syntax bug
2020-02-08 13:07:19 +08:00
581c771ff3 [Doc] Add create index usage document (#2832) 2020-02-05 15:35:41 +08:00
89c7234c1c Support starts_with (str, prefix) function (#2813)
Support starts_with function
2020-01-21 14:09:08 +08:00
acc89411dc Fix docs sequence error (#2814) 2020-01-20 22:35:40 +08:00
010f6cd1c1 Update installing/compilation.md (#2816)
Fix docker images version
2020-01-20 22:27:22 +08:00
58ff952837 [Stmt] Support new show functions syntax to make user search function more conveniently (#2800)
SHOW [FULL] [BUILTIN] FUNCTIONS [IN|FROM db] [LIKE 'function_pattern'];
2020-01-20 14:14:42 +08:00
fc55423032 [SQL] Support Grouping Sets, Rollup and Cube to extend group by statement
Support Grouping Sets, Rollup and Cube to extend group by statement
support GROUPING SETS syntax 
```
SELECT a, b, SUM( c ) FROM tab1 GROUP BY GROUPING SETS ( (a, b), (a), (b), ( ) );
```
cube  or rollup like 
```
SELECT a, b,c, SUM( d ) FROM tab1 GROUP BY ROLLUP|CUBE(a,b,c)
```

[ADD] support grouping functions in expr like grouping(a) + grouping(b) (#2039)
[FIX] fix analyzer error in window function(#2039)
2020-01-17 16:24:02 +08:00
3b24287251 Support 64 bits integers for BITMAP type (#2772)
Fixes #2771 

Main changes in this CL
* RoaringBitmap is renamed to BitmapValue and moved into bitmap_value.h
* leveraging Roaring64Map to support unsigned BIGINT for BITMAP type
* introduces two new format (SINGLE64 and BITMAP64) for BITMAP type

So far we have three storage format for BITMAP type

```
EMPTY := TypeCode(0x00)
SINGLE32 := TypeCode(0x01), UInt32LittleEndian
BITMAP32 := TypeCode(0x02), RoaringBitmap(defined by https://github.com/RoaringBitmap/RoaringFormatSpec/)
```

In order to support BIGINT element and keep backward compatibility, introduce two new format

```
SINGLE64 := TypeCode(0x03), UInt64LittleEndian
BITMAP64 := TypeCode(0x04), CustomRoaringBitmap64
```

Please note that SINGLE64/BITMAP64 doesn't replace SINGLE32/BITMAP32. Doris will choose the smaller (in terms of space) type automatically during serializing. For example, BITMAP32 is preferred over BITMAP64 when the maximum element is <= UINT32_MAX. This will also make BE rollback possible as long as user didn't write element larger than UINT32_MAX into bitmap column.

Another important design decision is that we fork and maintain our own version of Roaring64Map instead of using the one in "roaring/roaring64map.hh". The reasons are

1. RoaringBitmap doesn't define a standard for the binary format of 64-bits bitmap. As a result, different implementations of Roaring64Map use different format. For example the [C++ version](https://github.com/RoaringBitmap/CRoaring/blob/v0.2.60/cpp/roaring64map.hh#L545) is different from the [Java version](35104c564e/src/main/java/org/roaringbitmap/longlong/Roaring64NavigableMap.java (L1097)). Even for CRoaring, the format may change in future releases. However Doris require the serialized format to be stable across versions. Fork is a safe way to achieve this.
2. We may want to make some code changes to Roaring64Map according to our needs. For example, in order to use the BITMAP32 format when the maximum element can be represented in 32 bits, we may want to access the private member of Roaring64Map. Another example is we want to further customize and optimize the format for BITMAP64 case, such as using vint64 instead of uint64 for map size.
2020-01-17 14:13:38 +08:00
8df63bc191 [Doc] Add en doc for dynamic partition feature (#2764) 2020-01-16 21:54:26 +08:00
0ddca59d36 Add timestampadd/timestampdiff function (#2725) 2020-01-15 21:47:07 +08:00
9e54751098 [Snapshot] Modify the prefer snapshot version (#2748)
In this CL, prefer snapshot version in snapshot request is defined
in thrift. So that both FE and BE can use this version value.
2020-01-15 15:10:14 +08:00
7768629f08 Add bitmap_contains and bitmap_has_any functions (#2752) 2020-01-15 14:31:44 +08:00
e5717efc5a [Insert] Return more info of insert operation (#2718)
Standardize the return results of INSERT operations,
which is convenient for users to use and locate problems.

More details can be found in insert-into-manual.md
2020-01-15 10:39:53 +08:00
a36193dfab Support decimal and timestamp type in orc load (#2759) 2020-01-15 07:40:30 +08:00
f071d5a307 Support ends_with function (#2746) 2020-01-14 22:37:20 +08:00
e5197eff94 Update the doc of doris to fix some mistakes (#2758) 2020-01-14 22:26:49 +08:00
a99a49a444 Add bitamp_to_string function (#2731)
This CL changes:

1. add function bitmap_to_string and bitmap_from_string, which will
 convert a bitmap to/from string which contains all bit in bitmap
2. add function murmur_hash3_32, which will compute murmur hash for
input strings
3. make the function cast float to string the same with user result
logic
2020-01-13 12:31:37 +08:00
4e868252fc Add .clang-format and docs (#2724)
The problem of inconsistence style in Doris code is too big, it's hard to minimize modification when reformatting code.
So here, our aim is to make the style rules, tune the config in .clang-format.

Note: I choose clang-format-8.0+ to support richer sytle options.
2020-01-11 20:54:20 +08:00
ccaa97a5ac Make bitmap functions accept any expression that returns bitmap (#2728)
This CL make bitmap_count, bitmap_union, and bitmap_union_count accept any expression whose return type is bitmap as input so that we can support flexible bitmap expression such as bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))).

This CL also create separate documentation for each bitmap UDF to conform with other functions.
2020-01-11 14:02:12 +08:00
6bc54ef3f0 [Document] Add dynamic partition docs (#2711) 2020-01-08 23:08:48 +08:00
a028c52edd Add BE function bitmap_or and bitmap_and (#2707) 2020-01-08 19:59:44 +08:00
e94d4656d8 Fix roles typo in privilege document (#2702) 2020-01-08 15:39:10 +08:00
367b4c058c [Doc]Remove used doc content about be_rpc_port (#2694) 2020-01-07 22:20:50 +08:00
42dfe1369b Add filter conditions for 'show partitions from table' syntax (#2553)
Add filter conditions for show partitions from table syntax, to filter partitions needed
2020-01-03 19:52:25 +08:00
c098178f7a [Index] Implements create drop show index syntax for bitmap index [#2487] (#2573)
### create table with index 
```
CREATE TABLE table1
(
    siteid INT DEFAULT '10',
    citycode SMALLINT,
    username VARCHAR(32) DEFAULT '',
    pv BIGINT SUM DEFAULT '0',
    INDEX index_name [USING BITMAP] (siteid, citycode) COMMENT 'balabala'
)
AGGREGATE KEY(siteid, citycode, username)
DISTRIBUTED BY HASH(siteid) BUCKETS 10
PROPERTIES("replication_num" = "1");
```
### create index 
```
CREATE INDEX index_name  ON table1 (siteid, citycod) [USING BITMAP] COMMENT 'balabala';
or 
ALTER TABLE table1 ADD  INDEX index_name  [USING BITMAP] (siteid, citycod) COMMENT 'balabala';
```
### drop index
```
DROP INDEX index_name ON table1;
or
ALTER TABLE table1 DROP INDEX index_name 
```

### show index
```
SHOW INDEX[ES] FROM table1
```
output
```
+---------+-------------+-----------------+------------+---------+
| Table   | Index_name  | Column_name     | Index_type | Comment |
+---------+-------------+-----------------+------------+---------+
| table1  | index_name  | siteid,citycode | BITMAMP    | balabala|
+---------+-------------+-----------------+------------+---------+
```
2020-01-03 17:41:26 +08:00
3b5f608df7 Delete documents of count_distinct function (#2642) 2020-01-02 23:35:10 +08:00
4220e09724 Change introduction document (#2640)
change FE default maximum memory introduction document from 2G to 4G in english
2020-01-02 23:03:56 +08:00
b440ff286f Change introduction document (#2639)
Change FE default maximum memory introduction document  from 2G to 4G
2020-01-02 23:03:17 +08:00
9783fb7221 Fix: UDF version `GLIBCXX_3.4.21' not found (#2629) 2019-12-31 18:32:42 +08:00
3022955f32 Fix recover time in docs (#2588) 2019-12-27 14:51:41 +08:00
1113f951c3 Alter view stmt (#2522)
This commit adds a new statement named alter view, like
ALTER VIEW view_name
(
col_1,
col_2,
col_3,
)
AS SELECT k1, k2, SUM(v1) FROM exampleDb.testTbl GROUP BY k1,k2
2019-12-27 14:02:56 +08:00