Commit Graph

17549 Commits

Author SHA1 Message Date
e7b763309d Skip missing version replica when getQueryableReplicas (#2715) 2020-01-09 17:19:23 +08:00
425b1cf29b Fix port already in use (#2716) 2020-01-09 16:01:17 +08:00
f7cea6dda5 CreateViewStmt/AlterViewStmt support cte and fix bug (#2641)
This commit contains the following changes:
1. Let create/alter view statement support cte sql. (Issue #2625 )

e.g.
```
Alter view test_tbl_view (h1, h2)
as
with testTbl_cte (w1, w2) as 
(
    select col1, col2 from testDb.testTbl
)
select w1 as c1, sum(w2) as c2 from testTbl_cte 
where w1 > 10 
group by w1 order by w1
```

2. Fix the bug that view's schema remains unchanged after replaying alter view. (Issue #2624 )
2020-01-08 23:11:38 +08:00
87b266cd11 Add cpp connect sample (#2685) 2020-01-08 23:10:58 +08:00
6bc54ef3f0 [Document] Add dynamic partition docs (#2711) 2020-01-08 23:08:48 +08:00
e309eb1d40 [Compaction] Rowset with only one segment should be considered as non-overlapping (#2700)
Currently all singleton rowsets with data are considered overlapping upon construction, even when the rowset contains only one segment. In the meanwhile, singleton rowsets could be input to base compaction (when the tablet hasn't been compacted for base_compaction_interval_seconds_since_last_operation) as long as they are converted into non-overlapping rowsets by cumulative compactor.

By making rowset with one segment non-overlapping, we can avoid the work to convert such rowset to non-overlapping rowset in cumulative compaction.
2020-01-08 23:05:11 +08:00
1c9cfa7e0f Fix invalid to_bitmap input lead to BE core (#2706) 2020-01-08 22:14:37 +08:00
a028c52edd Add BE function bitmap_or and bitmap_and (#2707) 2020-01-08 19:59:44 +08:00
d4a3b34319 [Meta Serialization] Support GSON serialization for class "Type" (#2709)
"Type" is a abstract class, it has 4 sub classes:
1. ScalarType
2. ArrayType
3. MapType
4. StructType

This CL only support ScalarType. Other types can be added later.
2020-01-08 19:56:56 +08:00
e90170a5d0 Fix bug: map erase in txn_manager (#2705) 2020-01-08 18:53:11 +08:00
e94d4656d8 Fix roles typo in privilege document (#2702) 2020-01-08 15:39:10 +08:00
b2b2b661ee Update notice year to 2020 (#2696) 2020-01-08 12:54:03 +08:00
367b4c058c [Doc]Remove used doc content about be_rpc_port (#2694) 2020-01-07 22:20:50 +08:00
13e5fdd512 [AlphaRowset] set num_segments field in rowset meta if missing (#2658)
the num segments should be read from rowset meta pb.
But the previous code error caused this value not to be set in some cases.
So when init the rowset meta and find that the num_segments is 0(not set),
we will try to calculate the num segments from AlphaRowsetExtraMetaPB,
and then set the num_segments field.
This should only happen in some rowsets converted from old version.
and for all newly created rowsets, the num_segments field must be set.
2020-01-07 21:46:02 +08:00
4e2f01a9fa [Compaction] Fix a bug that CumulativeCompaction compares time of different precision (#2693)
time(NULL) returns second-resolution timestamp, however all compaction related time in Tablet are in millis-resolution. Therefore should use UnixMillis() instead.
2020-01-07 21:31:36 +08:00
844ccaafc9 Remove boost filesystem exception in FileUtils (#2692)
If `errer_code` is provided, then the `boost::filesystem` functions
will not throw an exception, so we do not need to catch it.
2020-01-07 07:29:05 -06:00
7d2610d091 Change bitmap functions return type to BITMAP (#2690) 2020-01-07 19:27:21 +08:00
852046de29 Fix incompatibility with arm architecture in olap #2645 (#2682) 2020-01-07 19:16:10 +08:00
72fd745086 [Load] Fix broker load's file format npe bug (#2689) 2020-01-07 16:50:02 +08:00
23e3149d70 [Variable] Fix default rowset type read (#2687)
fix bug of read default rowset type in HeartbeatFlags.
2020-01-07 15:24:03 +08:00
e2c174aecc Fix mysqlChannel npe in ConnectContext (#2684) 2020-01-07 14:54:08 +08:00
2326b478b6 Support load orc format in Apache Doris (#2554)
Support load orc format in Apache Doris
2020-01-07 14:22:43 +08:00
369b2c364f [Build] fix jemalloc download (#2681) 2020-01-07 11:33:55 +08:00
ec860c82c0 [Variable] Fix default rowset type variable (#2680)
Fix defaultRowsetType's type description in SessionVariable
2020-01-07 10:56:18 +08:00
f77171f85d Make bdbje lock timeout configurable (#2676) 2020-01-06 21:20:36 +08:00
40c8bddd24 Decrease transaction_clean_interval_second config (#2673) 2020-01-06 20:16:37 +08:00
9ca7fdfe1c Make MAX_SCHEDULING_TABLETS and MAX_BALANCING_TABLETS configurable (#2670) 2020-01-06 20:15:38 +08:00
4d6afdae4d Add nio support for mysql protocol implementation (#2603) 2020-01-06 18:56:21 +08:00
de4d1778c6 Fix incompatibility with arm architecture in util and gutil (#2650)
1. upgrade  gutil code from imapla  to new verison, include `cpuinfo`, `spinlock` and `linux_syscall_support `
2. impliments arm version  utf8 check code
3. remove incompatible code from stopwatch
2020-01-06 18:39:31 +08:00
c6badaec91 Fix bug: CreateIndexClause can be casted to AlterTableClause (#2667) 2020-01-06 18:31:40 +08:00
7f148c188e [Build]Make set target arch universal (#2660) 2020-01-06 14:46:07 +08:00
87a50070c4 Fix bug: parquet scanner don't seek (#2661) 2020-01-06 13:55:40 +08:00
8a5ee6ad21 Fix FE couldn't start (#2662) 2020-01-06 12:46:45 +08:00
220ed8436c [Unit Test]Fix Schema Change Test Case (#2659) 2020-01-05 20:08:23 +08:00
1648226927 Adapt arrow 0.15 API (#2657)
This CL supports arrow's zero copy read interface, which can make code
comply with arrow 0.15.
And the schema change unit test has some problem, I disable it in run-ut.sh
2020-01-04 15:54:29 +08:00
af9529a207 [Dynamic Partition] Support for automatically adding partitions
In some scenarios, when a user creates an olap table that is range partition by time, the user needs to periodically add and remove partitions to ensure that the data is valid. As a result, adding and removing partitions dynamically can be very useful for users.
2020-01-03 23:45:04 +08:00
1ca82122a8 Fix doris be compile error for Ubuntu14.04 (#2647) 2020-01-03 21:14:34 +08:00
42dfe1369b Add filter conditions for 'show partitions from table' syntax (#2553)
Add filter conditions for show partitions from table syntax, to filter partitions needed
2020-01-03 19:52:25 +08:00
458ed55fa5 Fix BITMAP_UNION_COUNT couldn't hit rollup table (#2655) 2020-01-03 19:27:40 +08:00
5dff936243 Fix HLL_UNION_AGG AnalyticFn result in BE core by adding hll_get_value (#2653) 2020-01-03 19:23:56 +08:00
c098178f7a [Index] Implements create drop show index syntax for bitmap index [#2487] (#2573)
### create table with index 
```
CREATE TABLE table1
(
    siteid INT DEFAULT '10',
    citycode SMALLINT,
    username VARCHAR(32) DEFAULT '',
    pv BIGINT SUM DEFAULT '0',
    INDEX index_name [USING BITMAP] (siteid, citycode) COMMENT 'balabala'
)
AGGREGATE KEY(siteid, citycode, username)
DISTRIBUTED BY HASH(siteid) BUCKETS 10
PROPERTIES("replication_num" = "1");
```
### create index 
```
CREATE INDEX index_name  ON table1 (siteid, citycod) [USING BITMAP] COMMENT 'balabala';
or 
ALTER TABLE table1 ADD  INDEX index_name  [USING BITMAP] (siteid, citycod) COMMENT 'balabala';
```
### drop index
```
DROP INDEX index_name ON table1;
or
ALTER TABLE table1 DROP INDEX index_name 
```

### show index
```
SHOW INDEX[ES] FROM table1
```
output
```
+---------+-------------+-----------------+------------+---------+
| Table   | Index_name  | Column_name     | Index_type | Comment |
+---------+-------------+-----------------+------------+---------+
| table1  | index_name  | siteid,citycode | BITMAMP    | balabala|
+---------+-------------+-----------------+------------+---------+
```
2020-01-03 17:41:26 +08:00
559a8d0514 Bump up apache arrow to 0.15.1 (#2646) 2020-01-03 13:15:29 +08:00
7951e15208 Fix estimate_segment_size problem #2643 (#2644) 2020-01-03 11:11:34 +08:00
3b5f608df7 Delete documents of count_distinct function (#2642) 2020-01-02 23:35:10 +08:00
4220e09724 Change introduction document (#2640)
change FE default maximum memory introduction document from 2G to 4G in english
2020-01-02 23:03:56 +08:00
b440ff286f Change introduction document (#2639)
Change FE default maximum memory introduction document  from 2G to 4G
2020-01-02 23:03:17 +08:00
9c90b09a3f [Alter Table] No need to check whether table is stable when doing some kinds of alter operation (#2617)
* [Alter Table] No need to check whether table is stable when doing some kinds of alter operation.

Not all alter table operation require table to be stable. Such as rename, modify meta data.
2020-01-02 20:51:23 +08:00
d05768ffd4 Fix core when es_scanner_node exit (#2634) 2020-01-02 16:30:11 +08:00
6cab929d6d [Compaction] Limit the max concurrency of running compaction tasks (#2635)
Compaction task may sometimes consume much memory and results in OOM.
And currently, there is no good way to predict the mem consumption of
a compaction task, so I add a new BE config: max_compaction_concurrency
to limit the max concurrency of running compaction tasks manually.
2020-01-02 14:47:54 +08:00
cc924c9e6a [Rowset Reader] Improve the merge read efficiency of alpha rowsets (#2632)
When merge reads from one rowset with multi overlapping segments, 
I introduce a priority queue(A Minimum heap data structure) for multipath merge sort, 
to replace the old N*M time complexity algorithm.

This can significantly improve the read efficiency when merging large number of 
overlapping data.

In mytest:
1. Compaction with 187 segments reduce time from 75 seconds to 42 seconds
2. Compaction with 3574 segments cost 43 seconds, and with old version, I kill the 
process after waiting more than 10 minutes...

This CL only change the reads of alpha rowset. Beta rowset will be changed in another CL.

ISSUE: #2631
2020-01-02 14:10:05 +08:00