Commit Graph

3421 Commits

Author SHA1 Message Date
df43752257 [Docs] Fix error KEY url (#6955) 2021-10-29 12:07:44 +08:00
b0926a317e Modify Chinese comment (#6951)
Modify Chinese comment
2021-10-28 13:56:59 +08:00
4170aabf83 [Optimize] optimize some session variable and profile (#6920)
1. optimize error message when using batch delete
2. rename session variable is_report_success to enable_profile
3. add table name to OlapScanner profile
2021-10-27 18:03:12 +08:00
a4a7e642b4 [Enhance] Add BackendHbResponse info (#6929)
when be has excepiton, fe doesn't log the BackendHbResponse info, so we can't know which be has exception

the exception log is:
`WARN (heartbeat mgr|31) [HeartbeatMgr.runAfterCatalogReady():141] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused)
`

so need add toString(), then fe can log the BackendHbResponse info
2021-10-27 09:56:07 +08:00
00fe9deaeb [Benchmark] Add star schema benchmark tools (#6925)
This CL mainly changes:

1. Add star schema benchmark tools in `tools/ssb-tools`, for user to easy load and test with SSB data set.
2. Disable the segment cache for some read scenario such as compaction and alter operation.(Fix #6924 )
3. Fix a bug that `max_segment_num_per_rowset` won't work(Fix #6926)
4. Enable `enable_batch_delete_by_default` by default.
2021-10-27 09:55:36 +08:00
Pxl
d4249e4f2d [Bug] fix Runtime filter can't find fragment-id when apply_filter called early (#6923)
#6921
2021-10-27 09:54:52 +08:00
77a954d02c [Bug] Fix treat tuple_is_null_predicate is const expr cause core problem (#6919)
Fix treat tuple_is_null_predicate is const expr cause core problem
2021-10-27 09:54:25 +08:00
29a4ff4bbe [Cache][Bug] Correct update cache timeout unit (#6888)
Now FE update cache use MICROSECONDS as TimeUnit.
Replace it by MILLISECONDS.
2021-10-27 09:53:58 +08:00
4f9b46d403 Fix String type column using zonemap to filter data maybe core dump (#6939)
Fix String type column using zonemap to filter data maybe core dump, because of not allocating memory before parsing string type zonemap
2021-10-27 09:25:38 +08:00
9d4e6d8362 [Spark-Doris-Connector] fixed some spark-doris-connector doc typo 2021-10-26 18:23:53 +08:00
adb6bfdf74 [Bug] Fix bug that truncate table may change the storage medium property (#6905) 2021-10-25 10:07:27 +08:00
ed7a873a44 [Memory Usage] Implement segment lru cache to save memory of BE (#6829) 2021-10-25 10:07:15 +08:00
2d298143cc [Bug] Fix bug of decommission (#6826) 2021-10-25 10:07:04 +08:00
ebb4c282b1 [Flink]Simplify the use of flink connector (#6892)
1. Simplify the use of flink connector like other stream sink by GenericDorisSinkFunction.
2. Add the use cases of flink connector.

## Use case
```
env.fromElements("{\"longitude\": \"116.405419\", \"city\": \"北京\", \"latitude\": \"39.916927\"}")
     .addSink(
          DorisSink.sink(
             DorisOptions.builder()
                   .setFenodes("FE_IP:8030")
                   .setTableIdentifier("db.table")
                   .setUsername("root")
                   .setPassword("").build()
                ));
```
2021-10-23 18:10:47 +08:00
469b05d708 [Cache][Bug] Fix sql_key of getting and updating Cache is inconsistent (#6903)
Fix #6735
2021-10-23 16:54:00 +08:00
090d99b690 [Docs] fix urls and format in routine load docs (#6896)
fix urls and format in routine load docs
2021-10-23 16:52:33 +08:00
88760d66d1 [MetaTool]add error message when loading meta by meta tool (#6893)
When loading meta by meta_tool goes wrong, we only get an error code from `json2pb`,
which is inconvenient for us to locate the problem.

This change is adding error message when loading meta goes wrong.

Log change is like below.

```
# before
./meta_tool --root_path=/home/disk1/qjl/mydoris/be/storage --operation=load_meta --json_meta_path=/home/disk1/qjl/data/meta-json.json
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1020 11:41:56.564241 74937 data_dir.cpp:837] path: /home/disk1/qjl/mydoris/be/storage total capacity: 7750843404288, available capacity: 7583325925376
I1020 11:41:56.564415 74937 data_dir.cpp:275] path: /home/disk1/qjl/mydoris/be/storage, hash: 7528840506668047470
load meta failed, status:-1410

# after 
./meta_tool --root_path=/home/disk1/qjl/mydoris/be/storage --operation=load_meta --json_meta_path=/home/disk1/qjl/data/meta-json.json
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1020 14:41:40.084342 50727 data_dir.cpp:837] path: /home/disk1/qjl/mydoris/be/storage total capacity: 7750843404288, available capacity: 7584601022464
I1020 14:41:40.084496 50727 data_dir.cpp:275] path: /home/disk1/qjl/mydoris/be/storage, hash: 7528840506668047470
E1020 14:41:40.163007 50727 tablet_meta_manager.cpp:161] JSON to protobuf message failed: Fail to decode base64 string=0
load meta failed, status:-1410
```
2021-10-23 16:51:58 +08:00
7b50409ada [Bug][Binlog] Fix the number of versions may exceed the limit during data synchronization (#6889)
Bug detail: #6887 

To solve this problem, the commit of transaction must meet any of the following conditions to avoid commit too freqently:

1. The current accumulated event quantity is greater than the `min_sync_commit_size`.
2. The current accumulated data size is greater than the `min_bytes_sync_commit`.

In addition, when the accumulated data size exceeds `max_bytes_sync_commit`, the transaction needs to be committed immediately.

Before:
![a5e0a2ba01ec4935144253fe0a364af7](https://user-images.githubusercontent.com/22125576/137933545-77018e89-fa2e-4d45-ae5d-84638cc0506a.png)

After:
![4577ec53afa47452c847bd01fa7db56c](https://user-images.githubusercontent.com/22125576/137933592-146bef90-1346-47e4-996e-4f30a25d73bc.png)
2021-10-23 16:47:32 +08:00
Wei
696790b397 [Refactor] remove unused code (#6879) 2021-10-23 16:47:10 +08:00
a8e3a74ac6 [Bug] Fix bug to reject request with no SQL in TableQueryPlanAction (#6843)
String.valueOf() returns string "null" with null input, in which case requests with no SQL
will be accepted by TableQueryPlanAction unexpectedly with potential risk.
2021-10-23 16:46:24 +08:00
6029082c2a [Flink][Bug] Fix potential NPE when cancel DorisSourceFunction (#6838)
Fix potential NPE of `scalaValueReader` when cancelling DorisSourceFunction.
2021-10-23 16:45:24 +08:00
149ce9ecf4 [Bug][Memory Leak] Fix the issue of Catalog instance leakage (#6895)
The Checkpoint Catalog instance may be incorrectly stored in MetricRepo, causing memory leaks
2021-10-23 16:44:51 +08:00
3267455eca Replace replica_allocation to replication_allocation (#6870)
Fix #6869
2021-10-20 15:32:35 +08:00
51e210869a [ARM64] Fix some problem when compiling on ARM64 platform (#6836) (#6872)
With thirdparties 1.4.0 to 1.4.1

1. Add patch for aws-c-cal-0.4.5
2. Add some solutions for `undefined reference libpsl`
3. Move libgsasl to fix link problme of libcurl.
4. Downgrade openssl to 1.0.2k to fix problem of low version glibc
2021-10-19 13:26:02 +08:00
bd25d1a828 [Doc] Add documents for MySQL Binlog Load (#6859)
* add zh-CN docs

* add en docs and image

* fix

* fix
2021-10-19 10:25:42 +08:00
e96882f6c5 Update materialized_view.md (#6867) 2021-10-19 10:24:38 +08:00
fbd75c88d0 [Docs] Fix exporter document error (#6864)
* fix exporter document error

* update en doc
2021-10-19 10:24:08 +08:00
63dbcbc4e1 [UT] Fix ut bugs (#6862)
Co-authored-by: morningman <chenmingyu@baidu.com>
2021-10-18 10:12:55 +08:00
da99749e7f [Bug] Fix bug that BE will crash when backup using S3 (#6855) 2021-10-17 22:54:42 +08:00
bb2b29c64f [Doc] Add type BOOLEAN when enter 'help create table' in mysql client (#6852)
some user do not know Doris support type boolean, they use TINYINT,
so i add type BOOLEAN when enter 'help create table' in mysql client.

currently, type BOOLEAN size is 1 byte, but the value of boolean column only in {0,1} ,
which waste some memory, and i want change it's implement to 1 bit in the future.
2021-10-17 22:54:12 +08:00
bc069eac8b [BUG] fix bug for schema schange (#6839)
This commit has an error:
#6791
when you only change the order of column, error will apear.
2021-10-17 22:53:28 +08:00
eff076b355 [BUG] Fix printing ReservationTrackerCounters cause BE crash when mem_limit is reached (#6849)
When the memory usage of BE reaches mem_limit, printing ReservationTrackerCounters through MemTracker
may cause BE crash in high concurrency.

ReservationTrackerCounters is not actually used in the current Doris, and the memory tracker in Doris
will be redesigned in the future.
2021-10-16 21:57:09 +08:00
59017cebe6 [ARM64] Fix some problem when compiling on ARM64 platform (#6836)
1. Refactor the create method of hdfs reader & writer.

    libhdfs3 does not support arm64. So we should not support hdfs reader & writer on arm64.

2. And micro for LowerUpperImpl
2021-10-16 21:56:49 +08:00
4cc01892f6 [SQL Cache] Add all view stmt as the suffix of cache sqlkey (#6832)
Use all view stmt as the cache sqlkey suffix, so that when the view is modified, the cache can recognize.
2021-10-16 21:56:24 +08:00
a0b3840daa [MemerySave] Change TabletSchema in tablet to reference to save mem (#6814)
Change TabletSchema in tablet to reference to save memory
2021-10-16 21:54:32 +08:00
8baded8a0e [BUG] Key is 'True',Extra is 'NONE' when add rollup for DUP table (#6763)
fix #6762

The result is displayed after the repair

```
mysql> desc test2 all;
+-----------+---------------+-------+---------------+------+-------+---------+-------+---------+
| IndexName | IndexKeysType | Field | Type          | Null | Key   | Default | Extra | Visible |
+-----------+---------------+-------+---------------+------+-------+---------+-------+---------+
| test2     | DUP_KEYS      | a     | BIGINT        | Yes  | true  | NULL    |       | true    |
|           |               | b     | BIGINT        | Yes  | true  | NULL    |       | true    |
|           |               | c     | BIGINT        | Yes  | false | NULL    | NONE  | true    |
|           |               | d     | BIGINT        | Yes  | false | NULL    | NONE  | true    |
|           |               | e     | VARCHAR(1024) | Yes  | false | NULL    | NONE  | true    |
|           |               | f     | VARCHAR(1024) | Yes  | false | NULL    | NONE  | true    |
|           |               |       |               |      |       |         |       |         |
| r1        | DUP_KEYS      | c     | BIGINT        | Yes  | true  | NULL    |       | true    |
|           |               | e     | VARCHAR(1024) | Yes  | true  | NULL    |       | true    |
|           |               | a     | BIGINT        | Yes  | false | NULL    | NONE  | true    |
+-----------+---------------+-------+---------------+------+-------+---------+-------+---------+

```
2021-10-16 21:54:00 +08:00
607eef8d4d [Doc] Update compile docs add 0.15 build support. (#6850) 2021-10-15 18:37:24 +08:00
24d38614a0 [Dependency] Upgrade thirdparty libs (#6766)
Upgrade the following dependecies:

libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0

after this pr compile doris should use build-env:1.4.0
2021-10-15 13:03:04 +08:00
adb9b0d9c6 [Bug] Return 0 when hex(0) (#6837) 2021-10-15 10:18:55 +08:00
fcd15edbf9 [Export] Support export job with label (#6835)
```
EXPORT TABLE xxx
...
PROPERTIES
(
    "label" = "mylabel",
    ...
);
```

And than user can use label to get the info by SHOW EXPORT stmt:
```
show export from db where label="mylabel";
```

For compatibility, if not specified, a random label will be used. And for history jobs, the label will be "export_job_id";

Not like LOAD stmt, here we specify label in `properties` because this will not cause grammatical conflicts,
and there is no need to modify the meta version of the metadata.
2021-10-15 10:18:11 +08:00
58440b90f0 [Bug] Left() string function behaves not identically to the mysql implementation (#6811)
See Fix #6810
2021-10-15 10:17:21 +08:00
05e59f6487 [Improvement] fix typo (#6831) 2021-10-14 14:22:53 +08:00
Wei
80c4007a34 [Refactor] Refactor ConnectProcessor's code doc and unused code (#6823)
Co-authored-by: wei.zhang2 <wei.zhang2@dmall.com>
2021-10-13 11:44:29 +08:00
3ff63fef99 [Bug] Select outfile failed after query stmt being rewritten. (#6816) 2021-10-13 11:44:01 +08:00
5d3ebad836 modify comments of DropPartitionClause (#6812)
Co-authored-by: qzsee <shizhiqiang03@meituan.com>
2021-10-13 11:40:19 +08:00
cfeb515f5e [Enhancement] add spark load config spark_load_checker_interval_second (#6809)
the internal seconds for spark load to check and update etl job status, 60 seconds for default

Co-authored-by: weixiang <weixiang06@meituan.com>
2021-10-13 11:39:51 +08:00
5ef3f59928 [Optimize][RoutineLoad] Avoid sending tasks if there is no data to be consumed (#6805)
1 Avoid sending tasks if there is no data to be consumed
By fetching latest offset of partition before sending tasks.(Fix [Optimize] Avoid too many abort task in routine load job #6803 )

2 Add a preCheckNeedSchedule phase in update() of routine load.
To avoid taking write lock of job for long time when getting all kafka partitions from kafka server.

3 Upgrade librdkafka's version to 1.7.0 to fix a bug of "Local: Unknown partition"
See offsetsForTimes fails with 'Local: Unknown partition' edenhill/librdkafka#3295

4 Avoid unnecessary storage migration task if there is no that storage medium on BE.
Fix [Bug] Too many unnecessary storage migration tasks #6804
2021-10-13 11:39:01 +08:00
Wei
e547e77f86 Fix variable dbName that is never used (#6802)
Co-authored-by: wei.zhang2 <wei.zhang2@dmall.com>
2021-10-13 11:37:58 +08:00
32f6dec80f fix dup table don't schema schange (#6791)
Co-authored-by: qzsee <shizhiqiang03@meituan.com>
2021-10-13 11:37:39 +08:00
6a058792af [Feature][Step1] Support lateral view FE part (#6745)
* [Feature] Support lateral view

The syntax:
```
select k1, e1 from test lateral view explode_split(k1, ",") tmp as e1;
```
```explode_split``` is a special function of doris,
which is used to separate the string column according to the specified split string,
and then convert the row to column.
This is a conforming function of string separation + table function,
and its behavior is equivalent to explode in hive ```explode(split(string, string))```

The implement:
A tablefunction operator is added to the implementation to handle the syntax of the lateral view separately.
The query plan is following:
```
MySQL [test]> explain select k1, e1 from test_explode lateral view explode_split (k2, ",") tmp as e1;
+---------------------------------------------------------------------------+
| Explain String                                                            |
+---------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                           |
|  OUTPUT EXPRS:`k1` | `e1`                                                 |
|                                                                           |
|   RESULT SINK                                                             |
|                                                                           |
|   1:TABLE FUNCTION NODE                                                   |
|   |  table function: explode_split(`k2`, ',')                             |
|   |                                                                       |
|   0:OlapScanNode                                                          |
|      TABLE: test_explode                                                  |
+---------------------------------------------------------------------------+
```

* Add ut

* Add multi table function node

* Add session variables 'enable_lateral_view'

* Fix ut
2021-10-13 11:37:12 +08:00