Commit Graph

3726 Commits

Author SHA1 Message Date
a6ff1bd79e Flink / Spark connector compilation problem (#7725)
Flink / Spark connector compilation problem
2022-01-14 22:14:48 +08:00
e7d65e488c [style] translate code annotations into english (#7752)
Translate Chinese code comments into English,the following files has been modified:
1. be/src/olap/row_cursor.h
2. be/src/olap/compress.h
2022-01-14 09:37:46 +08:00
5c4055ac3a [style] Translate Chinese to English in be_olap_field.h (#7738) 2022-01-14 09:36:58 +08:00
fe80d1417f [style] replace Chinese comments with English comments (#7732) 2022-01-14 09:35:06 +08:00
f3817829bb [fix] fix malloc and free mismatch issue (#7702)
The memory allocate by `malloc` should be freed by `free`
2022-01-14 09:32:33 +08:00
6188ab20df [docs](faq) add multiple FE WEB UI login issues (#7654) 2022-01-14 09:26:39 +08:00
902ab93043 [fix](session-variable) fix bug that checkpoint may overwrite the global variables (#7526)
We should create temporary object for some static fields when doing checkpoint,
to avoid there variables to be overwritten by the checkpoint process.
2022-01-14 09:25:10 +08:00
d03151bda2 [chore](be) Add -Werror (#7744)
All Warning will be treated as Error when compiling BE
2022-01-14 09:21:57 +08:00
10709f315a [fix](github-action) fix the action of set-label-based-on-pr-title (#7758) 2022-01-14 09:20:42 +08:00
3da4425af5 [fix](github-action) fix the action of set-label-based-on-pr-title (#7757) 2022-01-13 23:35:00 +08:00
d1a994eff9 [fix](cpu-resource)(resource-tag) Allow set cpu_resource_limit to -1 and fix resource tag bug(#6830)
1. Allow set cpu_resource_limit

    -1 means unlimited

2. Drop replica not in valid tag

    Otherwise, the migration task from a resource group to another may never finish.
2022-01-13 23:11:37 +08:00
b51121fe86 [chore](github-action) Add label auto for pull requests (#7663) 2022-01-13 20:07:16 +08:00
ccb6c6ac2e [docs] update seatunnel.md (#7731)
correct some wrongly written words and update document format
2022-01-13 15:31:17 +08:00
5e1caea2b1 [fix](lateral-view) Fix some bugs about lateral view (#7721)
1.  fix core dump when using multi explode_bitmap #7716 
2. fix bug that json array extract by json path is wrong #7717 
3. fix bug that after lateral view, the null value become non-null value #7718 
4. fix bug that lateral view may return error: couldn't resolve slot descriptor 1. #7719 
5. fix error result when using lateral view with where predicate #7720
2022-01-13 15:30:38 +08:00
8ac32041e4 [fix](show) fix ConcurrentModificationException for show proc '/current_queries' (#7707) 2022-01-13 15:28:19 +08:00
db2649525f [docs](website) Add Database ODBC version correspondence (#7675) 2022-01-13 15:28:02 +08:00
a034c20d16 [fix](website) Add trademarks footer on official website (#7696) 2022-01-11 15:07:56 +08:00
2de79832fc [docs](hive)(function) fix Hive type error and optimize alias function example (#7694)
1. fix Hive type error 
2. optimize alias function example
2022-01-11 15:07:32 +08:00
1b2acb6acd [docs] update the document format (#7689) 2022-01-11 15:06:42 +08:00
2cf574dc01 [docs] Improve instructions for the configuration of BE. (#7620) 2022-01-11 15:06:05 +08:00
8685b6b985 [improvement](executor) Optimize lock of client cache (#7543) 2022-01-11 15:05:24 +08:00
6864a376ca [improvement](spark-connector) Throw an exception when the data push fails and there are too many retries (#7531) 2022-01-11 15:03:06 +08:00
d4188877f1 [comminity](github) Polish PR template (#7638)
Improving PR templates.
2022-01-11 15:01:50 +08:00
4ac8b3c9a9 [fix][s3] Fix bug that can not visit aliyun oss with aws s3 sdk (#7691)
Close #7690

1. Exclude httpclient and httpcore dependencies from thrift@0.13

    Explicitly use httpclient@4.5.13 and httpcore@4.4.15
    https://stackoverflow.com/questions/59265959/java-lang-bootstrapmethoderror-call-site-initialization-exception-from-athena-j

2. Exclude aws-java-sdk-s3 dependency from hadoop-aws

    Explicitly use aws-java-sdk-s3@1.11.95
    https://github.com/aws/aws-sdk-java/issues/1032
2022-01-11 15:00:31 +08:00
83f6eef506 [improvement](routine-load) Make routine load work with old kafka version (#7630)
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2022-01-10 17:30:24 +08:00
68c87de69e [fix](website) fix CaseList component bug (#7683) 2022-01-10 14:46:05 +08:00
ff4284f3fa [feature](hint)(mysql-compatibility) Support general hints in select statement (#7664)
Support general hints.

Sql example:

```sql
SELECT /*+ one_hint(1000000) another_hint(k = "v")*/ 1;
```

hints syntax is:

```
/*+ [ HINT_NAME( [ key [ =value ]? ]* ) ]+ */
```

- support multi hints, sep with space
- hint name could be any string in identifier format
- hint could have zero or more parameters, sep with comma
- hint parameter must have one key
- hint parameter could have zero or one value
- hint parameter‘s key and value connected by equal sign
2022-01-09 16:59:08 +08:00
7254bcc8ca [refactor](spark-connector) delete useless maven dependencies and some code variable definition issues (#7655) 2022-01-09 16:58:16 +08:00
15d54bae0e [fix](error-hub) use lock to protect the creation of error hub (#7605)
Add a lock when creating error_hub to ensure that no multiple threads create error_hub
(which could lead to a CORE) #7604
2022-01-09 16:57:31 +08:00
9aaa3f63f7 [improvement](spark-connector) Stream load http exception handling (#7514)
Stream load http exception handling
2022-01-09 16:54:55 +08:00
3a8a85b739 [Optimize][Extension] optimize extension datax doriswriter,Remove import doris via csv in Dataxwriter, only support via json (#7568)
* 1.Remove import doris via csv in Dataxwriter, only support via json;
2.Format Dataxwriter code;
3.Optimize exception handling and reduce multiple output of exception logs;
4.Update the dataxwriter's documentation;

* Delete DorisCsvCodec.java

delete unused file extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisCsvCodec.java

* 1.remove `format` config key;
2.Optimize serialization code in DorisJsonCodec class
2022-01-09 13:27:52 +08:00
ad35067a2a [chore][docs] add deploy spark/flink connectors to maven release repo docs (#7616) 2022-01-06 23:23:33 +08:00
482bf05da7 [refactor](log) remove RewriteClasses unused LOG reference (#7609) 2022-01-06 23:22:09 +08:00
1e0e472784 [fix](audit-plugin) Fix audit load plugin may stopped when throw unexpected exceptions (#7607)
Fix audit load may stopped when throw unexpected exceptions
2022-01-06 23:21:13 +08:00
90aa6c8a72 [fix](syntax) Add STRUCT to keywords (#7606) 2022-01-06 23:20:20 +08:00
1f88c5f849 [improvement](git) add vscode devcontainer config into git ignore (#7602)
When I use dev container feature in vscode, there is some config file that shouldn't be put in git.
So it's better to add config file into gitignore for convenience.
2022-01-06 23:19:50 +08:00
831f4cd71e [improvement](website)(proc) Make web page base proc dir and variables orderly (#7535) 2022-01-06 23:16:50 +08:00
563545475e [Optimize](Runtime Filter) Support merge in runtime filter(#7546) (#7547)
Support merge IN predicate when exist remote target(e.g. shuffle hash join).
Remote the code that IN predicate implicit conversion to Bloom filter then exist  remote target.

Close related #7546
2022-01-06 19:08:35 +08:00
e1374d8536 [fix](tablet-scheduler) Fix decommission backend bug (#7563)
Fix bug that decommission backend operation blocked with error:
`no proper tag is chose for tablet.`
2022-01-06 00:08:06 +08:00
2a2f12ca51 [refactor & fix](exce & olap) refactor reader: rename Reader to TabletReader (#7544)
1. Consider the responsibility of Reader,  Rename Reader to TabletReader, I think the new name TabletReader can represent its function exactly,  it is more suitable and meaningful
2. add virtual keyword for the destructor of OlapScanner, because VOlapScanner is derived from it
3. refactor struct ReaderParams and KeysParam as TabletReader's inner struct,guard by TabletReader name scope, it's also more reasonable
4. reduce OlapScanner's member data amount, just use _parent->member_data is simpler
5. bugfix: TupleReader has the same memeber data _collect_iter to its parent class Reader, this usage is dangerous, the writer may make some mistake, so i delete TupleReader::_collect_iter to fix it.
6. call set_tablet_reader() in OlapScanner::prepare() to setup _tablet_reader, VOlapScanner should override set_tablet_reader to new BlockReader instead,  use this way to avoid new Reader twice by reset unique_ptr _tablet_reader
7. if the member data is a inseparable part of a class, i suggest using normal variable while not pointer variable, because pointer bring a indirect lay and must handle coping and destructing carefully, it's not necessary
8. some other small changes for readability or design
2022-01-06 00:00:32 +08:00
738d2d2e07 [refactor] update parent pom version and optimize build scripts (#7548) 2022-01-05 10:45:11 +08:00
9ddcf0625c [improvement](load) Transaction for load job with no data for all partitions should be considered as normal and should not be aborted (#7240)
If the load result set is empty, or the load data is all filtered by the `where` condition,
it will not return failed with msg `all partitions have no load data`, but will return success directly.
2022-01-05 10:38:33 +08:00
5c104ec2d1 [Improvement] use "storage_cooldown_seconds" property when storage medium is SSD (#7532)
Refer to this issue #7528

When setting property `default_storage_medium=ssd` and `storage_cooldown_second=xxx` in `fe.conf`
`cooldownTime=System.currentTimeMillis()+ storage_cooldown_second` , not always `MAX_COOLDOWN_TIME_MS`
2022-01-04 10:32:57 +08:00
bf4a867e85 [improvement](tablet-repair) add a config repair_slow_replica (#7423)
Add a new FE config `repair_slow_replica`    
when this config is true, Doris will try to delete the replica
with the largest number of versions, and then rebalance the replica.
Usually, when the number of versions of a certain replica is much higher
then that of other replicas, there are some problems with the current be's compilation.
Migrating to other machines can typically solve this problem.
2022-01-04 10:28:14 +08:00
6657524c51 [feature](sql-block-rule) add partition_num, tablet_num, cardinality in SqlBlockRule to block big/slow sql (#7403)
Add partitionNum, tabletNum, cardinality in SqlBlockRule to block large/slow sql.

1. set partitionNum, tabletNum, cardinality as limitations to block sqls
2. compatible with lower version
3. add unit tests
4. add docs
2022-01-04 09:59:41 +08:00
7b13ac5b31 [deps][chore] make openssl works with old glibc version (#7541)
1. build OpenSSL with --with-rand-seed=devrandom
2. Modified: brpc 1.0.0-rc02 -> 1.0.0
2021-12-31 23:19:04 +08:00
a60d86c1e1 [improvement](broker) add disable cache config for broker (#7506) 2021-12-31 16:48:55 +08:00
d457ab3122 [imporvement] remove unused method from AggregateFunction (#7496) 2021-12-31 16:35:23 +08:00
d6cc3fdf03 [fix](materialized-view) forbidden create materialized view with distinct (#7494) 2021-12-31 16:08:37 +08:00
46ca012e2b [fix](bloom-filter) Fix error when handle empty string in bloom filter (#7448) 2021-12-31 16:05:33 +08:00