Commit Graph

70 Commits

Author SHA1 Message Date
0b1b3e225d Revert "[Thirdparty]Add llvm for codegen (#8938)" (#8948)
This reverts commit 32133621c69a2d7544549c5ea54ed6d9de60415e.

Reverts #8938
The LLVM requires GLIBC_2.15. I decided to create a branch for the llvm feature first.
And once we resolve the low version glibc issue, it will be merged back to master.
2022-04-11 13:03:16 +08:00
32133621c6 [Thirdparty]Add llvm for codegen (#8938) 2022-04-10 20:37:09 +08:00
989e03ddf9 [improvement] Improve sig handler (#8545)
* Refactor glog's default signal handler

Co-authored-by: Zhengguo Yang <780531911@qq.com>
2022-03-22 10:40:31 +08:00
848acec584 [chore](dependency) update Croaring for good performance (#8492)
update Croaring for good performance, according to RoaringBitmap/CRoaring#320
2022-03-17 10:07:55 +08:00
f3c44bcd75 [chore][fix](librdkafka) disable librdkafka assert and update some thirdparty (#8425)
1. comment  librdkafka `rd_assert(thrd_is_current(rkb->rkb_thread));` to avoid core dump
2. upgrade arrow to 7.0.0
3. upgrade aws sdk to 1.9
4. upgrade orc to 1.7.2
2022-03-12 22:09:06 +08:00
50a59f3f86 [license] Organize third-party dependent licenses for bianry releases (#8350) 2022-03-07 23:18:58 +08:00
789472a87d [build] change docker image add perf tools and simdjson (#8012) 2022-02-11 18:54:09 +08:00
f2cbf0a8d2 [chore] Improve the ldb toolchain compilation documentation (#7829)
Add document for compiling Doris with ldb toolchain
2022-01-21 21:36:43 +08:00
800a36343a [chore] Prolog of hermetic build with GCC 11 and Clang 13. (#7712)
Prepare to generate hermetic build using GCC 11 and Clang 13.
The ideal toolchain would be ldb toolchain generated by [ldb_toolchain_gen.sh](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.3/ldb_toolchain_gen.sh)

To kick off a clang build, set `DORIS_TOOLCHAIN=clang` before running any build scripts.
2022-01-21 12:12:04 +08:00
0efef1b332 [fix](schema-change) Fix bug that schema change may return -102 error (#7808)
When using linked schema change, we need to check if all rowsets are of the same type,
ALPHA or BETA. otherwise, we need to use direct schema change to convert the data.
2022-01-21 10:59:54 +08:00
7b13ac5b31 [deps][chore] make openssl works with old glibc version (#7541)
1. build OpenSSL with --with-rand-seed=devrandom
2. Modified: brpc 1.0.0-rc02 -> 1.0.0
2021-12-31 23:19:04 +08:00
2d72c039ad [deps](openssl) upgrade openssl to 1.1.1m (#7446)
upgrade openssl to 1.1.1m, ready for support SM2 / SM3 / SM4 national secret (national commercial password) algorithm
2021-12-21 10:09:36 +08:00
382351b0ee [fix](ut) Fix run fe ut failed, be ut memory leak and build thirdparty failed (#7377) 2021-12-15 11:00:20 +08:00
00e653e812 [deps] add breakpad for minidump (#7128)
Related #7123
2021-11-18 14:34:38 +08:00
6c6380969b [refactor] replace boost smart ptr with stl (#6856)
1. replace all boost::shared_ptr to std::shared_ptr
2. replace all boost::scopted_ptr to std::unique_ptr
3. replace all boost::scoped_array to std::unique<T[]>
4. replace all boost:thread to std::thread
2021-11-17 10:18:35 +08:00
aba7d2ccae [Thirdparty] Fix flatbuffers download url error (#6968)
Change google flatbuffers download URL in thirdparty.var.sh
2021-11-02 16:34:17 +08:00
51e210869a [ARM64] Fix some problem when compiling on ARM64 platform (#6836) (#6872)
With thirdparties 1.4.0 to 1.4.1

1. Add patch for aws-c-cal-0.4.5
2. Add some solutions for `undefined reference libpsl`
3. Move libgsasl to fix link problme of libcurl.
4. Downgrade openssl to 1.0.2k to fix problem of low version glibc
2021-10-19 13:26:02 +08:00
24d38614a0 [Dependency] Upgrade thirdparty libs (#6766)
Upgrade the following dependecies:

libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0

after this pr compile doris should use build-env:1.4.0
2021-10-15 13:03:04 +08:00
5ef3f59928 [Optimize][RoutineLoad] Avoid sending tasks if there is no data to be consumed (#6805)
1 Avoid sending tasks if there is no data to be consumed
By fetching latest offset of partition before sending tasks.(Fix [Optimize] Avoid too many abort task in routine load job #6803 )

2 Add a preCheckNeedSchedule phase in update() of routine load.
To avoid taking write lock of job for long time when getting all kafka partitions from kafka server.

3 Upgrade librdkafka's version to 1.7.0 to fix a bug of "Local: Unknown partition"
See offsetsForTimes fails with 'Local: Unknown partition' edenhill/librdkafka#3295

4 Avoid unnecessary storage migration task if there is no that storage medium on BE.
Fix [Bug] Too many unnecessary storage migration tasks #6804
2021-10-13 11:39:01 +08:00
83003cc372 [Thirdparty] Change libhdfs3 download url to a stable one(#6744) 2021-10-03 11:56:36 +08:00
Pxl
4dd610c28d [Feature] Support for storage layer benchmark (#6506)
* add benchmark tool
2021-09-02 09:57:19 +08:00
abbc9202af support routine load isolation_level read_committed (#6191)
Co-authored-by: Geoffrey <gaofeng01@rd.netease.com>
2021-08-30 17:22:08 +08:00
9216735cfa [New Featrue] Support Vectorization Execution Engine Interface For Doris (#6329)
1. FE vectorized plan code
2. Function register vec function
3. Diff function nullable type
4. New thirdparty code and new thrift struct
2021-08-11 14:54:06 +08:00
de2dd2c015 [Thirdparty] Move DataTables download url to a fixed download address (#6189)
The origin download url is always changing: https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.25
So we put it in our own http server.
If someone can offer an official url for DataTables, please update this.
2021-07-10 10:13:26 +08:00
48bd680068 update download info for boost and datatables (#6008) 2021-06-18 11:38:41 +08:00
80220af271 [Enhancement] Use Parallel Hash Map Replace Unordered Map In Dict Encodeing Map And Hyper Set (#5990)
Use Parallel Hash Map Replace Unordered Map In Dict Encodeing Map And Hyper Set To Improve Ferformance
2021-06-10 17:38:08 +08:00
109b55ee5f [Shell] Add build parallel option (#5819)
Add build parallel option then we can build project with a user specified parallel not a fixed value.
2021-05-19 09:32:58 +08:00
29a3fa1084 [Feature] Support read data with format of parquet from hdfs, using libhdfs3 (#5686)
Add new lib, Backend can read data from hdfs without broker,
this patch include libhdfs3.a which can read file on hdfs.
This patch will make reading the data from hdfs with parquet possible.
By this, we will support more format of file on hdfs in the future,
and we will support other metadata in the future.
2021-04-24 17:41:48 +08:00
7445051174 [Refactor] fix warning in gcc8+, update rapidjson (#5649) 2021-04-20 09:14:44 +08:00
2b5e4dc951 Add fmt library to speed up mysql text result serialization (#5554)
* Add fmt library to speed up mysql text result serialization

* use BUILD_SYSTEM instead of make

Co-authored-by: gaodayue <gaodayue@bytedance.com>
2021-04-08 09:16:58 +08:00
08bfd1c762 [Bug] Fix some problem with downloading and building thirdparty (#5439) 2021-03-30 10:47:45 +08:00
ad67dd34a0 update gcc to gcc 10 and support c++17 (#5394)
* update gcc to gcc 10 and support c++17
    update brpc to 0.9.7
    update boost to 1.73
    remove third-party boost 1.54 for mysql

* update cmake version

* ignore jdk version

* remove unused patch

* avoid use SYS_getrandom call
2021-03-25 09:30:38 +08:00
3a6476b37b add as sdk to thirdparty (#5234) 2021-01-20 10:51:22 +08:00
58e58c94d8 [TSAN] Fix tsan bugs (part 1) (#5162)
ThreadSanitizer, aka TSAN, is a useful tool to detect multi-thread
problems, such as data race, mutex problems, etc.
We should detect TSAN problems for Doris BE, both unit tests and
server should pass through TSAN mode, to make Doris more robustness.
This is the very beginning patch to fix TSAN problems, and some
difficult problems are suppressed in file 'tsan_suppressions', you
can suppress these problems by setting:
export TSAN_OPTIONS="suppressions=tsan_suppressions"

before running:
`BUILD_TYPE=tsan ./run-be-ut.sh --run`
2021-01-15 09:45:11 +08:00
bebbc27a83 [Thirdparty] Fix the DataTables.zip download issue (#5128)
Modify the third-party library DataTables.zip download url from dt-1.10.22 to dt-1.10.23,
resolved the download failure issue.
2020-12-31 09:47:43 +08:00
ed09ea9cf7 [Bug] Fix bug that tablet report always out of date (#4695)
This is because the report version is not set correctly in the tablet report request.

Also modify the download url of libevent Fix #4689 Fix #4691
2020-10-09 15:54:31 +08:00
04e2ce2acb [Compile] Fix md5 of thirdparty(#4663)
Fix compile error when compile Doris from source and not using docker
2020-09-24 19:00:58 +08:00
67b842ce04 [License] Organize and modify the license of the code (#4371)
1. Disable the MySQL client and LZO library by default when building the Doris.

    MySQL client library is used for MySQL external table feature.
    This feature will be replaced by the new ODBC external table soon.

    LZO library is used to compress/decompress data of some old data format of Doris,
    which is no longer used anymore.

2. Add missing license to some files.

3. For all non-Apache-License code, all are explained in NOTICE file and the corresponding license is declared.

4. Remove the js source code from webroot, it will be downloaded as thirdparty
2020-08-24 21:51:55 +08:00
a8fe54b7b9 [ODBC SCAN NODE] 1/4 Add unix odbc library. (#4377) 2020-08-21 21:26:14 +08:00
6be7a6232f [Config] Add ignore config to determine whether to continue to start be when load tablet from header failed. (#3632)
Add config ignore_load_tablet_failure to determine whether to continue to start be when load tablet from header failed.
2020-05-20 09:40:50 +08:00
a6c0d376dd [Thirdparty] Update ORC lib download address (#3440) 2020-04-30 14:16:44 +08:00
3dc7ef634b [Dependency]Add cctz lib (#3280)
Add Google/CCTZ lib in Doris
2020-04-09 19:14:09 +08:00
f37dbbc890 Fix openssl download url is not avaliable (#3281) 2020-04-08 19:00:48 +08:00
fdbc0f7cca Change replace bzip and boost sources (#2878) (#2880)
replace fast sources for download bzip and boost when build
2020-02-11 12:08:47 +08:00
369b2c364f [Build] fix jemalloc download (#2681) 2020-01-07 11:33:55 +08:00
1ca82122a8 Fix doris be compile error for Ubuntu14.04 (#2647) 2020-01-03 21:14:34 +08:00
559a8d0514 Bump up apache arrow to 0.15.1 (#2646) 2020-01-03 13:15:29 +08:00
45fa9c999e Add Apache ORC lib in Doris (#2479) 2019-12-19 11:09:49 +08:00
05c05cfc83 Add apache arrow IPC module for Spark-Doris-Connector usage (#1958)
Add FLATBUFFERS to TP archive
2019-10-18 14:27:19 +08:00
011bdcd641 Bump thirdparty's BZIP2 version to 1.0.8 (#1559) 2019-07-29 00:10:44 +08:00