14d21edf65
[fix] croaringbitmap compile support USE_AVX2=0 ( #10140 )
...
* If we disable AVX2 by config USE_AVX2=0, we need to croaringbitmap with ROARING_DISABLE_AVX=ON
* update to trigger regression test again
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-06-16 18:17:46 +08:00
3f9436c6a8
[compile]fix simdjson compile flags ( #10054 )
2022-06-16 11:28:51 +08:00
39a2785ce2
[enhancement] support simd instructions on arm cpus through sse2neon ( #10068 )
...
* [enhancement] support simd instructions on arm cpus through sse2neon
2022-06-14 09:17:09 +08:00
7cf0cc7dd6
[deps] update libhdfs3 to fix a uuid set problem ( #10092 )
2022-06-14 09:16:32 +08:00
e0cf2677a0
[dependency][enhancement] support build libhdfs in arm cpus ( #10018 )
...
Supports native hdfs functionality on arm cpu
This pr mainly upgrades libdfs3 and supports running on arm，and make libhdfs3 with kerberos as default
2022-06-10 19:40:41 +08:00
a34d4b55f9
[dependency] Add opentelemetry related third-party library. ( #9990 )
2022-06-10 09:59:07 +08:00
73a3093539
[dependency] add hyperscan and its dependency ragel to thirdparty ( #9964 )
2022-06-09 23:36:09 +08:00
c0ad1be1bd
[Enhancement][Chore] remove breakpad and unused variable ( #9937 )
2022-06-02 20:52:17 +08:00
9f26c6a34f
[chore] fix some compatible on old linux distributions ( #9815 )
2022-06-02 10:37:10 +08:00
635b5630e3
[fix](third-party) Fix build issues of third-party in ubuntu docker ( #9805 )
...
Fixes:
1. Fully static linkage of protoc may lead to core dump when using it.
2. Building libmyclient may fail due to miss-used option `-v` of `make`
3. Building lzma implicitly requires autopoint
4. Building xml2 implicitly requires pkg-config
2022-05-29 17:33:59 +08:00
cbbda7857b
[feature-wip](parquet-orc) Support orc scanner in vectorized engine ( #9541 )
2022-05-26 21:39:12 +08:00
ddda91c89d
[doc] Update dev image ( #9721 )
2022-05-23 11:59:15 +08:00
b3a2a92bf5
[deps] libhdfs3 build enable kerberos support ( #9524 )
...
Currently, the libhdfs3 library integrated by doris BE does not support accessing the cluster with kerberos authentication
enabled, and found that kerberos-related dependencies（gsasl and krb5） were not added when build libhdfs3.
so, this pr will enable kerberos support and rebuild libhdfs3 with dependencies gsasl and krb5:
- gsasl version: 1.8.0
- krb5 version: 1.19
2022-05-22 20:58:19 +08:00
defdae1e7d
[improvement](stream-load) adjust read unit of http to optimize stream load ( #9154 )
2022-05-20 09:52:36 +08:00
0b1b3e225d
Revert "[Thirdparty]Add llvm for codegen ( #8938 )" ( #8948 )
...
This reverts commit 32133621c69a2d7544549c5ea54ed6d9de60415e.
Reverts #8938
The LLVM requires GLIBC_2.15. I decided to create a branch for the llvm feature first.
And once we resolve the low version glibc issue, it will be merged back to master.
2022-04-11 13:03:16 +08:00
32133621c6
[Thirdparty]Add llvm for codegen ( #8938 )
2022-04-10 20:37:09 +08:00
79be81a8a4
[chore] Optimize build_lz4 in build-thirdparty.sh ( #8653 )
2022-03-28 10:24:32 +08:00
b2861f36c4
[chore] optimize aws thirdparty package download. ( #8637 )
2022-03-28 09:35:51 +08:00
989e03ddf9
[improvement] Improve sig handler ( #8545 )
...
* Refactor glog's default signal handler
Co-authored-by: Zhengguo Yang <780531911@qq.com >
2022-03-22 10:40:31 +08:00
848acec584
[chore](dependency) update Croaring for good performance ( #8492 )
...
update Croaring for good performance, according to RoaringBitmap/CRoaring#320
2022-03-17 10:07:55 +08:00
a4b710cb2d
[chore](dependency) fix build thirdparty errors ( #8456 )
...
1. the patch for aws-c-cal-0.4.5 does not need anymore
2. remove duplicate bit_length document
3. add some debug log for routine load
2022-03-13 22:11:24 +08:00
f3c44bcd75
[chore][fix](librdkafka) disable librdkafka assert and update some thirdparty ( #8425 )
...
1. comment librdkafka `rd_assert(thrd_is_current(rkb->rkb_thread));` to avoid core dump
2. upgrade arrow to 7.0.0
3. upgrade aws sdk to 1.9
4. upgrade orc to 1.7.2
2022-03-12 22:09:06 +08:00
50a59f3f86
[license] Organize third-party dependent licenses for bianry releases ( #8350 )
2022-03-07 23:18:58 +08:00
7f3564cca6
[chore] Support aarch64 target with ldb_toolchain ( #8249 )
2022-02-27 10:38:17 +08:00
0d5b297cad
[chore] add aclocal/automake version check ( #8055 )
2022-02-22 09:19:14 +08:00
16020cbdf9
[fix](lateral-view) Fix bug that explode_json_array_string return unstable result ( #8152 )
...
Co-authored-by: morningman <chenmingyu@baidu.com >
2022-02-21 09:38:36 +08:00
789472a87d
[build] change docker image add perf tools and simdjson ( #8012 )
2022-02-11 18:54:09 +08:00
b1b1a8c64d
[chore] Fix the return code of function download_func ( #7919 )
2022-01-31 22:14:46 +08:00
3904447db8
[fix](toolchain) Allow building when system contains libunwind ( #7866 )
2022-01-29 12:33:32 +08:00
ea96242529
[chore] add set -e to download_thirdparty.sh ( #7867 )
...
Let the build process stop immediately if some command execute failed.
For example, patch failed due to command not found, otherwise it will stop until compile error.
2022-01-27 10:33:08 +08:00
ca0fac0722
[chore](ldb-toolchain) Support ldb_toolchain on ubuntu 20 ( #7846 )
2022-01-24 13:16:37 +08:00
f2cbf0a8d2
[chore] Improve the ldb toolchain compilation documentation ( #7829 )
...
Add document for compiling Doris with ldb toolchain
2022-01-21 21:36:43 +08:00
800a36343a
[chore] Prolog of hermetic build with GCC 11 and Clang 13. ( #7712 )
...
Prepare to generate hermetic build using GCC 11 and Clang 13.
The ideal toolchain would be ldb toolchain generated by [ldb_toolchain_gen.sh](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.3/ldb_toolchain_gen.sh )
To kick off a clang build, set `DORIS_TOOLCHAIN=clang` before running any build scripts.
2022-01-21 12:12:04 +08:00
0efef1b332
[fix](schema-change) Fix bug that schema change may return -102 error ( #7808 )
...
When using linked schema change, we need to check if all rowsets are of the same type,
ALPHA or BETA. otherwise, we need to use direct schema change to convert the data.
2022-01-21 10:59:54 +08:00
7b13ac5b31
[deps][chore] make openssl works with old glibc version ( #7541 )
...
1. build OpenSSL with --with-rand-seed=devrandom
2. Modified: brpc 1.0.0-rc02 -> 1.0.0
2021-12-31 23:19:04 +08:00
2d72c039ad
[deps](openssl) upgrade openssl to 1.1.1m ( #7446 )
...
upgrade openssl to 1.1.1m, ready for support SM2 / SM3 / SM4 national secret (national commercial password) algorithm
2021-12-21 10:09:36 +08:00
e9536a8cf1
[deps](cyrus_sasl) Add -fPIC for cyrus_sasl ( #7408 )
2021-12-17 13:11:25 +08:00
382351b0ee
[fix](ut) Fix run fe ut failed, be ut memory leak and build thirdparty failed ( #7377 )
2021-12-15 11:00:20 +08:00
92020e6e85
[deps](librdkafka) set --enable-sasl option in rdkafka build to enable plain password auth at routine load ( #7251 )
...
```
create routine load rd_001
on tb1
with append
COLUMNS(user_id, date)
properties (
"desired_concurrent_number" = "3",
"max_batch_interval" = "5",
"max_batch_rows" = "300000",
"max_batch_size" = "209715200",
"max_error_number" = "100",
"format" = "json"
)
from KAFKA (
"kafka_broker_list" = "127.0.0.1:9092",
"kafka_topic" = "topic1",
"property.security.protocol" = "sasl_plaintext",
"property.sasl.mechanism" = "PLAIN",
"property.sasl.username" = "your-username",
"property.sasl.password" = "your-password",
"property.group.id" ="group1",
"property.client.id" = "client-1",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
```
2021-12-02 11:44:37 +08:00
b6a9207a25
[deps](brpc) fix compile bug that could not find protobuf lib during compile ( #7197 )
2021-11-24 10:44:26 +08:00
00e653e812
[deps] add breakpad for minidump ( #7128 )
...
Related #7123
2021-11-18 14:34:38 +08:00
6c6380969b
[refactor] replace boost smart ptr with stl ( #6856 )
...
1. replace all boost::shared_ptr to std::shared_ptr
2. replace all boost::scopted_ptr to std::unique_ptr
3. replace all boost::scoped_array to std::unique<T[]>
4. replace all boost:thread to std::thread
2021-11-17 10:18:35 +08:00
e476e155c6
[Bug] Fix inappropriate Cmake option used to build ZSTD ( #7111 )
2021-11-16 10:04:16 +08:00
aba7d2ccae
[Thirdparty] Fix flatbuffers download url error ( #6968 )
...
Change google flatbuffers download URL in thirdparty.var.sh
2021-11-02 16:34:17 +08:00
51e210869a
[ARM64] Fix some problem when compiling on ARM64 platform ( #6836 ) ( #6872 )
...
With thirdparties 1.4.0 to 1.4.1
1. Add patch for aws-c-cal-0.4.5
2. Add some solutions for `undefined reference libpsl`
3. Move libgsasl to fix link problme of libcurl.
4. Downgrade openssl to 1.0.2k to fix problem of low version glibc
2021-10-19 13:26:02 +08:00
24d38614a0
[Dependency] Upgrade thirdparty libs ( #6766 )
...
Upgrade the following dependecies:
libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0
after this pr compile doris should use build-env:1.4.0
2021-10-15 13:03:04 +08:00
5ef3f59928
[Optimize][RoutineLoad] Avoid sending tasks if there is no data to be consumed ( #6805 )
...
1 Avoid sending tasks if there is no data to be consumed
By fetching latest offset of partition before sending tasks.(Fix [Optimize] Avoid too many abort task in routine load job #6803 )
2 Add a preCheckNeedSchedule phase in update() of routine load.
To avoid taking write lock of job for long time when getting all kafka partitions from kafka server.
3 Upgrade librdkafka's version to 1.7.0 to fix a bug of "Local: Unknown partition"
See offsetsForTimes fails with 'Local: Unknown partition' edenhill/librdkafka#3295
4 Avoid unnecessary storage migration task if there is no that storage medium on BE.
Fix [Bug] Too many unnecessary storage migration tasks #6804
2021-10-13 11:39:01 +08:00
83003cc372
[Thirdparty] Change libhdfs3 download url to a stable one( #6744 )
2021-10-03 11:56:36 +08:00
3812cbf117
fix librdkafka depends on high version of glibc ( #6634 )
2021-09-14 09:58:23 +08:00
4dd610c28d
[Feature] Support for storage layer benchmark ( #6506 )
...
* add benchmark tool
2021-09-02 09:57:19 +08:00
