635b5630e3
[fix](third-party) Fix build issues of third-party in ubuntu docker ( #9805 )
...
Fixes:
1. Fully static linkage of protoc may lead to core dump when using it.
2. Building libmyclient may fail due to miss-used option `-v` of `make`
3. Building lzma implicitly requires autopoint
4. Building xml2 implicitly requires pkg-config
2022-05-29 17:33:59 +08:00
cbbda7857b
[feature-wip](parquet-orc) Support orc scanner in vectorized engine ( #9541 )
2022-05-26 21:39:12 +08:00
ddda91c89d
[doc] Update dev image ( #9721 )
2022-05-23 11:59:15 +08:00
b3a2a92bf5
[deps] libhdfs3 build enable kerberos support ( #9524 )
...
Currently, the libhdfs3 library integrated by doris BE does not support accessing the cluster with kerberos authentication
enabled, and found that kerberos-related dependencies（gsasl and krb5） were not added when build libhdfs3.
so, this pr will enable kerberos support and rebuild libhdfs3 with dependencies gsasl and krb5:
- gsasl version: 1.8.0
- krb5 version: 1.19
2022-05-22 20:58:19 +08:00
defdae1e7d
[improvement](stream-load) adjust read unit of http to optimize stream load ( #9154 )
2022-05-20 09:52:36 +08:00
0b1b3e225d
Revert "[Thirdparty]Add llvm for codegen ( #8938 )" ( #8948 )
...
This reverts commit 32133621c69a2d7544549c5ea54ed6d9de60415e.
Reverts #8938
The LLVM requires GLIBC_2.15. I decided to create a branch for the llvm feature first.
And once we resolve the low version glibc issue, it will be merged back to master.
2022-04-11 13:03:16 +08:00
32133621c6
[Thirdparty]Add llvm for codegen ( #8938 )
2022-04-10 20:37:09 +08:00
79be81a8a4
[chore] Optimize build_lz4 in build-thirdparty.sh ( #8653 )
2022-03-28 10:24:32 +08:00
b2861f36c4
[chore] optimize aws thirdparty package download. ( #8637 )
2022-03-28 09:35:51 +08:00
989e03ddf9
[improvement] Improve sig handler ( #8545 )
...
* Refactor glog's default signal handler
Co-authored-by: Zhengguo Yang <780531911@qq.com >
2022-03-22 10:40:31 +08:00
848acec584
[chore](dependency) update Croaring for good performance ( #8492 )
...
update Croaring for good performance, according to RoaringBitmap/CRoaring#320
2022-03-17 10:07:55 +08:00
a4b710cb2d
[chore](dependency) fix build thirdparty errors ( #8456 )
...
1. the patch for aws-c-cal-0.4.5 does not need anymore
2. remove duplicate bit_length document
3. add some debug log for routine load
2022-03-13 22:11:24 +08:00
f3c44bcd75
[chore][fix](librdkafka) disable librdkafka assert and update some thirdparty ( #8425 )
...
1. comment librdkafka `rd_assert(thrd_is_current(rkb->rkb_thread));` to avoid core dump
2. upgrade arrow to 7.0.0
3. upgrade aws sdk to 1.9
4. upgrade orc to 1.7.2
2022-03-12 22:09:06 +08:00
50a59f3f86
[license] Organize third-party dependent licenses for bianry releases ( #8350 )
2022-03-07 23:18:58 +08:00
7f3564cca6
[chore] Support aarch64 target with ldb_toolchain ( #8249 )
2022-02-27 10:38:17 +08:00
0d5b297cad
[chore] add aclocal/automake version check ( #8055 )
2022-02-22 09:19:14 +08:00
16020cbdf9
[fix](lateral-view) Fix bug that explode_json_array_string return unstable result ( #8152 )
...
Co-authored-by: morningman <chenmingyu@baidu.com >
2022-02-21 09:38:36 +08:00
789472a87d
[build] change docker image add perf tools and simdjson ( #8012 )
2022-02-11 18:54:09 +08:00
b1b1a8c64d
[chore] Fix the return code of function download_func ( #7919 )
2022-01-31 22:14:46 +08:00
3904447db8
[fix](toolchain) Allow building when system contains libunwind ( #7866 )
2022-01-29 12:33:32 +08:00
ea96242529
[chore] add set -e to download_thirdparty.sh ( #7867 )
...
Let the build process stop immediately if some command execute failed.
For example, patch failed due to command not found, otherwise it will stop until compile error.
2022-01-27 10:33:08 +08:00
ca0fac0722
[chore](ldb-toolchain) Support ldb_toolchain on ubuntu 20 ( #7846 )
2022-01-24 13:16:37 +08:00
f2cbf0a8d2
[chore] Improve the ldb toolchain compilation documentation ( #7829 )
...
Add document for compiling Doris with ldb toolchain
2022-01-21 21:36:43 +08:00
800a36343a
[chore] Prolog of hermetic build with GCC 11 and Clang 13. ( #7712 )
...
Prepare to generate hermetic build using GCC 11 and Clang 13.
The ideal toolchain would be ldb toolchain generated by [ldb_toolchain_gen.sh](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.3/ldb_toolchain_gen.sh )
To kick off a clang build, set `DORIS_TOOLCHAIN=clang` before running any build scripts.
2022-01-21 12:12:04 +08:00
0efef1b332
[fix](schema-change) Fix bug that schema change may return -102 error ( #7808 )
...
When using linked schema change, we need to check if all rowsets are of the same type,
ALPHA or BETA. otherwise, we need to use direct schema change to convert the data.
2022-01-21 10:59:54 +08:00
7b13ac5b31
[deps][chore] make openssl works with old glibc version ( #7541 )
...
1. build OpenSSL with --with-rand-seed=devrandom
2. Modified: brpc 1.0.0-rc02 -> 1.0.0
2021-12-31 23:19:04 +08:00
2d72c039ad
[deps](openssl) upgrade openssl to 1.1.1m ( #7446 )
...
upgrade openssl to 1.1.1m, ready for support SM2 / SM3 / SM4 national secret (national commercial password) algorithm
2021-12-21 10:09:36 +08:00
e9536a8cf1
[deps](cyrus_sasl) Add -fPIC for cyrus_sasl ( #7408 )
2021-12-17 13:11:25 +08:00
382351b0ee
[fix](ut) Fix run fe ut failed, be ut memory leak and build thirdparty failed ( #7377 )
2021-12-15 11:00:20 +08:00
92020e6e85
[deps](librdkafka) set --enable-sasl option in rdkafka build to enable plain password auth at routine load ( #7251 )
...
```
create routine load rd_001
on tb1
with append
COLUMNS(user_id, date)
properties (
"desired_concurrent_number" = "3",
"max_batch_interval" = "5",
"max_batch_rows" = "300000",
"max_batch_size" = "209715200",
"max_error_number" = "100",
"format" = "json"
)
from KAFKA (
"kafka_broker_list" = "127.0.0.1:9092",
"kafka_topic" = "topic1",
"property.security.protocol" = "sasl_plaintext",
"property.sasl.mechanism" = "PLAIN",
"property.sasl.username" = "your-username",
"property.sasl.password" = "your-password",
"property.group.id" ="group1",
"property.client.id" = "client-1",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
```
2021-12-02 11:44:37 +08:00
b6a9207a25
[deps](brpc) fix compile bug that could not find protobuf lib during compile ( #7197 )
2021-11-24 10:44:26 +08:00
00e653e812
[deps] add breakpad for minidump ( #7128 )
...
Related #7123
2021-11-18 14:34:38 +08:00
6c6380969b
[refactor] replace boost smart ptr with stl ( #6856 )
...
1. replace all boost::shared_ptr to std::shared_ptr
2. replace all boost::scopted_ptr to std::unique_ptr
3. replace all boost::scoped_array to std::unique<T[]>
4. replace all boost:thread to std::thread
2021-11-17 10:18:35 +08:00
e476e155c6
[Bug] Fix inappropriate Cmake option used to build ZSTD ( #7111 )
2021-11-16 10:04:16 +08:00
aba7d2ccae
[Thirdparty] Fix flatbuffers download url error ( #6968 )
...
Change google flatbuffers download URL in thirdparty.var.sh
2021-11-02 16:34:17 +08:00
51e210869a
[ARM64] Fix some problem when compiling on ARM64 platform ( #6836 ) ( #6872 )
...
With thirdparties 1.4.0 to 1.4.1
1. Add patch for aws-c-cal-0.4.5
2. Add some solutions for `undefined reference libpsl`
3. Move libgsasl to fix link problme of libcurl.
4. Downgrade openssl to 1.0.2k to fix problem of low version glibc
2021-10-19 13:26:02 +08:00
24d38614a0
[Dependency] Upgrade thirdparty libs ( #6766 )
...
Upgrade the following dependecies:
libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0
after this pr compile doris should use build-env:1.4.0
2021-10-15 13:03:04 +08:00
5ef3f59928
[Optimize][RoutineLoad] Avoid sending tasks if there is no data to be consumed ( #6805 )
...
1 Avoid sending tasks if there is no data to be consumed
By fetching latest offset of partition before sending tasks.(Fix [Optimize] Avoid too many abort task in routine load job #6803 )
2 Add a preCheckNeedSchedule phase in update() of routine load.
To avoid taking write lock of job for long time when getting all kafka partitions from kafka server.
3 Upgrade librdkafka's version to 1.7.0 to fix a bug of "Local: Unknown partition"
See offsetsForTimes fails with 'Local: Unknown partition' edenhill/librdkafka#3295
4 Avoid unnecessary storage migration task if there is no that storage medium on BE.
Fix [Bug] Too many unnecessary storage migration tasks #6804
2021-10-13 11:39:01 +08:00
83003cc372
[Thirdparty] Change libhdfs3 download url to a stable one( #6744 )
2021-10-03 11:56:36 +08:00
3812cbf117
fix librdkafka depends on high version of glibc ( #6634 )
2021-09-14 09:58:23 +08:00
4dd610c28d
[Feature] Support for storage layer benchmark ( #6506 )
...
* add benchmark tool
2021-09-02 09:57:19 +08:00
abbc9202af
support routine load isolation_level read_committed ( #6191 )
...
Co-authored-by: Geoffrey <gaofeng01@rd.netease.com >
2021-08-30 17:22:08 +08:00
d9cc235d6d
Fix typo pdqosrt ( #6441 )
2021-08-14 10:24:12 +08:00
9216735cfa
[New Featrue] Support Vectorization Execution Engine Interface For Doris ( #6329 )
...
1. FE vectorized plan code
2. Function register vec function
3. Diff function nullable type
4. New thirdparty code and new thrift struct
2021-08-11 14:54:06 +08:00
de2dd2c015
[Thirdparty] Move DataTables download url to a fixed download address ( #6189 )
...
The origin download url is always changing: https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.25
So we put it in our own http server.
If someone can offer an official url for DataTables, please update this.
2021-07-10 10:13:26 +08:00
48bd680068
update download info for boost and datatables ( #6008 )
2021-06-18 11:38:41 +08:00
d7e62e361f
[Bug] Fix that build thirdparty of parallel-hashmap-1.33 failed on ubuntu18.04 ( #6033 )
...
Co-authored-by: caiconghui <caiconghui@xiaomi.com >
2021-06-17 14:45:34 +08:00
80220af271
[Enhancement] Use Parallel Hash Map Replace Unordered Map In Dict Encodeing Map And Hyper Set ( #5990 )
...
Use Parallel Hash Map Replace Unordered Map In Dict Encodeing Map And Hyper Set To Improve Ferformance
2021-06-10 17:38:08 +08:00
109b55ee5f
[Shell] Add build parallel option ( #5819 )
...
Add build parallel option then we can build project with a user specified parallel not a fixed value.
2021-05-19 09:32:58 +08:00
2883ebeb0a
[Compile] fix s2 link /usr/local/openssl ( #5810 )
2021-05-17 11:52:21 +08:00
