Commit Graph

54 Commits

Author SHA1 Message Date
ba351af452 [enhancement](thirdparty) upgrade thirdparty libs - again (#23414)
submit again #23290 (not upgrade brpc, because bthread local has error)

protobuf 3.15.0 -> 21.11
glog 0.4.0 -> 0.6.0
lz4 1.9.3 -> 1.9.4
curl 7.79.0 -> 8.2.1
zstd 1.5.2 -> 1.5.5
arrow 7.0.0 -> 13.0.0
abseil 20220623.1 -> 20230125.3
orc 1.7.2 -> 1.9.0
jemalloc for arrow 5.2.1 -> 5.3.0
xsimd 7.0.0 -> 13.0.0
opentelemetry-proto 0.19.0 -> 1.0.0
opentelemetry 1.8.3 -> 1.10.0

new:
c-ares -> 1.19.1
grpc -> 1.54.3
2023-08-26 22:59:10 +08:00
303d5ac3f7 Revert "[enhancement](thirdparty) upgrade thirdparty libs (#23290)" (#23420)
This reverts commit 2185268bab2f5890a2c3c9a3c315375a971c4416.

Co-authored-by: stephen <hello-stephen@qq.com>
2023-08-24 14:40:51 +08:00
2185268bab [enhancement](thirdparty) upgrade thirdparty libs (#23290)
protobuf 3.15.0 -> 21.11
glog 0.4.0 -> 0.6.0
lz4 1.9.3 -> 1.9.4
curl 7.79.0 -> 8.2.1
brpc 1.4.0 -> 1.6.0
zstd 1.5.2 -> 1.5.5
arrow 7.0.0 -> 13.0.0
abseil 20220623.1 -> 20230125.3
orc 1.7.2 -> 1.9.0
jemalloc for arrow 5.2.1 -> 5.3.0
xsimd 7.0.0 -> 13.0.0
opentelemetry-proto 0.19.0 -> 1.0.0
opentelemetry 1.8.3 -> 1.10.0

new:
c-ares -> 1.19.1
grpc -> 1.54.3
2023-08-23 20:25:49 +08:00
f68d3a660e [improvement](opentelemetry) upgrade opentelemetry jar to v1.26.0 and opentelemetry-cpp to v1.8.3 (#19733)
why upgrade? anything wrong?

Try to fix the problem about opentelemetry::v1::ext::http::client::curl::HttpOperation::Send(), I have updated the pr info.
2023-05-18 18:46:20 +08:00
Pxl
a2c9ed7be8 [Chore](build) fix some undefined behavior about incomplete type vector #19753 2023-05-18 15:13:45 +08:00
e98f4c4a5e [fix](be) BE UT built against Clang-16 failed (#19610)
If we use Clang-16 to build the third-party libraries and build doris_be_test against them, we can not run doris_be_test successfully. Some errors with BRPC occur.

I tested this on Linux (x86_64) and macOS (x86_64/arm64), these errors always raised.
2023-05-13 22:32:29 +08:00
673cbe3317 [chore](build) Porting to GCC-13 (#19293)
Support using GCC-13 to build the codebase.
2023-05-08 10:42:06 +08:00
5c8ecfbf9c [fix](thirdparty) fix opentelemetry error message compiling with ubsan (#18912) 2023-05-05 19:09:43 +08:00
f9c4542d04 [chore](build) Porting to Clang-16 (#18196)
This PR ports the codebase to Clang-16.

Upgrade some third-party libraries:
1. Apache BRPC: 1.2.0 -> 1.4.0 (Some bugs are fixed and all patches for 1.2.0 can be removed.)
2. Boost: 1.73.0 -> 1.81.0 (Porting to Clang-16)
3. libclucene: 2.4.6 -> 2.4.8 (Porting to Clang-16)
2023-03-30 10:36:29 +08:00
fd0c352144 [fix](thirdparty) patch sqltypes.h in odbc to avoid conflict with clucene macro TCHAR (#15926)
Fix conflit name TCHAR in odbc sqltypes.h and clucene clucene-config.h.
Change TCHAR to TWCHAR in odbc sqltypes.h, because TCHAR in odbc is not found used in doris,
but there are too many places to call clucene's TCHAR.

thirdparty/installed/include/sqltypes.h: 
`typedef char                            TCHAR;`

thirdparty/installed/include/CLucene/clucene-config.h: 
`typedef  wchar_t TCHAR;`
2023-01-15 10:10:00 +08:00
fe02b08e04 [Improvement](thirdparty)upgrade simdjson from 1.0.2 to 3.0.1 (#15412)
Upgrade simdjson from 1.0.2 to latest version 3.0.1 to avoid -mlzcnt compiler flag causing BE UT(macOS) failure.
simdjson is now only used by VJsonScanner and disabled by default. So the impact of upgrade is limited.
2022-12-28 12:24:16 +08:00
7d49ddf50c [bugfix](thirdparty) patch simdjson to avoid conflict with odbc macro BOOL (#15223)
fix conflit name BOOL in odbc sqltypes.h and simdjson element.h. Change BOOL to BOOLEAN in simdjson.

- thirdparty/installed/include/sqltypes.h

> #define	BOOL				int


- thirdparty/src/simdjson-1.0.2/include/simdjson/dom/element.h

> enum class element_type {
>   ARRAY = '[',     ///< dom::array
>   OBJECT = '{',    ///< dom::object
>   INT64 = 'l',     ///< int64_t
>   UINT64 = 'u',    ///< uint64_t: any integer that fits in uint64_t but *not* int64_t
>   DOUBLE = 'd',    ///< double: Any number with a "." or "e" that fits in double.
>   STRING = '"',    ///< std::string_view
>   BOOL = 't',      ///< bool
>   NULL_VALUE = 'n' ///< null
> };
>
2022-12-22 09:40:04 +08:00
Pxl
c816d9d6e5 [Chore](s2geo) disable ABSL_BTREE_ENABLE_GENERATIONS and use CMAKE_BUILD_TYPE to rep… (#15104)
disable ABSL_BTREE_ENABLE_GENERATIONS and use CMAKE_BUILD_TYPE to replace o3
2022-12-15 21:36:28 +08:00
Pxl
decabbb933 [Chore](s2geo) upgrade s2geo to 0.10.0 (#15002)
upgrade s2geo to 0.10.0
2022-12-13 10:34:51 +08:00
7624c80d83 [Feature](Kafka) Add kerberos support for kafka (#14431)
Compile librdkafka with Kerberos SASL GSSAPI support.
2022-11-21 20:45:50 +08:00
015f8ab78d [enhancement](thirdparty) support create stripe reader by column names (#14184)
ORC NextStripeReader now only support read columns by indices, but it is hard to get column indices for complex types.
We patch ORC adapter to support read columns by column names.
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-11-11 15:10:20 +08:00
9350c0be80 [Bug](aarch64) Fix the BRPC "sched_to itself" error when building BE by Clang on Linux aarch64 (#13270) 2022-10-11 15:38:28 +08:00
429ac929fb [chore](build) Support building from source on ubuntu-22.04 (aarch64) (#12813)
Support building from source on ubuntu-22.04
2022-09-27 10:29:13 +08:00
cb06e67fba [fix](tracing) Fix opentelemetry log output to be.out (#11856) 2022-09-18 17:40:23 +08:00
47d43b34b3 [enhancement](thirdparty) Compile Jemalloc separately on thirdparty (#12577)
Compile Jemalloc separately and optimize the configuration
2022-09-14 23:31:48 +08:00
a912910d4b [fix](thirdparty) Fix the errors while building brpc (#11714) 2022-08-12 13:59:44 +08:00
0f555e90cb [deps](brpc) Upgrade Brpc to 1.2.0 fix _dl_sym undefined reference on new verison of glibc (#11688) 2022-08-12 09:07:57 +08:00
c6f520fab4 [thirdparty](brpc) fix _dl_sym undefined reference on Ubuntu22.04 (#11643)
Co-authored-by: qcloud <ubuntu@localhost.localdomain>
2022-08-10 19:23:10 +08:00
6a53cf8efa [dependency](be) Add vectorscan for support hypserscan on ARM (#11102)
hyperscan is a high-performance regular expression matching library, but can not be used on ARM.
vectorscan is an ARM port for hyperscan, it can be used as a drop in replacement.

Since hyperscan is original created by Intel and is popular and mature on x86, so we just use vectorscan
only for aarch64 when build thirdparty.
2022-07-26 11:28:25 +08:00
4d158f9050 [dependency](arrow) Add GetRawORCReader function for arrow orc reader (#11069)
Add a new function in arrow adapter to get the raw orc reader which we can get more information
from such offset or min/max value.
And this will be used in #1046
This modify is inspired by Clickhouse
2022-07-21 22:23:05 +08:00
e3c19ded44 [enhancement](thirdparty) Support building thirdparty on macOS (#10677) 2022-07-18 10:50:30 +08:00
e159e748df [chore](dependency) fix opentelemetry-cpp enable o3 optimization will core. (#10675) 2022-07-08 10:08:07 +08:00
e0cf2677a0 [dependency][enhancement] support build libhdfs in arm cpus (#10018)
Supports native hdfs functionality on arm cpu
This pr mainly upgrades libdfs3 and supports running on arm,and make libhdfs3 with kerberos as default
2022-06-10 19:40:41 +08:00
a34d4b55f9 [dependency] Add opentelemetry related third-party library. (#9990) 2022-06-10 09:59:07 +08:00
73a3093539 [dependency] add hyperscan and its dependency ragel to thirdparty (#9964) 2022-06-09 23:36:09 +08:00
b3a2a92bf5 [deps] libhdfs3 build enable kerberos support (#9524)
Currently, the libhdfs3 library integrated by doris BE does not support accessing the cluster with kerberos authentication 
enabled, and found that kerberos-related dependencies(gsasl and krb5) were not added when build libhdfs3.

so, this pr will enable kerberos support and rebuild libhdfs3 with dependencies gsasl and krb5:

- gsasl version: 1.8.0
- krb5 version: 1.19
2022-05-22 20:58:19 +08:00
defdae1e7d [improvement](stream-load) adjust read unit of http to optimize stream load (#9154) 2022-05-20 09:52:36 +08:00
848acec584 [chore](dependency) update Croaring for good performance (#8492)
update Croaring for good performance, according to RoaringBitmap/CRoaring#320
2022-03-17 10:07:55 +08:00
f3c44bcd75 [chore][fix](librdkafka) disable librdkafka assert and update some thirdparty (#8425)
1. comment  librdkafka `rd_assert(thrd_is_current(rkb->rkb_thread));` to avoid core dump
2. upgrade arrow to 7.0.0
3. upgrade aws sdk to 1.9
4. upgrade orc to 1.7.2
2022-03-12 22:09:06 +08:00
7f3564cca6 [chore] Support aarch64 target with ldb_toolchain (#8249) 2022-02-27 10:38:17 +08:00
3904447db8 [fix](toolchain) Allow building when system contains libunwind (#7866) 2022-01-29 12:33:32 +08:00
ca0fac0722 [chore](ldb-toolchain) Support ldb_toolchain on ubuntu 20 (#7846) 2022-01-24 13:16:37 +08:00
800a36343a [chore] Prolog of hermetic build with GCC 11 and Clang 13. (#7712)
Prepare to generate hermetic build using GCC 11 and Clang 13.
The ideal toolchain would be ldb toolchain generated by [ldb_toolchain_gen.sh](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.3/ldb_toolchain_gen.sh)

To kick off a clang build, set `DORIS_TOOLCHAIN=clang` before running any build scripts.
2022-01-21 12:12:04 +08:00
51e210869a [ARM64] Fix some problem when compiling on ARM64 platform (#6836) (#6872)
With thirdparties 1.4.0 to 1.4.1

1. Add patch for aws-c-cal-0.4.5
2. Add some solutions for `undefined reference libpsl`
3. Move libgsasl to fix link problme of libcurl.
4. Downgrade openssl to 1.0.2k to fix problem of low version glibc
2021-10-19 13:26:02 +08:00
24d38614a0 [Dependency] Upgrade thirdparty libs (#6766)
Upgrade the following dependecies:

libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0

after this pr compile doris should use build-env:1.4.0
2021-10-15 13:03:04 +08:00
bd88309346 [Refactor] fix warning in gcc8+, fix warning from brpc, s2 (#5763)
Fix warning from brpc, S2
Fix -Warray-bounds
2021-05-12 10:38:46 +08:00
29a3fa1084 [Feature] Support read data with format of parquet from hdfs, using libhdfs3 (#5686)
Add new lib, Backend can read data from hdfs without broker,
this patch include libhdfs3.a which can read file on hdfs.
This patch will make reading the data from hdfs with parquet possible.
By this, we will support more format of file on hdfs in the future,
and we will support other metadata in the future.
2021-04-24 17:41:48 +08:00
ad67dd34a0 update gcc to gcc 10 and support c++17 (#5394)
* update gcc to gcc 10 and support c++17
    update brpc to 0.9.7
    update boost to 1.73
    remove third-party boost 1.54 for mysql

* update cmake version

* ignore jdk version

* remove unused patch

* avoid use SYS_getrandom call
2021-03-25 09:30:38 +08:00
12c59ba889 [Thirdparty][glog][bug] convert init be log file length use fopen function (#3649) 2020-05-26 22:42:50 +08:00
0d457692bc [incubator-doris][thirdpary][glog][bug] Calucate file length at the be start (#3594) 2020-05-15 15:15:54 +08:00
7099fcf2d3 Remove unused file (#2819)
This file was replace by thirdparty/patches/incubator-brpc-0.9.5.patch in pr #2798 . But this file was forgotten to remove in #2798
2020-01-21 13:43:48 +08:00
0f829ca4c4 Add arm compatible patches (#2798) 2020-01-20 00:21:47 +08:00
726fa923c9 Support compiling LLVM in aarch64 (#2559) 2019-12-26 21:59:06 +08:00
a76333a400 Support s2 on aarch64 (#2568) 2019-12-25 18:56:52 +08:00
35503cf8a3 Support glog on aarch64 (#2563) 2019-12-25 13:56:15 +08:00