163a38a527
[opt](Nereids) support sql cache ( #22144 )
...
1. let Nereids support sql cache
2. let legacy planner's sql cache supports union all
2023-07-27 09:57:31 +08:00
82fe78ce84
Update table binlog config disable failure when db binlog is enable && ( #22253 )
...
modify table binlog more than one property
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-07-27 09:54:24 +08:00
8fb28ecc9e
[test](partial-update) add some cases for partial-update ( #22210 )
2023-07-27 09:52:40 +08:00
dcd6844ea5
[improvement](regression-test) add partial update with schema change case ( #22213 )
2023-07-27 09:51:42 +08:00
fb41265c27
[opt](Nereids) add boolean type signature for sum aggregate function ( #21959 )
2023-07-27 09:41:19 +08:00
12222eb145
forbid: regression-test/pipeline/p0/conf/regression-conf.groovy ( #22271 )
2023-07-26 23:27:18 +08:00
7ed997cba8
[improvement](s3) increase the connection num of s3 client ( #22049 )
...
The default maxConnection of s3 client is 25.
It should be increased to improve the query performance.
In my test, a tpch 300 benchmark with data stored on object storage, the total time
can reduce from 430s -> 330s
2023-07-26 22:52:40 +08:00
560731f392
[Bug](runtime-filter) fix probe expr prepared twice on minmax runtime filter ( #22229 )
...
fix probe expr prepared twice on minmax runtime filter
2023-07-26 19:44:35 +08:00
331ae6d2cb
[fix](Unique-Key) fix version upgrade caused MOR to become MOW ( #22243 )
2023-07-26 19:38:24 +08:00
490ad14720
[stats](nereids)in predicate range adjust ( #22156 )
...
1. refactor in-predicate filter estimation
example: A in (1, 2, 3, 4)
after in-preidcate filter, A.stats.max<=4 and A.stats.min>=1
2. maintain minExpr and maxExpr in in-predicate stats derive
2023-07-26 19:10:04 +08:00
21ea0055fc
[improvement](scanner) use batch size of session instead of limit to improve performance of reading ( #22240 )
2023-07-26 18:57:42 +08:00
fdb26a524a
[fix](compile) Fix FE compile error ( #22261 )
...
Fix FE compile error
2023-07-26 18:39:21 +08:00
9e16c69925
[improvement](compression) support LZ4_HC algorithm and parse LZ4_RAW ( #22165 )
2023-07-26 18:23:39 +08:00
4e57f45d8e
[fix](partial-update) sequence column not updated if using function_column.sequence_type ( #22250 )
2023-07-26 18:22:43 +08:00
9a07ae890a
[fix](point query) Fix ArrayIndexOutOfBoundsException if close a prepare stmt ( #22237 )
2023-07-26 18:22:07 +08:00
2c7d7937e5
[doc](fe) Add split-brain influence for metadata_failure_recovery ( #21801 )
2023-07-26 18:21:29 +08:00
8ff487cc4b
[fix](cast) fix invalid value error when casting null date value to string then casting to date value ( #22223 )
2023-07-26 17:59:01 +08:00
14dcc53135
[fix](Nereids) cast time should turn nullable on all valid types ( #22242 )
...
valid types to cast to time/timev2:
- TINYINT
- SMALLINT
- INT
- BIGINT
- LARGEINT
- FLOAT
- DOUBLE
- CHAR
- VARCHAR
- STRING
2023-07-26 17:56:19 +08:00
be69025878
[opt](Nereids) add partial update support for delete stmt ( #22184 )
...
Currently, the new optimizer don't consider anything about partial update.
This PR add the ability to convert a delete statement to a partial update insert statement
for merge-on-write unique table
2023-07-26 17:34:31 +08:00
f621d8ea65
[doc](stats) full auto analyze docs #21918
...
Add description for some fe config about analyze
2023-07-26 17:30:10 +08:00
0b05c0b841
[chore](tools) add icon.svg to .idea to let idea present doris's logo ( #22234 )
2023-07-26 17:27:47 +08:00
582acad8a1
[feature](stats) Enable period time with cron expr ( #22095 )
...
Support such grammar
ANALYZE TABLE test WITH CRON "* * * * * ?"
Such job would be scheduled as the cron expr specifie, but natively support minute-level schedule only
2023-07-26 17:25:57 +08:00
964ac4e601
[opt](nereids) Retry when async analyze task failed ( #21889 )
...
Retry at most 5 times when async analyze task execution failed
2023-07-26 17:16:56 +08:00
a6311e2f95
fix erase ( #22235 )
2023-07-26 17:06:37 +08:00
9451382428
[Improvement](aggregate) optimization for AggregationMethodKeysFixed::insert_keys_into_columns ( #22216 )
...
optimization for AggregationMethodKeysFixed::insert_keys_into_columns
2023-07-26 16:19:15 +08:00
af20d0c521
[fix](binlog) Fix BinlogUtils getExpiredMs overflow ( #22174 )
2023-07-26 15:15:34 +08:00
6c0c66e664
[exceptionsafe](expr) exprcontext's open and prepare method should catch exception ( #22230 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >ExprContext may throw exception during expr->open, and it will core in non-pipeline mode.
2023-07-26 14:46:24 +08:00
9f3960b460
[fix](kerberos)fix kerberos config read ( #22081 )
...
we should read kerberos config from properties first, so use override hive conf to set.
2023-07-26 13:36:12 +08:00
bb67a1467a
[fix](Nereids): mergeGroup should merge target Group into existed Group ( #22123 )
2023-07-26 13:13:25 +08:00
1f3de0eae3
[fix](memory) fix invalid large memory check && fix memory info thread safety ( #22027 )
...
fix invalid large memory check
fix memory info thread safety
2023-07-26 12:18:31 +08:00
21a3593a9a
[fix](Nereids) translate failed when enable topn two phase opt ( #22197 )
...
1. should not add rowid slot to reslovedTupleExprs
2. should set notMaterialize to sort's tuple when do two phase opt
2023-07-26 11:38:50 +08:00
f4396ef8c7
[Fix](regression-test) nereids_p0/javaudf and nereids_p0/outfile cases cannot run on multi be cluster ( #21929 )
...
the cases as title will not pass in multi-be environment because the be queried doesn't contain outfile data. We will copy the outfile to every instance to fix it.
2023-07-26 11:33:51 +08:00
4c4f08f805
[fix](hudi) the required fields are empty if only reading partition columns ( #22187 )
...
1. If only read the partition columns, the `JniConnector` will produce empty required fields, so `HudiJniScanner` should read the "_hoodie_record_key" field at least to know how many rows in current hoodie split. Even if the `JniConnector` doesn't read this field, the call of `releaseTable` in `JniConnector` will reclaim the resource.
2. To prevent BE failure and exit, `JniConnector` should call release methods after `HudiJniScanner` is initialized. It should be noted that `VectorTable` is created lazily in `JniScanner`, so we don't need to reclaim the resource when `HudiJniScanner` is failed to initialize.
## Remaining works
Other jni readers like `paimon` and `maxcompute` may encounter the same problems, the jni reader need to handle this abnormal situation on its own, and currently this fix can only ensure that BE will not exit.
2023-07-26 10:59:45 +08:00
d4a4c172ea
[Improve](serde)update serialize and deserialize text for data type ( #21109 )
2023-07-26 10:06:16 +08:00
ea2a7a8e56
[Docs](docs) Rename Release Note Title and File name of CN & EN Version ( #22157 )
2023-07-26 09:20:06 +08:00
9abf32324b
[improvement](jdbc) add timestamp put to datev2 ( #21680 )
2023-07-26 09:10:34 +08:00
b12c993f05
[Fix](multi-catalog) Do not throw exceptions when file not exists for external hive tables. ( #22140 )
...
* [Fix](multi-catalog) Not throw exceptions when file not exists for query of hms catalog.
* [Fix](multi-catalog) Not throw exceptions when file not exists for query of hms catalog.
---------
Co-authored-by: 王翔宇 <wangxiangyu@360shuke.com >
2023-07-26 09:05:52 +08:00
5f846056f7
[fix](forward) fix MissingFormatArgumentException when failed to forward stmt to Master ( #22142 )
2023-07-26 09:00:04 +08:00
7b270d1ae9
[Fix](mutli-catalog) Fix orc reader crashed when hdfs reading error by catching exception. ( #22193 )
...
orc reader crashed when hdfs reading error.
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/zcp_repo/be/src/common/signal_handler.h:413
1# 0x00007F6F8B3C00C0 in /lib/x86_64-linux-gnu/libc.so.6
2# raise in /lib/x86_64-linux-gnu/libc.so.6
3# abort in /lib/x86_64-linux-gnu/libc.so.6
4# _gnu_cxx::_verbose_terminate_handler() [clone .cold] at ../../../../libstdc+-v3/libsupc+/vterminate.cc:75
5# _cxxabiv1::_terminate(void ()) at ../../../../libstdc+-v3/libsupc+/eh_terminate.cc:48
6# 0x0000555CBC4718C1 in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
7# 0x0000555CBC471A14 in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
8# doris::vectorized::ORCFileInputStream::read(void*, unsigned long, unsigned long) at /home/zcp/repo_center/zcp_repo/be/src/vec/exec/format/orc/vorc_reader.cpp:121
9# orc::SeekableFileInputStream::Next(void const*, int) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
10# orc::DecompressionStream::readHeader() in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
11# orc::DecompressionStream::Next(void const*, int) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
12# void orc::RleDecoderV2::next<long>(long*, unsigned long, char const*) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
13# orc::StringDictionaryColumnReader::loadDictionary() in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
14# orc::StructColumnReader::loadStringDicts(std::unordered_map<unsigned long, std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unordered_map<std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, orc::StringDictionary*, std::hash<std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, orc::StringDictionary*> > >, orc::StringDictFilter const) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
15# orc::RowReaderImpl::startNextStripe(orc::ReadPhase const&) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
16# orc::RowReaderImpl::nextBatch(orc::ColumnVectorBatch&, void*) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
17# doris::vectorized::OrcReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /home/zcp/repo_center/zcp_repo/be/src/vec/exec/format/orc/vorc_reader.cpp:1420
18# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/zcp_repo/be/src/vec/exec/scan/vfile_scanner.cpp:250
19# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
20# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) at /home/zcp/repo_center/zcp_repo/be/src/vec/exec/scan/scanner_scheduler.cpp:335
21# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::
2023-07-26 08:57:31 +08:00
b20af13966
[fix][jdbc_case]Change the method of obtaining the driver for case test_doris_jdbc_catalog 0724 #22164
2023-07-26 08:48:16 +08:00
cf717882d8
[fix](jdbc catalog) fix hana jdbc table bug ( #22190 )
2023-07-26 08:45:06 +08:00
ba3a0922eb
[fix](ipv6)Support IPV6 ( #22219 )
...
fe：Remove restrictions from IPv4
be: thrift server Specify binding address
be: Restore changed code of “be/src/olap/task/engine_clone_task.cpp”
2023-07-26 08:40:32 +08:00
e8f4323e0f
[Fix](jdbcCatalog) fix typo of some variable #22214
2023-07-26 08:34:45 +08:00
111957401b
[improvement](invert index) Added lucene9.5 unicode tokenizer ( #22217 )
2023-07-26 00:50:24 +08:00
3414d1a61f
[fix](hudi) table schema is not the same as parquet schema ( #22186 )
...
Upgrade hudi version from 0.13.0 to 0.13.1, and keep the hudi version of jni scanner the same as that of FE.
This may fix the bug of the table schema is not same as parquet schema.
2023-07-26 00:29:53 +08:00
cf677b327b
[fix](jdbc catalog) Fixed mappings with type errors for bool and tinyint(1) ( #22089 )
...
First of all, mysql does not have a boolean type, its boolean type is actually tinyint(1), in the previous logic, We force tinyint(1) to be a boolean by passing tinyInt1isBit=true, which causes an error if tinyint(1) is not a 0 or 1, Therefore, we need to match tinyint(1) according to tinyint instead of boolean, and this change will not affect the correctness of where k = 1 or where k = true queries
2023-07-25 22:45:22 +08:00
b2be42c31c
[fix](jdbc catalog) fix jdbc catalog like expr query error ( #22141 )
2023-07-25 22:30:28 +08:00
f44660db1a
[chore](merge-on-write) disable single replica load and compaction for mow table ( #22188 )
2023-07-25 22:05:22 +08:00
5c8eda8685
[enhencement](regression) add UPDATE & DELETE tests for MOW partial update ( #22212 )
2023-07-25 22:03:38 +08:00
c498b2cf69
[fix](partial-update) disable partial update when undergoing a schema changing process ( #22133 )
2023-07-25 21:33:20 +08:00
