cee8cc44e2
[Bug](insert)fix insert wrong data on mv when stmt have multiple values ( #27297 )
...
fix insert wrong data on mv when stmt have multiple values
2023-11-21 12:55:30 +08:00
c0f22e8feb
[FIX](complextype)fix struct nested complex collection type and and regresstest ( #26973 )
2023-11-20 22:29:12 +08:00
0b459e50fb
[fix](partial update) keep case insensitivity and use the columns' origin names in partialUpdateCols in origin planner ( #27223 )
...
close : #27161
2023-11-20 21:16:28 +08:00
273cbfc36c
[fix](inverted index) reset fs_writer to nullptr before throw exception ( #27202 )
2023-11-20 17:40:56 +08:00
7164b7cebb
[case](regression) add read json by line test case ( #26670 )
...
Co-authored-by: qinhao <qinhao@newland.com.cn >
2023-11-20 17:33:40 +08:00
add6bdb240
[fix](multi-catalog)add the max compute fe ut and fix download expired ( #27007 )
...
1. add the max compute fe ut and fix download expired
2. solve memery leak when allocator close
3. add correct partition rows
2023-11-20 10:42:07 +08:00
febd60c75f
[fix](join) incorrect result of left join with other conjuncts ( #27238 )
2023-11-19 15:36:39 +08:00
b560d863c2
[fix](dynamic) Fix error reporting when dynamic partition properties contain incorrect attributes ( #25373 )
2023-11-19 09:51:57 +08:00
329abc3452
[feature](nereids) runtime filter prune when column stats are not available ( #27099 )
...
1. prune rf when column stats are not available
2. print rf in "explain shape plan", both join side and apply side
3. add regression case to check plan shape/rf/rf prune for tpch_sf1000 (stats are not available)
2023-11-18 19:32:33 +08:00
b42828cf69
[fix](window_function) min/max/sum/avg should be always nullable ( #27104 )
...
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com >
2023-11-18 18:41:42 +08:00
2f41e0c823
[FIX](complextype)fix information schema for complex type ( #27203 )
...
when we select in information schema , here do not show complex type information
2023-11-18 11:32:32 +08:00
4b6330cb93
[regression test](http stream) add case for strict_mode ( #27098 )
2023-11-18 00:10:02 +08:00
fc47eb51f7
[regression test](schema change) add some schema change regression cases ( #27112 )
2023-11-18 00:08:53 +08:00
b477839bce
[enhancement](jdbc catalog) Add lowercase column name mapping to Jdbc data source & optimize database and table mapping ( #27124 )
...
This PR adds the processing of lowercase Column names in Oracle Jdbc Catalog. In the previous behavior, we changed all Oracle columns to uppercase queries by default, but could not handle the lowercase case. This PR can solve this situation and improve All Jdbc Catalog works
2023-11-17 23:51:47 +08:00
52995c528e
[fix](iceberg) iceberg use customer method to encode special characters of field name ( #27108 )
...
Fix two bugs:
1. Missing column is case sensitive, change the column name to lower case in FE for hive/iceberg/hudi
2. Iceberg use custom method to encode special characters in column name. Decode the column name to match the right column in parquet reader.
2023-11-17 18:38:55 +08:00
0ece18d6cd
[FIX](regresstest) fix test_map_nested_array csv file for id( #27105 )
2023-11-17 04:20:02 -06:00
a0661ed9d2
[Fix](multi-catalog) Fix complex type crash when using dict filter facility in the parquet-reader. ( #27151 )
...
- Fix complex type crash when using the dict filter facility in the parquet-reader by turning off the dict filter facility in this case.
- Add orc complex types regression test.
2023-11-17 13:43:58 +08:00
c7d961cb11
[regression test](stream load) add case for strict_mode=true and max_filter_ratio=0.5 ( #27125 )
2023-11-17 13:39:01 +08:00
ee08958526
[regression test](http_stream) case for timezone ( #27149 )
...
It does not work now, anyway we need a case.
2023-11-17 13:36:41 +08:00
4fff9a5937
[Improvement](inverted index) delay inverted index col read to reduce IO ( #26080 ) ( #26337 )
2023-11-17 13:12:12 +08:00
4ac460af28
[decimal](tests) add test case for least/greatest for decimalv3 type ( #26930 )
2023-11-17 12:09:59 +08:00
91af86bc78
[fix](function) fix error when use negative number in explode_numbers #27020
2023-11-17 12:02:14 +08:00
334260dff7
[feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton ( #25510 )
2023-11-17 10:27:07 +08:00
a510b5be81
[regression](delete) add regression test for every type delete ( #26954 )
2023-11-16 08:03:31 -06:00
54989175fb
[case] Load json data with enable_simdjson_reader=false ( #26601 )
2023-11-16 14:40:59 +08:00
230d8af777
[regression test](temporary_partitions) add case for temporary_partitions #27063
2023-11-16 11:49:37 +08:00
3ad865fef9
[refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits ( #26191 )
2023-11-15 21:34:49 +08:00
00896d8954
[fix](agg) fix coredump of multi distinct of decimal128I ( #27014 )
...
* [fix](agg) fix coredump of multi distinct of decimal128
* fix
2023-11-15 17:37:20 +08:00
2f529c1c7b
[feature](Nereids): remove True in Join condition ( #26951 )
...
Remove `True` in Join Condition like `SELECT * FROM t1 JOIN t2 on True`;
2023-11-15 15:58:33 +08:00
5dbc3cbba4
[test](information_schema)append information_schema external_table_p0 case. ( #26846 )
2023-11-15 14:30:16 +08:00
2c6d2255c3
[fix](Nereids) nested type literal type coercion and insert values with map ( #26669 )
2023-11-14 21:13:26 -06:00
df867a1531
[fix](catalog) Fix ClickHouse DataTime64 precision parsing ( #26977 )
2023-11-15 10:23:21 +08:00
cdc5626930
[regression](partial update) Add row store cases for all existing partial update cases ( #26924 )
2023-11-15 08:20:55 +08:00
add160b768
[improvement](regression-test) add more group commit regression-test ( #26952 )
2023-11-15 00:01:13 +08:00
88d909b4dd
[test](regression) Add more alter stmt regression case ( #26988 )
2023-11-14 23:58:15 +08:00
f1169d3c58
[regression-test](TRIM_DOUBLE_QUOTES) add case for TRIM_DOUBLE_QUOTES ( #26998 )
2023-11-14 23:52:40 +08:00
9a4fd5be79
[nereids](datetime) fix wrong result type of datetime add with interval as first arg ( #26957 )
...
Incorrect result data type cause be cordump:
drop table if exists testaaa;
create table testaaa(k1 tinyint, k2 smallint, k3 int, k4 bigint, k5 decimal(9,3), k6 char(5), k10 date, k11 datetime, k7 varchar(20), k8 double max, k9 float sum) engine=olap distributed by hash(k1) buckets 5 properties("storage_type"="column","replication_num"="1") ;
insert into testaaa values(1,1,1,1,9.3, "k6", "2023-11-14", "2023-11-14", "k7", 9.99, 9.99);
select interval 10 year + k10 from testaaa;
The plan result type is DATE:
mysql [test]>explain verbose select interval 10 year + k10 from testaaa;
+-------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| years_add(k10, INTERVAL 10 YEAR)[#11 ] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCAL |
| |
| 64:VEXCHANGE |
| offset: 0 |
| tuple ids: 1N |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: HASH_PARTITIONED: k1[#0 ] |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 64 |
| UNPARTITIONED |
| |
| 58:VOlapScanNode |
| TABLE: default_cluster:test.testaaa(testaaa), PREAGGREGATION: OFF. Reason: No aggregate on scan. |
| partitions=1/1 (testaaa), tablets=5/5, tabletList=945025,945027,945029 ... |
| cardinality=1, avgRowSize=9885.0, numNodes=1 |
| pushAggOp=NONE |
| projections: years_add(k10[#6 ], INTERVAL 10 YEAR) |
| project output tuple id: 1 |
| tuple ids: 0 |
| |
| Tuples: |
| TupleDescriptor{id=0, tbl=testaaa, byteSize=8} |
| SlotDescriptor{id=6, col=k10, colUniqueId=6, type=DATEV2, nullable=true, isAutoIncrement=false} |
| |
| TupleDescriptor{id=1, tbl=testaaa, byteSize=32} |
| SlotDescriptor{id=11, col=null, colUniqueId=null, type=DATE, nullable=true, isAutoIncrement=false} |
+-------------------------------------------------------------------------------------------------------+
39 rows in set (1 min 31.50 sec)
coredump stack:
F1109 20:11:37.677680 323805 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVector
*** Check failure stack trace: ***
F1109 20:11:37.680608 323800 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVector
*** Check failure stack trace: ***
F1109 20:11:37.680608 323800 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVectorF1109 20:11:37.681102 323808 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVector
*** Check failure stack trace: ***
@ 0x56489d591d3d google::LogMessage::Fail()
@ 0x56489d591d3d google::LogMessage::Fail()
@ 0x56489d591d3d google::LogMessage::Fail()
@ 0x56489d594279 google::LogMessage::SendToLog()
@ 0x56489d594279 google::LogMessage::SendToLog()
@ 0x56489d594279 google::LogMessage::SendToLog()
@ 0x56489d5918a6 google::LogMessage::Flush()
@ 0x56489d5918a6 google::LogMessage::Flush()
@ 0x56489d5918a6 google::LogMessage::Flush()
@ 0x56489d5948e9 google::LogMessageFatal::~LogMessageFatal()
@ 0x56489d5948e9 google::LogMessageFatal::~LogMessageFatal()
@ 0x56489d5948e9 google::LogMessageFatal::~LogMessageFatal()
@ 0x56487a2a8a0c assert_cast<>()
@ 0x56487a2a8a0c assert_cast<>()
@ 0x56487a2a8a0c assert_cast<>()
@ 0x5648893d8312 doris::vectorized::ColumnVector<>::insert_range_from()
@ 0x5648893d8312 doris::vectorized::ColumnVector<>::insert_range_from()
@ 0x5648893d8312 doris::vectorized::ColumnVector<>::insert_range_from()
@ 0x56488924a670 doris::vectorized::ColumnNullable::insert_range_from()
@ 0x56488924a670 doris::vectorized::ColumnNullable::insert_range_from()
@ 0x56488924a670 doris::vectorized::ColumnNullable::insert_range_from()
@ 0x56487a454475 doris::ExecNode::do_projections()
@ 0x56487a454475 doris::ExecNode::do_projections()
@ 0x56487a454475 doris::ExecNode::do_projections()
@ 0x56487a454b89 doris::ExecNode::get_next_after_projects()
@ 0x56487a454b89 doris::ExecNode::get_next_after_projects()
*** Query id: a467995b35334741-b625042f56495aaf ***
*** tablet id: 0 ***
*** Aborted at 1699531898 (unix time) try "date -d @1699531898" if you are using GNU date ***
*** Current BE git commitID: 0d83327a7c ***
*** SIGABRT unknown detail explain (@0x190d64) received by PID 1641828 (TID 1642168 OR 0x7f6ff96c0700) from PID 1641828; stack trace: ***
@ 0x556ca2a3ab8f std::_Function_handler<>::_M_invoke()
@ 0x556c9f322787 std::function<>::operator()()
@ 0x556ca29da0b0 doris::Thread::supervise_thread()
@ 0x556c9f322787 std::function<>::operator()()
@ 0x7f71b9c38609 start_thread
@ 0x556ca29da0b0 doris::Thread::supervise_thread()
@ 0x7f71b9c38609 start_thread
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/signal_handler.h:417
1# 0x00007F71B9E09090 in /lib/x86_64-linux-gnu/libc.so.6
2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
4# 0x0000556CC51F3729 in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
5# 0x0000556CC51E8D3D in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
6# google::LogMessage::SendToLog() in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
7# google::LogMessage::Flush() in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
9# doris::vectorized::ColumnVector const& assert_cast const&, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&) in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
10# doris::vectorized::ColumnVector::insert_range_from(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/vec/columns/column_vector.cpp:354
11# doris::vectorized::ColumnNullable::insert_range_from(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/vec/columns/column_nullable.cpp:289
12# doris::ExecNode::do_projections(doris::vectorized::Block*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/exec/exec_node.cpp:573
13# doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*, std::function const&, bool) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/exec/exec_node.cpp:592
14# doris::pipeline::SourceOperator::get_block(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/pipeline/exec/operator.h:413
15# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/pipeline/pipeline_task.cpp:259
16# doris::pipeline::TaskScheduler::_do_work(unsigned
2023-11-14 20:28:41 +08:00
3585c7e216
[test](parquet)append parquet reader byte_array_decimal and rle_bool case ( #26751 )
2023-11-14 15:05:10 +08:00
a16517a061
[test](tvf) append tvf read hive_text file regression case. ( #26790 )
2023-11-14 15:03:19 +08:00
23e2bded1a
[fix](Nereids) column pruning under union broken unexpectedly ( #26884 )
...
introduced by PR #24060
2023-11-14 00:39:32 -06:00
f1d90ffc4e
[regression](nereids) add test case for partition prune ( #26849 )
...
* list selected partition name in explain
* add prune partition test case （multi-range key）
2023-11-14 11:51:32 +08:00
0a9d71ebd2
[Fix](Planner) fix varchar does not show real length ( #25171 )
...
Problem:
when we create table with datatype varchar(), we regard it to be max length by default. But when we desc, it does not show
real length but show varchar()
Reason:
when we upgrade version from 2.0.1 to 2.0.2, we support new feature of creating varchar(), and it shows the same way with
ddl schema. So user would confuse of the length of varchar
Solved:
change the showing of varchar() to varchar(65533), which in compatible with hive
2023-11-14 10:49:21 +08:00
cd7ad99de0
[improvement](regression-test) add chunked transfer json test ( #26902 )
2023-11-14 10:31:30 +08:00
fef627c0ba
[Fix](Txn) Fix transaction write to sequence column error ( #26748 )
2023-11-14 10:30:10 +08:00
ec40603b93
[fix](parquet) compressed_page_size has the same meaning in page v1 and v2 ( #26783 )
...
1. Parquet with page v2 is parsed error when using other codec except snappy. Because `compressed_page_size` has the same meaning in page v1 and v2, it always contains the bytes of definition level, repetition level and compressed data.
2. Add regression test for `fix_length_byte_array` stored decimal type, and dictionary encoded date/datetime type.
2023-11-14 08:30:42 +08:00
2853efd4ee
[fix](partial update) Fix NPE when the query statement of an update statement is a point query in OriginPlanner ( #26881 )
...
close #26882
We should not use the singleNodePlan to generate the rootPlanFragment if the query is inside a insert statement or distributedPlanner will be null.
introduced in #15491
2023-11-13 22:08:06 +08:00
4ecaa921f9
[regression-test](num_as_string) test num_as_string ( #26842 )
2023-11-13 21:48:13 +08:00
ebc15fc6cc
[fix](transaction) Fix concurrent schema change and txn cause dead lock ( #26428 )
...
Concurrent schema change and txn may cause dead lock. An example:
Txn T commit but not publish;
Run schema change or rollup on T's related partition, add alter replica R;
sc/rollup add a sched txn watermark M;
Restart fe;
After fe restart, T's loadedTblIndexes will clear because it's not save to disk;
T will publish version to all tablet, including sc/rollup's new alter replica R;
Since R not contains txn data, so the T will fail. It will then always waitting for R's data;
sc/rollup wait for txn before M to finish, only after that it will let R copy history data;
Since T's not finished, so sc/rollup will always wait, so R will nerver copy history data;
Txn T and sc/rollup will wait each other forever, cause dead lock;
Fix: because sc/rollup will ensure double write after the sched watermark M, so for finish transaction, when checking a alter replica:
if txn id is bigger than M, check it just like a normal replica;
otherwise skip check this replica, the BE will modify history data later.
2023-11-13 21:39:28 +08:00
2f32a721ee
[refactor](jni) unified jni framework for jdbc catalog ( #26317 )
...
This commit overhauls the JDBC connector logic within our project, transitioning from the previous mechanism of fetching data through JNI calls for individual ResultSet items to a more efficient and unified approach using the VectorTable data structure.
2023-11-13 14:28:15 +08:00
7e62c3c2de
[fix](Nereids) store user variable in connect context ( #26655 )
...
1.user variable should be case insensitive
2.user variable should be cleared after the connection reset
2023-11-13 12:25:08 +08:00
