26be313d40
[mv](nereids) mv cost related PRs ( #35652 #35701 #35864 #36368 #36789 #34970 ) ( #37097 )
...
## Proposed changes
pick from #35652 #35701 #35864 #36368 #36789 #34970
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-04 09:42:11 +08:00
077fda4259
[enhance](mtmv)show create materialized view ( #36188 ) ( #37125 )
...
pick: https://github.com/apache/doris/pull/36188
2024-07-03 22:48:43 +08:00
69aebc2d25
[branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067 " ( #37226 )
...
## Proposed changes
picks https://github.com/apache/doris/pull/37067
2024-07-03 22:42:51 +08:00
70e1c563b3
[Chore](runtime-filter) enlarge sync filter size rpc timeout limit ( #37103 ) ( #37225 )
...
pick from #37103
2024-07-03 21:02:26 +08:00
ffc57c9ef4
[Bug](runtime-filter) fix brpc ctrl use after free ( #37223 )
...
part of #35186
2024-07-03 21:01:50 +08:00
97945af947
[fix](merge-on-write) when full clone failed, duplicate key might occur ( #37001 ) ( #37229 )
...
cherry-pick #37001
2024-07-03 19:48:10 +08:00
a9f9113c48
[branch-2.1][test](external)move hive cases from p2 to p0 ( #37149 )
...
pk (#36855 )
test_hive_same_db_table_name
test_hive_special_char_partition
test_complex_types
test_wide_table
2024-07-03 19:44:52 +08:00
84f5bb73da
[refactor](nereids) refactor analyze view ( #37106 ) ( #37163 )
...
The Analyzer of NereidsPlanner use different rules to analyze normal plan and view, to prevent the plans in the views analyze multiple times, because some rules can not apply multiple times, say, decimal type coercion, if this rule applied multiple times, it will generate a wrong result.
But this design is trick. Normally, after process the LogicalView, the whole plan tree in the LogicalView should not contains unbound plan, but the current situation is not like this. And this problem block development of some rules, so I refactor it:
1. the Analyzer will not traverse the children of the LogicalView
2. After link the LogicalView to the outer plan tree, the whole plan tree of the LogicalView will not contains unbound plan
3. analyze view and table use the same rules, keep it simple
2024-07-03 19:09:49 +08:00
45fc1c7182
[opt](hive) save hive table schema in transaction for 2.1 ( #37127 )
...
## Proposed changes
pick #37008
2024-07-03 17:32:58 +08:00
e5695e058f
[test](migrate) move 2 cases from p2 to p0 ( #36935 ) ( #37200 )
...
bp #36935
Co-authored-by: zhangdong <493738387@qq.com >
2024-07-03 17:29:01 +08:00
b3f2bd20e3
[feat](nereids) support explain delete from clause #36782 ( #37100 )
...
## Proposed changes
pick from #36782
support explain like:
explain delete from T where A=1
Issue Number: close #xxx
<!--Describe your changes.-->
(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-03 15:08:08 +08:00
5969d6521f
[branch-2.1](function) fix nereids fold constant wrong result of abs ( #37065 ) ( #37108 )
...
pick https://github.com/apache/doris/pull/37065
2024-07-03 11:58:06 +08:00
fb642d0227
[Fix](hive-writer) Fixed the issue where uncompletedMpuPendingUploads did not remove objects correctly. ( #37173 )
...
Backport #36905 .
2024-07-03 11:09:46 +08:00
e857680661
[Migrate-Test](multi-catalog) Migrate p2 tests from p2 to p0. ( #37175 )
...
Backport #36989 .
2024-07-03 11:08:49 +08:00
4761090848
[fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF ( #37161 )
...
bp: #35489
2024-07-03 10:58:08 +08:00
17056dcdb9
[Fix](multi-catalog) Fix the transaction is not removed in abnormal situations by removing transaction in finally block. ( #37172 )
...
Backport #36705 .
2024-07-03 10:56:41 +08:00
0aeb768bf9
[Fix](export/outfile) Support compression when exporting data to Parquet / ORC. ( #37167 )
...
bp: #36490
2024-07-03 10:53:57 +08:00
6804ae5827
[enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change ( #36513 ) ( #37122 )
...
pick from https://github.com/apache/doris/pull/36513
2024-07-03 10:42:03 +08:00
03942f23f6
[fix](mtmv)fix when related table drop partition,mv partition is sync… ( #36602 )
...
pick: https://github.com/apache/doris/pull/36547
2024-07-02 23:08:58 +08:00
5bb6642b86
[fix](mtmv)mtmv support default key ( #36221 ) ( #36601 )
...
pick : https://github.com/apache/doris/pull/36221
2024-07-02 23:08:38 +08:00
fbca3196c5
[enhance](mtmv) not allow modify data of MTMV ( #35870 ) ( #37129 )
...
pick: https://github.com/apache/doris/pull/35870
2024-07-02 23:06:45 +08:00
0a1abf10d6
[fix](auth)support check priv when tvf use resource ( #36928 ) ( #37132 )
...
pick https://github.com/apache/doris/pull/36928
2024-07-02 23:03:29 +08:00
177764647d
[fix](mtmv)fix mtmv dead lock ( #37009 ) ( #37133 )
...
pick https://github.com/apache/doris/pull/37009
2024-07-02 23:01:31 +08:00
6716f9e736
[fix](mtmv)fix MTMVTask log has null pointer ( #36840 ) ( #37131 )
...
pick https://github.com/apache/doris/pull/36840
2024-07-02 23:01:15 +08:00
65375b48fc
[enhance](mtmv)support replace materialized view ( #36749 ) ( #37147 )
...
pick: https://github.com/apache/doris/pull/36749
2024-07-02 23:00:55 +08:00
b3eaf0e4d2
[bugfix](hive)Prevent multiple fs from being generated for 2.1 ( #37142 )
...
pick #36954
2024-07-02 22:54:40 +08:00
64a79cc03a
[bugfix](testcase)add java error log output for 2.1 ( #37128 )
...
pick #35998
2024-07-02 22:53:38 +08:00
e7e1e967cf
[test](migrate) move 2 cases from p2 to p0 for 2.1 ( #37139 )
...
pick #37004
2024-07-02 22:50:53 +08:00
b445c783eb
[test](tvf) move p2 tvf tests from p2 to p0 ( #37081 ) ( #37152 )
...
bp: #37081
2024-07-02 22:38:22 +08:00
74086189d3
[test](tvf) move p2 tvf tests from p2 to p0 ( #36871 ) ( #37150 )
...
bp: #36871
2024-07-02 22:37:43 +08:00
bd24a8bdd9
[Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values ( #37153 )
...
bp: #36668
2024-07-02 22:12:17 +08:00
e25717458e
[opt](catalog) add some profile for parquet reader and change meta cache config ( #37040 ) ( #37146 )
...
bp #37040
2024-07-02 20:58:43 +08:00
c7549acf29
[Fix](Nereids) fix leading with different be instance number ( #36613 ) ( #36967 )
...
cherry-pick: #36613
Problem:
When use different be number to test leading explain shape plan,
physical distribute plan would differ due to different be numbers
Solved:
Disable physical distribute node showing in fix leading cases
2024-07-02 19:56:17 +08:00
42f4271e9d
[enhancement](nereids) speedup sql cache with variable ( #37090 ) ( #37119 )
...
cherry pick from #37090
2024-07-02 19:25:22 +08:00
f5572ac732
[pick]reset memtable flush thread num ( #37092 )
...
## Proposed changes
pick #37028
2024-07-02 19:20:17 +08:00
b63e6ac350
[fix](ES Catalog)Add array types support in esquery function ( #36936 ) ( #37054 )
...
backport #36936
2024-07-02 17:48:35 +08:00
239bc1a7e0
[fix](compile) fix compile failed on MacOS due to ambiguous std::abs ( #37136 )
...
cherry-pick #35125 to branch-2.1
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com >
2024-07-02 17:45:33 +08:00
f5d0cdeeb4
[branch-2.1] avoid glog coredump when running with ASAN ( #37134 )
...
## Proposed changes
This is just a workround try avoid coredump like this:
```
#0 0x56414f0e8ed1 in __asan::CheckUnwind() crtstuff.c
#1 0x56414f1009a2 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) crtstuff.c
#2 0x56414f0ecbf3 in __asan::AsanThread::GetStackFrameAccessByAddr(unsigned long, __asan::AsanThread::StackFrameAccess*) crtstuff.c
#3 0x56414f050d87 in __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, bool) crtstuff.c
#4 0x56414f052a73 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long) crtstuff.c
#5 0x56414f0e6a9e in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) crtstuff.c
#6 0x56414f066885 in gmtime_r (/mnt/hdd01/ci/branch21-deploy/be/lib/doris_be+0x17ef3885) (BuildId: f58eb5e327529636)
#7 0x564177940521 in google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) crtstuff.c
#8 0x564151de36fc in doris::Status doris::ThriftRpcHelper::rpc(std::__cxx11::basic_string, std::allocator> const&, int, std::function&)>, int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_rpc_helper.cpp:76:13
#9 0x56417603cda7 in doris::vectorized::VRowDistribution::automatic_create_partition() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/vrow_distribution.cpp:99:5
#10 0x56417614cffa in doris::vectorized::VTabletWriter::_send_new_partition_batch() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/writer/vtablet_writer.cpp:1346:9
....
```
2024-07-02 17:45:04 +08:00
cf86eb8647
[test](migrate) move test_hive_text_complex_type from p2 to p0 ( #37007 ) ( #37123 )
...
bp: #37007
2024-07-02 17:36:37 +08:00
d0eea3886d
[fix](multi-catalog) Revert #36575 and check nullptr of data column ( #37086 )
...
Revert #36575 , because `VScanner::get_block` will check
`DCHECK(block->rows() == 0)`, so block should be cleared when `eof =
true`.
2024-07-02 15:32:52 +08:00
7443e8fcf2
[cherry-pick](branch-2.1) fix single compaction test p2 #34568 #36881 ( #37075 )
2024-07-02 15:22:04 +08:00
859a7c80b8
[fix](parquet) prevent parquet page reader print much warning logs ( #37012 )
...
bp #37011
2024-07-02 14:33:01 +08:00
496024ffb5
[fix](mtmv) mtmv add case and fix bug ( #36019 ) ( #36600 )
...
pick from: #36019
2024-07-02 14:24:53 +08:00
ebc1afed4a
[enhance](mtmv)when calculating the availability of MTMV, no longer c… ( #36617 )
...
pick: #36507
2024-07-02 14:24:33 +08:00
0c8934642e
[opt](nereids)using mv's derived stats #35721 ( #37098 )
...
pick from #35721
(cherry picked from commit a481d79b1e258f7b6711157bb0d5e4c224e680e5)
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-02 13:54:05 +08:00
c66975b244
[branch-2.1] Pick "[Fix](regression) Fix p0 case test_unique_table_auto_inc_concurrent #37048 " ( #37050 )
...
## Proposed changes
pick https://github.com/apache/doris/pull/37048
2024-07-02 11:15:50 +08:00
a9764b9d39
[chore](multi catalog) Print serde properties when show create hive-external-table ( #34966 ) ( #37096 )
...
bp #34966
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com >
2024-07-02 11:07:10 +08:00
6789f5bc80
[fix](null safe equal join) fix coredump if both sides of the conjunct is not nullable #36263 ( #37073 )
2024-07-02 11:01:55 +08:00
011f203d71
[opt](log) Remove unnecessary warning log ( #37093 )
...
When enable_profile = true or report_succeed=true, it is very likely
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java::reportExecStatus
will print much warning log. It is not necesary.
2024-07-02 10:53:36 +08:00
3f382b797a
[branch-2.1][improvement](sqlserver catalog) Configurable whether to use encrypt when connecting to SQL Server using the catalog ( #36971 )
...
pick (#36659 )
pick #37015
In previous versions, we used druid as the default JDBC connection pool,
which can use custom decryption to parse the certificate when SQL Server
encryption is turned on. However, in the new version, after changing
HikariCP as the default connection pool, the SQLServer certificate
cannot be parsed, so encryption needs to be turned off for normal use.
Therefore, a parameter is added to decide whether to disable SQLServer
encryption. It is not disabled by default.
2024-07-02 10:14:43 +08:00
