Commit Graph

14536 Commits

Author SHA1 Message Date
9243de1898 [fix](planner) add IP type to compatibilityMatrix (#26095) 2023-11-03 01:19:22 -05:00
0f24500ff8 [fix](Nereids) RewriteCteChildren not work with cost based rewritter (#26326)
we use a map to record rewrite cte children result to avoid rewrite
twice in cost based rewritter. However, we record cte outer and
inner in one map, and use null as outer result's key, use cte id as
inner result's key. This is wrong, because every anchor has an outer,
and we could only record one outer. So when we use the cache in cost
based rewritter, we get wrong outer plan from the cache. Then the error
will be thrown as below:

```
Caused by: java.lang.IllegalArgumentException: Stats for CTE: CTEId#1 not found
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143) ~[guava-32.1.2-jre.jar:?]
    at org.apache.doris.nereids.stats.StatsCalculator.visitLogicalCTEConsumer(StatsCalculator.java:1049) ~[classes/:?]
    at org.apache.doris.nereids.stats.StatsCalculator.visitLogicalCTEConsumer(StatsCalculator.java:147) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer.accept(LogicalCTEConsumer.java:111) ~[classes/:?]
    at org.apache.doris.nereids.stats.StatsCalculator.estimate(StatsCalculator.java:222) ~[classes/:?]
    at org.apache.doris.nereids.stats.StatsCalculator.estimate(StatsCalculator.java:200) ~[classes/:?]
    at org.apache.doris.nereids.jobs.cascades.DeriveStatsJob.execute(DeriveStatsJob.java:108) ~[classes/:?]
    at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:39) ~[classes/:?]
    at org.apache.doris.nereids.jobs.executor.Optimizer.execute(Optimizer.java:51) ~[classes/:?]
    at org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJob.getCost(CostBasedRewriteJob.java:98) ~[classes/:?]
    at org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJob.execute(CostBasedRewriteJob.java:64) ~[classes/:?]
    at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:119) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visit(RewriteCteChildren.java:72) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visit(RewriteCteChildren.java:56) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.visitor.PlanVisitor.visitLogicalSink(PlanVisitor.java:118) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.visitor.SinkVisitor.visitLogicalResultSink(SinkVisitor.java:72) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalResultSink.accept(LogicalResultSink.java:58) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:86) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:56) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:60) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:86) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:56) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:60) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.rewriteRoot(RewriteCteChildren.java:67) ~[classes/:?]
    at org.apache.doris.nereids.jobs.rewrite.CustomRewriteJob.execute(CustomRewriteJob.java:58) ~[classes/:?]
    at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:119) ~[classes/:?]
    at org.apache.doris.nereids.NereidsPlanner.rewrite(NereidsPlanner.java:275) ~[classes/:?]
    at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:218) ~[classes/:?]
    at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:118) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:81) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:550) ~[classes/:?]
```
2023-11-03 01:12:31 -05:00
6892fc99f8 [fix](Nereids) only search internal funcftion when dbName is empty (#26296)
if call function with database name. we should only search UDF
2023-11-03 01:01:25 -05:00
9019a71770 [Fix](multi-catalog) Filter _temporary dir which is generated by spark. (#26194)
Sometimes there are some temporary files which are generated by some spark jobs, we need filter these files.

```
errCode = 2, detailMessage = (xxx.xxx.xxx.xxx.xxx)
[CANCELLED][INTERNAL_ERROR]failed to init reader for file hdfs://xxx/user/hive/warehouse/xxx.db/xxx/_temporary/0/_temporary/attempt_202311010952379045223848154629697_0013_m_000442_3355/part-00442-3f97ac18-0583-4d2c-8995-8d21ceda56ec-c000.snappy.orc, 
err: [INTERNAL_ERROR]Init OrcReader failed. reason = Failed to parse the postscript from hdfs://xxx/user/hive/warehouse/xxx.db/xxx/_temporary/0/_temporary/attempt_202311010952379045223848154629697_0013_m_000442_3355/part-00442-3f97ac18-0583-4d2c-8995-8d21ceda56ec-c000.snappy.orc   
0.  @ 0x000000000bdbd89f
   doris::Status doris::Status::Error<6, true, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)  /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  1.  @ 0x000000000eccecfe
   doris::vectorized::OrcReader::_create_file_reader()  /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  2.  @ 0x000000000ecceff9
   doris::vectorized::OrcReader::init_reader(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, doris::ColumnValueRange<(doris::PrimitiveType)4>, doris::ColumnValueRange<(doris::PrimitiveType)5>, doris::ColumnValueRange<(doris::PrimitiveType)6>, doris::ColumnValueRange<(doris::PrimitiveType)7>, doris::ColumnValueRange<(doris::PrimitiveType)15>, doris::ColumnValueRange<(doris::PrimitiveType)10>, doris::ColumnValueRange<(doris::PrimitiveType)23>, doris::ColumnValueRange<(doris::PrimitiveType)11>, doris::ColumnValueRange<(doris::PrimitiveType)25>, doris::ColumnValueRange<(doris::PrimitiveType)12>, doris::ColumnValueRange<(doris::PrimitiveType)26>, doris::ColumnValueRange<(doris::PrimitiveType)20>, doris::ColumnValueRange<(doris::PrimitiveType)2>, doris::ColumnValueRange<(doris::PrimitiveType)19>, doris::ColumnValueRange<(doris::PrimitiveType)28>, doris::ColumnValueRange<(doris::PrimitiveType)29>, doris::ColumnValueRange<(doris::PrimitiveType)30> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, doris::ColumnValueRange<(doris::PrimitiveType)4>, doris::ColumnValueRange<(doris::PrimitiveType)5>, doris::ColumnValueRange<(doris::PrimitiveType)6>, doris::ColumnValueRange<(doris::PrimitiveType)7>, doris::ColumnValueRange<(doris::PrimitiveType)15>, doris::ColumnValueRange<(doris::PrimitiveType)10>, doris::ColumnValueRange<(doris::PrimitiveType)23>, doris::ColumnValueRange<(doris::PrimitiveType)11>, doris::ColumnValueRange<(doris::PrimitiveType)25>, doris::ColumnValueRange<(doris::PrimitiveType)12>, doris::ColumnValueRange<(doris::PrimitiveType)26>, doris::ColumnValueRange<(doris::PrimitiveType)20>, doris::ColumnValueRange<(doris::PrimitiveType)2>, doris::ColumnValueRange<(doris::PrimitiveType)19>, doris::ColumnValueRange<(doris::PrimitiveType)28>, doris::ColumnValueRange<(doris::PrimitiveType)29>, doris::ColumnValueRange<(doris::PrimitiveType)30> > > > >*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, bool, doris::TupleDescriptor const*, doris::RowDescriptor const*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const*, std::unordered_map<int, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > > > > const*)  /root/doris-master/be/src/common/status.h:448  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  3.  @ 0x000000000ff43da0
   doris::vectorized::VFileScanner::_get_next_reader()  /root/doris-master/be/src/vec/exec/scan/vfile_scanner.cpp:800  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  4.  @ 0x000000000ff4256f
   doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*)  /root/doris-master/be/src/common/status.h:448  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  5.  @ 0x000000000ffdb1aa
   doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*)  /root/doris-master/be/src/vec/exec/scan/vscanner.cpp:0  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  6.  @ 0x000000000ff3b379
   doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>)  /root/doris-master/be/src/common/status.h:346  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  7.  @ 0x000000000ff3ca11  std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::{lambda()#4}>::_M_invoke(std::_Any_data const&)  /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  8.  @ 0x000000000c986608
   doris::ThreadPool::dispatch_thread()  /root/doris-master/be/src/util/threadpool.cpp:0  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  9.  @ 0x000000000c97c2ac
   doris::Thread::supervise_thread(void*)  /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  10. @ 0x0000000000007ea5  start_thread  in /usr/lib64/libpthread-2.17.so  11. @ 0x00000000000feb0d  __clone  in /usr/lib64/libc-2.17.so    0.  @ 0x000000000c647366
   doris::Status doris::Status::Error<6, true, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)  /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  1.  @ 0x000000000ff42dc3
   doris::vectorized::VFileScanner::_get_next_reader()  /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  2.  @ 0x000000000ff4256f
   doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*)  /root/doris-master/be/src/common/status.h:448  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  3.  @ 0x000000000ffdb1aa
   doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*)  /root/doris-master/be/src/vec/exec/scan/vscanner.cpp:0  in /hdpData10/yarn/localcache/usercache/hive/appcache/application_1696938614818_1832823/filecache/13/doris-be-360-master.tar.gz/apache-doris-be-360-master-bin-x86_64/lib/doris_be  4.  @ 0x000000000ff3b379
   doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>)  /root/doris-master/be/src/common/status.h:346  in /hdpData10/yarn/localcache/usercache/hive/appcache/a…
```
Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
2023-11-03 13:42:29 +08:00
3d403b061c [test](coverage) Improve test coverage for runtime filter (#26314) 2023-11-03 12:17:03 +08:00
b19f275714 [improvement](insert) refactor group commit insert into (#25795) 2023-11-03 12:02:40 +08:00
61edb9e4d8 [feature](mtmv)(2)Implementing mtmv using antlr (#26102)
Implementing mtmv using antlr,

No specific business logic was implemented, therefore an exception was thrown during execution
```
throw new AnalysisException("current not support.");
```
2023-11-03 11:57:00 +08:00
509723c1d6 [bug](bitmap) fix bitmap value not clear get wrong data (#26321) 2023-11-03 10:25:13 +08:00
03ab822ba6 [fix](backup) support aws_token for backup/restore (#26242) 2023-11-03 10:18:37 +08:00
91946dc8b0 [minor](serde) 'null' in the serde should be represented by constants, instead of a string variable (#26301) 2023-11-02 23:13:10 +08:00
93a934e775 [Improve](map) support map cast with map literal and implicate nested scala cast (#26126) 2023-11-02 09:56:42 -05:00
3f47eb5e89 [Fix](inverted index) fix compound query result error when disable inverted_index_query session variable (#26327) 2023-11-02 08:36:08 -05:00
e307884e57 [enhancement](Nereids): GroupPlan don't generate ObjectId (#26315) 2023-11-02 21:35:48 +08:00
8bd06aff7e [Chore](MoW) remove unused code about rowset tree (#26282) 2023-11-02 20:25:27 +08:00
d9621404b3 [improvement](fe) Add more log about add/remove frontends (#26306) 2023-11-02 20:19:51 +08:00
c098b0393a [test](regression) Add backup/restore DB case (#26312) 2023-11-02 20:18:45 +08:00
a5ef90dacc [enhancement](recover) support skipping missing version in select by session variable (#25654) 2023-11-02 20:01:51 +08:00
89cf828f03 [enhancement](udf) add the switch enable_java_udf to control the creation and use of java_udf. (#26213) 2023-11-02 19:33:17 +08:00
c46fa339d0 [feature](Nereids): make count() avaliable for all data type (#26180)
make count() avaliable for all data type
2023-11-02 17:20:24 +08:00
6828250207 [profile](refactor) Fix invalid shuffle profile (#26298) 2023-11-02 17:07:05 +08:00
dd8bcc831c [keyword](decimalv2) Add DecimalV2 keyword (#26283) 2023-11-02 16:27:12 +08:00
f1362e0d1e [fix](planner) Fix sample partition table (#25912)
In the past, two conditions needed to be met when sampling a partitioned table: 1. Data is evenly distributed between partitions; 2. Data is evenly distributed between buckets. Finally, the number of sampled rows in each partition and each bucket is the same.

Now, sampling will be proportional to the number of partitioned and bucketed rows.
2023-11-02 15:52:48 +08:00
3e9e8becc6 [fix](regression) Add auto-expired prefix for backup/restore cases (#26272) 2023-11-02 15:21:43 +08:00
Pxl
1f9fdac148 [Chore](case) adjust some mv regression test case (#26279)
adjust some mv regression test case
2023-11-02 14:17:24 +08:00
Pxl
3004dbbc78 [Bug](materialized-view) SelectMaterializedIndexWithAggregate do not … (#26192)
SelectMaterializedIndexWithAggregate do not change plan when match base index
2023-11-02 14:12:22 +08:00
3ce1bdae84 [Feature](Nereids) Add explain tree functionality. (#25446)
Add explain tree functionality which contains fragment ids for profile analysis.
```
mysql> explain tree select sum(l_orderkey) from lineitem, orders where l_orderkey = o_orderkey;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| [414]:[414: ResultSink]||[Fragment: 0]||VRESULT SINK||   MYSQL_PROTOCAL||                                                                                                                                         |
| --[414]:[414: VAGGREGATE (merge finalize)]||[Fragment: 0]||cardinality=1||                                                                                                                                        |
| ----[411]:[411: VEXCHANGE]||[Fragment: 0]||offset: 0||                                                                                                                                                            |
| ------[411]:[411: DataStreamSink]||[Fragment: 1]||STREAM DATA SINK||  EXCHANGE ID: 411||  UNPARTITIONED                                                                                                           |
| --------[408]:[408: VAGGREGATE (update serialize)]||[Fragment: 1]||cardinality=1||                                                                                                                                |
| ----------[402]:[402: VHASH JOIN]||[Fragment: 1]||join op: INNER JOIN(PARTITIONED)[]||cardinality=593,132,346||                                                                                                   |
| ------------[392]:[392: VEXCHANGE]||[Fragment: 1]||offset: 0||                                                                                                                                                    |
| --------------[392]:[392: DataStreamSink]||[Fragment: 2]||STREAM DATA SINK||  EXCHANGE ID: 392||  HASH_PARTITIONED                                                                                                |
| ----------------[386]:[386: VHIVE_SCAN_NODE]||[Fragment: 2]||table: lineitem||inputSplitNum=144, totalFileSize=16632158739, scanRanges=144||partition=1/1||cardinality=600037902, numNodes=1||pushdown agg=NONE|| |
| ------------[399]:[399: VEXCHANGE]||[Fragment: 1]||offset: 0||                                                                                                                                                    |
| --------------[399]:[399: DataStreamSink]||[Fragment: 3]||STREAM DATA SINK||  EXCHANGE ID: 399||  HASH_PARTITIONED                                                                                                |
| ----------------[393]:[393: VHIVE_SCAN_NODE]||[Fragment: 3]||table: orders||inputSplitNum=48, totalFileSize=3984213353, scanRanges=48||partition=1/1||cardinality=150000000, numNodes=1||pushdown agg=NONE||      |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
12 rows in set (0.04 sec)
```
2023-11-02 14:06:24 +08:00
fb622800aa [fix](mtmv)Compatible old mtmv metadata (#26193)
make sure user can upgrade from v2.0 to master
2023-11-02 12:04:04 +08:00
b1f3d4f15a exclude case test_jdbc_query_mysql (#26073) 2023-11-02 09:26:42 +08:00
80f654ec2a [Fix](statistics)Fix analyze min max sql syntax error. #26240 2023-11-02 09:22:32 +08:00
a4e415ab09 [feature](hive)Support hive tables after alter type. (#25138)
1.Reconstruct the logic of decode to read parquet. The parquet  reader first reads the data according to the parquet physical type, and then performs a type conversion.

2.Support hive alter table.
2023-11-02 00:24:21 +08:00
3e10e5af39 [Fix](Serde) Fix content displayed by complex types in MySQL Client (#25946)
This pr makes three changes to the display of complex types:
1. NULL value in complex types refers to being displayed as `null`, not `NULL`
2. struct type is displayed as "column_name": column_value
3. Time types such as `datetime` and `date`, are displayed with double quotes in complex types. like
    `{1, "2023-10-26 12:12:12"}`

This pr also do a code refactor:
1. nesting_level is set to a member variable of the `DataTypeSerDe`, rather than a parameter in methods.

What's more, this pr fix a bug that fileSize is not correct, introduced by this pr: #25854
2023-11-01 23:48:55 +08:00
f8a594c25d fix column is decimalv3 cast to decimalv2 make be core (#26204) 2023-11-01 22:30:30 +08:00
Pxl
d7ea4d31fb [Chore](hash-table) remove unused code about HashTableTraits (#26202)
remove unused code about HashTableTraits
2023-11-01 21:36:50 +08:00
1018f8918c [pipelineX](bug) Fix build side output flag (#26233) 2023-11-01 20:13:41 +08:00
6010be88bd [enhancement](Nereids): optimize GroupExpressionMatching (#26196) 2023-11-01 19:05:08 +08:00
502f5778f4 [enhancement](Nereids): set maxJoinNumBushyTree = 8 (#26173) 2023-11-01 19:04:49 +08:00
aafd53766b [chore](file-reader) rm unused interface from generic reader (#26205) 2023-11-01 18:43:14 +08:00
f0f9c3a160 [improvement](drop tablet) impr gc shutdown tablet lock (#26151) 2023-11-01 18:42:21 +08:00
be7a10162a [enhance](S3) Add timeout for s3 buffer allocation and corresponding observability (#26125) 2023-11-01 17:55:11 +08:00
49439175a0 [fix](move-memtable) init delta writer pool in sink prepare (#26108) 2023-11-01 17:52:10 +08:00
ec85e22506 [enhance](scanner) pass the tablet in NewOlapScanner's ctor (#26167) 2023-11-01 17:50:14 +08:00
67e556af59 [regression](fe) Add more regression test for FE (#26206)
* add add/drop node test
* add metrics test
* add meta_action test
2023-11-01 17:48:58 +08:00
eb9ba59996 [improvement](show trash) Fix be restart slow when too many trash files (#26147) 2023-11-01 17:43:24 +08:00
1c2831e06f [fix](spill) disable spill of sort and agg for now to avoid diisk overflow (#26209) 2023-11-01 17:20:26 +08:00
bfca1bf206 [Enhancement] Implement format methods for Status (#26133) 2023-11-01 16:55:25 +08:00
387e33fa34 [enhancement](group commit)Add group commit block queues memory back pressure (#26045) 2023-11-01 16:29:45 +08:00
683832230c [Performance](Nereids): pass ConnectContext to avoid ThreadLocal.get() (#26165) 2023-11-01 16:16:42 +08:00
1770224322 [fix](planner) Fix decimal precision and scale wrong when create table like (#25802)
Use field datatype such as decimal(10, 0) to create table like. Because the scale is 0, the precision and scale will lost when create table like done. this will fix the bug.

**Before fix, create table with following SQL**:
CREATE TABLE IF NOT EXISTS db_test.table_test
(
    `name` varchar COMMENT "1m size",
    `id` SMALLINT COMMENT "[-32768, 32767]",
    `timestamp0` decimal null comment "c0",
    `timestamp1` decimal(38, 0) null comment "c1"
)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES ('replication_num' = '1');

**and Then run**
CREATE TABLE db_test.table_test_like LIKE db_test.table_test
SHOW CREATE TABLE db_test.table_test_like;

the field `timestamp1` will be decimal(9, 0), it's wrong. this will fix it.
2023-11-01 03:11:43 -05:00
4644191fd0 [fix](broker-read) refactor broker reading process to avoid null broker connection (#26050) 2023-11-01 15:58:30 +08:00
f2874b9452 [bug](shared scan) Fix use-after-free when enable pipeline shared scanning (#26199)
When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
2023-11-01 15:51:20 +08:00