database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,891 Commits 1 Branch 0 Tags
eeacca9e8eaafb4431e11896cbf0be2f0cd60bfd
Commit Graph

3243 Commits

Author SHA1 Message Date
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
seawinde
021982fc71 [fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678) 
## Proposed changes

pr: https://github.com/apache/doris/pull/39041
commitId: 22562985

pr: https://github.com/apache/doris/pull/38958
commitId: c365cb64

pr: https://github.com/apache/doris/pull/39541
commitId: 89bb669c
 2024-08-22 10:27:55 +08:00
wangbo
a55e109e97 [pick][Improment]Add schema table workload_group_privileges (#38436) (#39708) 
pick #38436
 2024-08-22 00:44:43 +08:00
Pxl
63d45f5d89 [Bug](predicate) fix wrong result of AcceptNullPredicate (#39497) (#39672) 
pick from #39497
 2024-08-22 00:24:57 +08:00
Mryange
e51dd68b93 [fix](local shuffle) Fix correctness for bucket hash shuffle exchange… (#39691) 
…r (#39568)

For query plan


![image](https://github.com/user-attachments/assets/334cc4c4-49ae-4330-83ff-03b9bae00e3c)

we will plan local exchangers  and get a new plan


![image](https://github.com/user-attachments/assets/2b8ece64-3aa0-423c-9db0-fd02024957db)

and the hash join operator will get probe and build data which are
different distributed (one is HASH shuffle and another is Bucket hash
shuffle). This PR fix it.
<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
 2024-08-22 00:23:39 +08:00
slothever
ebbebdf590 [regression](kerberos)add hive with kerberos write back case (#39682) 
bp #38647
 2024-08-21 18:29:42 +08:00
meiyi
1460878bdf [fix](cluster key) forbid cluster key and remove case (#39679) 
branch-2.1 does not support mow cluster key
 2024-08-21 14:31:54 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
xzj7019
8a562aeb77 [opt](nereids) recover adoptive bucket shuffle (#39598) 
## Proposed changes

pick from https://github.com/apache/doris/pull/36784

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-21 09:26:53 +08:00
wuwenchi
6df6f1dc97 [improvement](iceberg)]support doris's char/varchar to iceberg's string for 2.1 #38807 (#39645) 
bp: #38807
 2024-08-21 09:19:10 +08:00
daidai
27ba2542e2 [case](iceberg)append iceberg schema change case. (#38766) (#39630) 
bp #38766
 2024-08-21 09:17:12 +08:00
Socrates
bb687bd69c [cherry-pick](branch-2.1) add function regexp_extract_or_null (#39561) 
# Proposed changes

pick https://github.com/apache/doris/pull/38296
 2024-08-21 09:14:58 +08:00
Sun Chenyang
8e9bc7449b [test](inverted index) add test for need read data opt (#38261) (#39534) 
## Proposed changes


pick from master #38261
 2024-08-21 09:01:12 +08:00
zzzxl
273a62584c [opt](inverted index) unified optimization judgment to prevent omissions (#39473) 
https://github.com/apache/doris/pull/38027
 2024-08-17 16:57:19 +08:00
zy-kkk
20936fe054 [branch-2.1][improvement](jdbc catalog) Compatible with ojdbc6 by adding version check (#39408) 
pick (#39341)

In previous versions, we used a method based on JDBC 4.2 to read data,
so it was equivalent to abandoning support for ojdbc6. However, we
recently found that a large number of users still use Oracle version
11g, which will have some unexpected compatibility issues when using
ojdbc8 to connect. Therefore, I use version verification to make it
compatible with both ojdbc6 and ojdbc8, so that good compatibility can
be obtained through ojdbc6, and better reading efficiency can be
obtained through ojdbc8.
 2024-08-17 16:43:01 +08:00
amory
7687f2c53a [fix](ip-funcs) fix ip inet6_aton funcs #39415 (#39513) 2024-08-17 10:56:06 +08:00
zhangdong
ae8073f155 [opt](mtmv) partition rollup support week and quarter (#39286) (#39477) 
pick from master #39286​
 2024-08-16 20:01:06 +08:00
zy-kkk
2948b5ea2b [branch-2.1][fix](jdbc scan) Remove the conjuncts.remove call in JdbcScan (#39407) 
pick (#39180)

In #37565, due to the change in the calling order of finalize, the final
generated Plan will be missing the PREDICATES that have been pushed down
in Jdbc. Although this behavior is correct, before perfectly handling
the push down of various PREDICATES, we need to keep all conjuncts to
ensure that we can still filter data normally when the data returned by
Jdbc is a superset.
 2024-08-16 19:01:40 +08:00
LiBinfeng
4458302a77 [Fix](Planner) fix delete from using does not attach partition information (#39020) (#39418) 
cherry-pick from master #39020
Problem:
when use delete from using clause and assign partition information, it
would delete more data from other partition
Solved:
add partition information when transfer delete clause into insert into
select clause
 2024-08-16 17:16:08 +08:00
lihangyu
824f035b98 [pick](Row store) fix row store with invalid json string in variant ty… (#39456) 
#39394
 2024-08-16 14:43:11 +08:00
morrySnow
d56000e924 [opt](Nereids) polish aggregate function signature matching (#39352) (#39460) 
pick from master #39352

use double to match string
- corr
- covar
- covar_samp
- stddev
- stddev_samp

use largeint to match string
- group_bit_and
- group_bit_or
- group_git_xor

use double to match decimalv3
- topn_weighted

optimize error message
- multi_distinct_sum
- multi_distinct_sum0
 2024-08-16 13:57:11 +08:00
TengJianPing
021678c7c3 [fix](window_funnel) fix wrong result of window_funnel #38954 (#39270) 
## Proposed changes

BP #38954
 2024-08-16 09:59:31 +08:00
morrySnow
4380f3cb51 [fix](variable) support all type functions (#39144) (#39438) 
pick from master #39144
 2024-08-16 09:51:02 +08:00
morrySnow
3aaee8f7d5 [fix](Nereids) polish function signature search algorithm (#38497) (#39436) 
pick from master #38497 and #39342

use array<double> for array<string>
- array_avg
- array_cum_sum
- array_difference
- array_product

use array<bigint> for array<string>
- bitmap_from_array

use double first
- fmod
- pmod

let high order function throw friendly exception
- array_filter
- array_first
- array_last
- array_reverse_split
- array_sort_by
- array_split

let return type same as parameter's type
- array_push_back
- array_push_front
- array_with_constant
- if

let greatest / least work same as mysql's greatest
 2024-08-16 08:24:25 +08:00
amory
6257e706fa [improve](ip)update ip for bloom_filter (#39414) 
## Proposed changes
backport: https://github.com/apache/doris/pull/39253
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-16 08:20:19 +08:00
seawinde
aebc70d75a revert [improvement](mv) Support to use cast when create sync materialized view #38008 (#39378) 
## Proposed changes

this is brought by https://github.com/apache/doris/pull/38008
if use cast(FLOOR(MINUTE(time) / 15) as decimal(9, 0)) in group by
clause when sync materialized view. if downgrade from 2.1.6 to 2.1.5 or
upgrade 2.1.6 to 3.0.0
this may cause fe can not run. So revert the function.
 2024-08-15 14:16:57 +08:00
zclllhhjj
4acd69590d [Fix](function) fix wrong nullable signature of function corr (#39380) 
## Proposed changes

Issue Number: close #xxx

before `corr(nullable_x, nullable_y)` will core dump. not fixed.
no need to patch in master because the refactor
https://github.com/apache/doris/pull/37330 already changed the
implementation context
 2024-08-15 14:10:09 +08:00
amory
1accde9fb3 [fix](nestedtype) support nested type for schema change reorder (#39392) 
## Proposed changes
backport: https://github.com/apache/doris/pull/39210
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-15 14:03:03 +08:00
zy-kkk
c12137a8d6 [branch-2.1][fix](expr) Enhance SQL Expression Handling by Introducing printSqlInParens to CompoundPredicate (#39082) 
pick #39064
 2024-08-14 21:14:58 +08:00
amory
226e01889c [fix](array_apply) pick array apply fix (#39328) 
## Proposed changes
backport: https://github.com/apache/doris/pull/39105
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-14 18:52:29 +08:00
amory
78d6e318fb [fix](ip)pick ip rowstore (#39345) 
## Proposed changes
backport: https://github.com/apache/doris/pull/39258
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-14 18:51:58 +08:00
Mryange
b26af32934 [fix](function) fix error return type in corr(float32,float32) (#39251) (#39350) 
https://github.com/apache/doris/pull/39251
```
mysql [test11]>select corr(cast(x as float),cast(y as float)) from test_corr;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]column_type not match data_types in agg node, column_type=Nullable(Float64), data_types=Nullable(Float32),column name=

```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-14 18:47:14 +08:00
Mryange
a9692a305e [fix](function)timediff with now function causes a error signature (… (#39349) 
…#39322)
https://github.com/apache/doris/pull/39322
## Proposed changes

```
mysql [(none)]>select round(timediff(now(),'2024-08-15')/60/60,2);
ERROR 1105 (HY000): errCode = 2, detailMessage = argument 1 requires datetimev2 type, however 'now()' is of datetime type
```
The reason is that the function parameter types were modified in
expectedInputTypes, which led to no match being found. The code here is
from a long time ago. Because the precision of datetimev2 could not be
deduced in the past, a separate implementation was made here. This code
can be safely deleted.


<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-14 18:36:14 +08:00
Tiewei Fang
d2709f8600 [fix](test) fix test_numbers case (#39303) 
bp: #38687

we use `order_qt` rather than `qt` to promise the order of results.
 2024-08-13 22:26:39 +08:00
lihangyu
677435cef8 [Pick](Branch-2.1) pick json reader fix and support specify $. as column (#39271) 
#39206
#38213
 2024-08-13 17:44:45 +08:00
Pxl
33220109f7 [Bug](materialized-view) fix analyze where clause failed on mv (#39061) (#39209) 
## Proposed changes
pick from #39061
fix analyze where clause failed on mv
do not analyze slot after replaceSlot to avoid duplicate columns in desc
 2024-08-13 16:08:20 +08:00
Sun Chenyang
228f78b80d [fix] (nereids) fix Match Expreesion in filter estimation (#39050) (#39215) 
## Proposed changes

pick from master #39050
 2024-08-13 10:57:53 +08:00
Sun Chenyang
a6155a517d [fix] (topn) fix uncleared block in topn_next() (#39119) (#39224) 
## Proposed changes

pick from master #39119
 2024-08-13 10:34:17 +08:00
bobhan1
c2044409da [branch-2.1] Picks "[fix](delete) Fix delete stmt on MOW table doesn't use partial update in Nereids planner #38751" (#39214) 
## Proposed changes

picks https://github.com/apache/doris/pull/38751
 2024-08-12 17:02:48 +08:00
LiBinfeng
0c39b88804 [Improvement](expr) fold child when const expr not folded (#38493) (#38961) 
cherry-pick from master 38493

1. fold child when const expr not folded
2. do not fold function `sleep`
3. move all exceptional expression into shouldSkipFold

before

mysql [test]>explain select sleep(sign(1)*100);
+-----------------------------------------------+
| Explain String(Nereids Planner)               |
+-----------------------------------------------+
| PLAN FRAGMENT 0                               |
|   OUTPUT EXPRS:                               |
|     sleep(cast((sign(1.0) * 100) as INT))[#0] |
|   PARTITION: UNPARTITIONED                    |
|                                               |
|   HAS_COLO_PLAN_NODE: false                   |
|                                               |
|   VRESULT SINK                                |
|      MYSQL_PROTOCAL                           |
|                                               |
|   0:VUNION(32)                                |
|      constant exprs:                          |
|          sleep(CAST((sign(1) * 100) AS int))  |
+-----------------------------------------------+
13 rows in set (15.02 sec)

mysql [test]>select sleep(sign(1)*100);
+-----------------------------------------------------+
| sleep(cast((sign(cast(1 as DOUBLE)) * 100) as INT)) |
+-----------------------------------------------------+
|                                                   1 |
+-----------------------------------------------------+
1 row in set (1 min 55.34 sec)


after

mysql [test]>explain select sleep(sign(1)*100);
+---------------------------------+
| Explain String(Nereids Planner) |
+---------------------------------+
| PLAN FRAGMENT 0                 |
|   OUTPUT EXPRS:                 |
|     sleep(100)[#0]              |
|   PARTITION: UNPARTITIONED      |
|                                 |
|   HAS_COLO_PLAN_NODE: false     |
|                                 |
|   VRESULT SINK                  |
|      MYSQL_PROTOCAL             |
|                                 |
|   0:VUNION(32)                  |
|      constant exprs:            |
|          sleep(100)             |
+---------------------------------+
13 rows in set (0.23 sec)

mysql [test]> select sleep(sign(1)*100);
+-----------------------------------------------------+
| sleep(cast((sign(cast(1 as DOUBLE)) * 100) as INT)) |
+-----------------------------------------------------+
|                                                   1 |
+-----------------------------------------------------+
1 row in set (1 min 40.37 sec)


Co-authored-by: Pxl <pxl290@qq.com>
 2024-08-12 15:13:48 +08:00
lihangyu
635ea39aaf [fix](regression-test) fix output of jsonb (#39217) 
introduced by  #39025
 2024-08-12 15:07:10 +08:00
Jerry Hu
96260d97bd [fix](ub) undefined behavior in FixedContainer (#39191) (#39201) 
## Proposed changes

pick #39191
Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-12 10:02:07 +08:00
Mryange
ebf5d70c9d [fix](function) MicroSecondsSub without scale (#38945) (#39194) 
## Proposed changes
https://github.com/apache/doris/pull/38945
Added the computeSignature function for millisecond/microsecond
calculation functions to generate parameters and return values with the
appropriate precision.
Modified the microSecondsAdd function, which was used for constant
folding, because constant folding uses the precision of the parameters
for calculation. However, for millisecond/microsecond calculations, it
is necessary to set the precision to the maximum to ensure correct
display.


before
```
mysql> SELECT MICROSECONDS_SUB('2010-11-30 23:50:50', 2);
+-------------------------------------------------------------------+
| microseconds_sub(cast('2010-11-30 23:50:50' as DATETIMEV2(0)), 2) |
+-------------------------------------------------------------------+
| 2010-11-30 23:50:49                                               |
+-------------------------------------------------------------------+
```
now
```
mysql> SELECT MICROSECONDS_SUB('2010-11-30 23:50:50', 2);
+-------------------------------------------------------------------+
| microseconds_sub(cast('2010-11-30 23:50:50' as DATETIMEV2(0)), 2) |
+-------------------------------------------------------------------+
| 2010-11-30 23:50:49.999998                                        |
+-------------------------------------------------------------------+
```


<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-12 10:01:28 +08:00
daidai
3da2d1c9d6 [bug](parquet)Fix the problem that the parquet reader reads the missing sub-columns of the struct and fails. (#38718) (#39192) 
bp #38718
 2024-08-11 20:37:40 +08:00
hui lai
0ee0dd6ae3 [fix](routine load) reset Kafka progress cache when routine load job topic change (#38474) (#39181) 
pick (#38474)

When change routine load job topic from test_topic_before to
test_topic_after by
```
ALTER ROUTINE LOAD FOR test_topic_change FROM KAFKA("kafka_topic" = "test_topic_after");
```
(test_topic_before has 5 rows and test_topic_after has 1 rows)

Exception happened, which cannot consume any data:
```
2024-07-29 15:57:28,122 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,123 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,125 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,126 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,128 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,129 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,131 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,133 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,134 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,136 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,137 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
```

It is necessary to reset Kafka progress cache when routine load job
topic change.
 2024-08-10 23:00:39 +08:00
yangshijie
5f77f909d9 [cherry-pick](branch-2.1) Pick "[feature](function) support ip functions named ipv4_to_ipv6 and cut_ipv6" (#39058) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
pick https://github.com/apache/doris/pull/36883 and
https://github.com/apache/doris/pull/35239
 2024-08-10 18:37:11 +08:00
zhangdong
9b4354fcb7 [fix](mtmv) second level MTMV always refresh all partition by mistake… (#39167) 
pick: 
https://github.com/apache/doris/pull/38698
 2024-08-10 18:26:56 +08:00