database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,238 Commits 1 Branch 0 Tags
011baeb2d2dcbd58ccc81d81607b691806d2a824
Commit Graph

20238 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yiguolei
011baeb2d2 [bugfix](allocatebytes) ignore null ptr column in Block (#41093) (#41099) 
## Proposed changes
Sometimes if expr failed, then there will be a nullptr column left in
the block.
We should ignore the nullptr column, or exception will be thrown and
some profile will not computed correctly.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-09-23 12:18:22 +08:00
Mryange
0e5c4281dc [fix](function) fix Substring/SubReplace error result with input utf8… (#40954) 
… string (#40929)
https://github.com/apache/doris/pull/40929
```

mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1)     |
+-------------------------------------+
| �a�好世界                             |
+-------------------------------------+

mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647)     |
+------------------------------------------+
| 中文测试                                 |
+------------------------------------------+
1 row in set (0.04 sec)

now
mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1)     |
+-------------------------------------+
| 你a世界                             |
+-------------------------------------+
1 row in set (0.05 sec)

mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647)     |
+------------------------------------------+
|                                          |
+------------------------------------------+
1 row in set (0.13 sec)
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-23 10:44:03 +08:00
zclllhhjj
d44ee1cddb [branch-2.1](session-variable) Add a debug variable of constant fold (#35584) (#41032) 
pick https://github.com/apache/doris/pull/35584
 2024-09-23 10:01:05 +08:00
bobhan1
7d64c8cbc6 [branch-2.1] Picks "[opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column #40280" (#41096) 
picks https://github.com/apache/doris/pull/40280
 2024-09-23 09:30:15 +08:00
minghong
9dc55f90eb [opt](nereids) set lower bound for range-selectivity（2.1） (#41061) 
## Proposed changes
pick #40089
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-22 07:32:22 +08:00
Siyang Tang
d1d52ae68c [feature](compaction) Add an http action for visibility of compaction score on each tablet (#38489) (#40826) 
pick: #38489 

Usage:
1. `curl http://be_ip:be_host/api/compaction_score?top_n=10` Returns a
json object contains compaction score for top n, n=top_n.
```
[
    {
        "compaction_score": "5",
        "tablet_id": "42595"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42587"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42593"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42597"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42589"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42599"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42601"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42591"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42585"
    },
    {
        "compaction_score": "4",
        "tablet_id": "10034"
    }
]
```
If top_n is not specified, return all compaction score for all tablets.
If top_n is illegal, raise an error.
```
invalid argument: top_n=wrong
```

2. `curl http://be_ip:be_host/api/compaction_score?sync_meta=true`
`sync_meta` is only available on cloud mode, will sync meta from meta
service. It can cooperate with top_n.
If add param `sync_meta` on non-cloud mode, will raise an error.
```
sync meta is only available for cloud mode
```

3. In the future, this endpoint may extend other utility, like fetching
tablet compaction score by table id, etc.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-21 20:35:55 +08:00
Mryange
9877a08834 [feature](function) support ngram_search function #38226 (#40893) 
https://github.com/apache/doris/pull/38226 
mysql [test]>select ngram_search('123456789' , '12345' , 3);
+---------------------------------------+
| ngram_search('123456789', '12345', 3) |
+---------------------------------------+
|                                   0.6 |
+---------------------------------------+
1 row in set (0.01 sec)

mysql [test]>select ngram_search("abababab","babababa",2);
+-----------------------------------------+
| ngram_search('abababab', 'babababa', 2) |
+-----------------------------------------+
|                                       1 |
+-----------------------------------------+
1 row in set (0.01 sec)
```

doc https://github.com/apache/doris-website/pull/899

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-21 20:34:44 +08:00
lihangyu
d5115a21b5 [pick](ShortCircuit) Conjuncts outof key columns's order should be handled (#41071) 
#37900
 2024-09-21 20:34:05 +08:00
Qi Chen
36a45c5ebf [Fix](parquet-reader) Fix parquet reader crash in set_dict(). (#41074) 
## Proposed changes

Backport #40643
 2024-09-21 08:48:06 +08:00
wuwenchi
e568ba3b4c [bugfix](hive)Handle exceptions when submitting tasks fails to prevent dead loop for 2.1 (#40708) (#41052) 
bp: #40708
 2024-09-21 08:47:31 +08:00
lihangyu
c6a6adb3a4 [Fix](topn) avoid missmatched row count when upgrading (#40999) 
#41000
 2024-09-21 08:46:57 +08:00
daidai
c9d71cdc64 [Enhancement](brpc)Added enable_brpc_builtin_services parameter in be.conf (#40718) (#41047) 
bp #40718 

## Proposed changes

Since [brpc's built-in
services](https://brpc.apache.org/zh/docs/builtin-services/buildin_services/)
will expose some internal server status, for security reasons, add the
parameter `enable_brpc_builtin_services` in `be.conf`. When the
parameter is false, this built-in service can be disabled.
 2024-09-21 08:45:38 +08:00
seawinde
4967635e54 [test](mtmv) Fix regression test unstable (#40871) (#41082) 
## Proposed changes

commitId: da6ac0c5
pr: https://github.com/apache/doris/pull/40871
 2024-09-20 23:49:23 +08:00
zhiqiang
63a1e0dccb [fix](profile) Fix npe when updating profile for internal load task. #40990 (#41077) 
cherry pick from #40990
 2024-09-20 23:46:43 +08:00
Vallish Pai
ff22059d5d [cherry-pick](branch-2.1)[fix](sys_table) fix potential NPE when query table_properties and partitions sys table (#41081) 
## Proposed changes

Issue Number: close #xxx

bp #40972
 2024-09-20 23:43:33 +08:00
wuwenchi
f4b06e83eb [bugfix](hive)Fix cache inconsistency issue for 2.1 (#40729) (#41050) 
## Proposed changes
bp: #40729
 2024-09-20 20:28:55 +08:00
Kaijie Chen
96b2fa1a24 [fix](move-memtable) fix bvar g_load_stream_file_writer_cnt (#39075) (#40985) 
backport #39075
 2024-09-20 20:20:49 +08:00
starocean999
f60a881425 [fix](nereids)adjust conjunct's nullable info in LogicalExternalRelation (#41031) 
## Proposed changes

pick from master https://github.com/apache/doris/pull/41014
<!--Describe your changes.-->
 2024-09-20 20:20:12 +08:00
wuwenchi
2bb83302e4 [bugfix](hive)Remove redundant split operations for 2.1 (#40683) (#41055) 
## Proposed changes
bp: #40683
 2024-09-20 20:09:11 +08:00
wuwenchi
1686650a56 [bugfix](test)Set the enable_fallback_to_original_planner behavior for 2.1 (#40913)(#40989) (#41065) 
bp:  #40913 #40989
 2024-09-20 20:08:38 +08:00
Qi Chen
7e0d498043 [Fix](orc-reader) Fix StringRef nullptr data in orc-reader. (#41073) 
## Proposed changes

backport #40857
 2024-09-20 20:06:50 +08:00
starocean999
dbd72deaa5 [opt](nereids)show user friendly error msg when window function contains order by expression (#40937) (#41036) 
pick from master #40937
 2024-09-20 18:50:55 +08:00
Siyang Tang
a8aebe6394 [fix](scanner-builder) Fix scan builder datarace beacause of picked from master's different codebase (#41019) 
## Proposed changes

As title. Master will hold header lock internally when capturing read
resource.
 2024-09-20 18:17:59 +08:00
daidai
c744eb87c5 [fix](regression)fix some regression test (#40928) (#41046) 
bp #40928
 2024-09-20 18:17:44 +08:00
zhiqiang
1259fe2bd5 [fix](covar) Fix covar nullable on branch-2.1 (#40841) 
covar should not be always nullable.

This fix on branch-2.1 makes covar same with master on FE.
 2024-09-20 17:35:27 +08:00
Mingyu Chen
596cfc9b18 [opt](cpu-profile) enable cpu profile in BE webui (#40330) (#41044) 
bp #40330
 2024-09-20 17:18:16 +08:00
Mingyu Chen
8e860a26a7 [fix](systable) fix unstable case for partitions table (#40553) (#41043) 
bp #40553
 2024-09-20 17:13:30 +08:00
Mingyu Chen
3484e7b2ff [fix](table_options) fix potential NPE when quering table_options systable (#40944) (#41040) 
bp #40944
 2024-09-20 16:57:02 +08:00
airborne12
68b7783e3a [Fix](scanner) lazy update block mem usage when status is not ok #40953 (#41034) 
cherry pick from #40953
 2024-09-20 15:09:26 +08:00
Vallish Pai
d7f60c5ef1 [Cherry-pick](branch 2.1)[Enhancement] support last_query_id function (#41025) 
## Proposed changes

Issue Number: close #xxx

backport #40739
 2024-09-20 14:07:39 +08:00
TengJianPing
6539c8fd35 [fix](decimal) throw overflow exception if result is NaN of Infinit when converting from decimal to float (#40290) (#41007) 
## Proposed changes

Issue Number: close #xxx

BP #40290
 2024-09-20 14:05:51 +08:00
Socrates
e0fac66223 [branch-2.1](fix) fix snappy decompressor bug (#40862) 
## Proposed changes
Hadoop snappycodec source :

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc
Example:
OriginData(The original data will be divided into several large data
block.) :
     large data block1 | large data block2 | large data block3 | ....
The large data block will be divided into several small data block.
Suppose a large data block is divided into three small blocks:
large data block1: | small block1 | small block2 | small block3 |
CompressData: <A [B1 compress(small block1) ] [B2 compress(small block1)
] [B3 compress(small block1)]>

A : original length of the current block of large data block.
sizeof(A) = 4 bytes.
A = length(small block1) + length(small block2) + length(small block3)
Bx : length of  small data block bx.
sizeof(Bx) = 4 bytes.
Bx = length(compress(small blockx))
 2024-09-20 11:57:14 +08:00
Xin Liao
3249931fa6 [Fix](test) fix flake case in both test_insert_strict_fail_url and test_insert_partition_fail_url #40998 (#41024) 
cherry pick from #40998
 2024-09-20 09:52:09 +08:00
bobhan1
64880a10d6 [branch-2.1] Picks "[Fix](partial update) Fix partial update failed when merge_type=MERGE #40730" (#40951) 
picks https://github.com/apache/doris/pull/40730
 2024-09-20 00:02:17 +08:00
zy-kkk
8b1906be81 [branch-2.1][fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP (#40923) 
pick  (#40773)

When FE collects statistics, its ThreadLocal will hold the ThreadLocal
of HikariCP in JdbcClient, making it difficult for HikariCP to be gc, so
this problem can be alleviated by enabling weak references of HikariCP.
 2024-09-19 23:59:14 +08:00
Kaijie Chen
549bc3e288 [fix](pipelinex) fix fragment instance progress reports (#40325) (#40987) 
backport #40325
 2024-09-19 23:58:38 +08:00
bobhan1
49dec9f39d [branch-2.1] Picks "[opt](merge-on-write) Reduce the version not continuous logs for merge-on-write table #40946" (#40996) 
picks https://github.com/apache/doris/pull/40946
 2024-09-19 23:58:05 +08:00
zhangdong
f81d087bab [enhance](mtmv)Improve the speed of obtaining table snapshots (#40717) (#40995) 
pick: https://github.com/apache/doris/pull/40717
 2024-09-19 22:47:50 +08:00
zhangdong
fd5423145d [enhance](mtmv)tasks tvf not getLock (#40816) (#40994) 
pick: https://github.com/apache/doris/pull/40816
 2024-09-19 22:40:17 +08:00
zhangdong
7ecf088a61 [enhance](mtmv)when drop temp partition, not trigger materialized vie… (#40993) 
…w refresh (#40875)

pick: https://github.com/apache/doris/pull/40875
 2024-09-19 22:35:31 +08:00
Kaijie Chen
f32d540ac5 [test](move-memtable) add more fault injection cases for writer v2 (#39169) (#40986) 
backport #39169
 2024-09-19 22:31:12 +08:00
zhangdong
7a24719f48 [enhance](mtmv) Refuse to execute insert overwrite on the same table … (#40992) 
…(#40558)

pick: https://github.com/apache/doris/pull/40558
 2024-09-19 22:18:46 +08:00
Kaijie Chen
b7ded71bfb [fix](stream-load) catch exception when parsing CONTENT_LENGTH (#40796) (#40988) 
backport #40796
 2024-09-19 22:15:08 +08:00
zy-kkk
5f583fa329 [branch-2.1][test](jdbc catalog) add oceanbase ce jdbc catalog test (#40978) 
pick #34972)
 2024-09-19 22:11:24 +08:00
Kaijie Chen
e7d41eb143 [test](p2) print error message when assert fails (#40699) (#40977) 
cherry-pick #40699
 2024-09-19 22:01:40 +08:00
Jerry Hu
b8bc9b699c [fix](scan) Incorrect scan keys lead to wrong query results. (#40814) (#40971) 
## Proposed changes
pick #40814
```
mysql [doris_14555]>select * from table_9436528_3;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4                   | col6               | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 |    1 |  -82 |    1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 |    1 |   92 |    1 | 2024-02-16 04:37:37.00 |   23423423.0324234 | NULL |
| -100 |    0 |  -82 |    0 | 2023-11-11 10:49:43.00 |   840968969.872149 | NULL |
```
wrong result:
```
mysql [doris_14555]>select * from table_9436528_3 where col1 <= -100 and col2 in (true, false) and col3 = -82;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4                   | col6               | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 |    1 |  -82 |    1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 |    1 |   92 |    1 | 2024-02-16 04:37:37.00 |   23423423.0324234 | NULL |
+------+------+------+------+------------------------+--------------------+------+
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-19 22:01:02 +08:00
Xin Liao
f2b93d5896 [Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump #40940 (#40974) 
cherry pick from #40940
 2024-09-19 17:40:46 +08:00
feiniaofeiafei
8302261dd2 [Fix](nereids) set all nullable aggregate function to alwaysnullable in window expression (#40693) (#40809) 
cherry-pick from master #40693
 2024-09-19 15:19:06 +08:00
amory
f483a7605c [fix](nestedtypes) fix nested type with is_exclusive (#40434) 
is_exclusive in column_array/map/struct has wrong semantic , we should
make sure it's nested column is also is_exclusive which can make
behavior right in some operator like join.

## Proposed changes
backport: https://github.com/apache/doris/pull/40398
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-19 12:28:51 +08:00
camby
84f0b1fbfe [feature](view) support create or replace view stmt (#40715) (#40915) 
pick #40715 to branch-2.1
 2024-09-19 01:10:43 +08:00