database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,310 Commits 1 Branch 0 Tags
741807bb22bf64ae3ca21486551e80938ffa57b5
Commit Graph

3069 Commits

Author SHA1 Message Date
Xin Liao
0a95757a4d [opt](test) Optimize execution time of test_s3_load case #37562 (#37612) 
cherry pick from #37562
 2024-07-10 19:09:46 +08:00
abmdocrt
a4aaace481 [Fix](branch-2.1) Fix partial update regression test (#37560) 
Version 2.1 does not support partial update for variant type data, so we
delete this case.
 2024-07-10 10:44:24 +08:00
morrySnow
db4d061a68 [fix](Nereids) null type result with alias name should keep alias name (#37457) (#37524) 
pick from master #37457
 2024-07-09 20:46:51 +08:00
feiniaofeiafei
8ef83259ff [fix](planner) fix bug of select stmt toSql(#37274) (#37344) 
cherry-pick from master #37274
 2024-07-09 20:34:57 +08:00
minghong
9b075bc873 [fix](nereids) derive column stats for 'expr and A is not null' (#37235) (#37498) 
pick from #37235 
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats after step1, leftStats
should be normalized to avoid abnormal cases, such as ndv > rowCount or
numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-09 17:46:57 +08:00
abmdocrt
2d5eb55310 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add some partial update case (#36385)" (#37382) 
#36385
 2024-07-09 10:52:04 +08:00
Mingyu Chen
ca0e44f83f [fix](case) fix struct format out files (#37350) (#37499) 
bp #37350
 2024-07-09 10:11:50 +08:00
amory
0a103aa11f [improve](json)improve json support empty keys #36762 (#37351) 2024-07-08 19:04:51 +08:00
seawinde
fbc954e8be [feat](mtmv) Support grouping_sets rewrite when query rewrite by materialized view (#36056) (#37436) 
cherry pick from master
pr: #36056
commitId: 569c9772
 2024-07-08 15:06:16 +08:00
seawinde
779a51570e [opt](mtmv) Set query rewrite by materialized view default enable (#35897) (#36949) 
cherry pick from master
pr: #35897
commitId: 603fa82f
 2024-07-08 14:29:38 +08:00
hui lai
c66df8d9e6 [branch-2.1](load) fix no error url if no partition can be found (#36831) (#37401) 
## Proposed changes

pick #36831

before
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0
}
```

after
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3"
}
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-08 10:41:33 +08:00
abmdocrt
32529ecda2 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add partial update mix cases (#37113)" (#37384) 
#37113
 2024-07-07 18:26:46 +08:00
zzzxl
c399a0e216 [opt](inverted index) reduce generation of the rowid_result if not necessary #35357 (#36569) 2024-07-06 21:33:03 +08:00
bobhan1
ef59af8df0 [branch-2.1] Picks "[fix](regression) Fix p0 case test_modify_reorder_column #37256" (#37332) 
## Proposed changes

picks https://github.com/apache/doris/pull/37256
 2024-07-05 22:20:14 +08:00
bobhan1
5de6aa74c0 [branch-2.1] Picks "[opt](autoinc) Forbid some schema change when the table has auto-increment column #37186" (#37331) 
## Proposed changes

picks https://github.com/apache/doris/pull/37186
 2024-07-05 21:59:30 +08:00
seawinde
8a0d05d9b0 [opt](mtmv) Materialized view partition track supports date_trunc and optimize the fail reason (#35562) (#36947) 
cherry pick from master #35562
commitId: 43d0f191
 2024-07-05 15:12:43 +08:00
qiye
f8cee439b6 [feature](ES Catalog) map nested/object type in ES to JSON type in Doris (#37101) (#37182) 
backport #37101
 2024-07-05 10:48:32 +08:00
morrySnow
256221a574 [fix](Nereids) normalize aggregate should not push down lambda's param (#37109) (#37285) 
pick from master #37109

ArrayItemSlot should not be inputSlot
 2024-07-05 09:33:57 +08:00
morrySnow
8373610281 [opt](ctas) add a variable to control varchar length in ctas (#37069) (#37284) 
pick from master #37069

add a new session variable: use_max_length_of_varchar_in_ctas

In CTAS (Create Table As Select), if CHAR/VARCHAR columns do not
originate from the source table, whether to set the length of such a
column to MAX, which is 65533. The default is true.
 2024-07-04 22:09:41 +08:00
feiniaofeiafei
4e4f3d204e [feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN (#36828) (#36981) 
cherry-pick #36828 to branch-2.1

The requirement for predicate pushdown through the window operator is
that the partition by slots of the window contains all slots in the
predicate. The original implementation of doris only allows predicate
pushdown with one slot. This PR relaxes this restriction and allows for
predicate pushdown with multiple slots. The same applies to the
predicate pushdown of the LogicalPartitionTopN operator. The following
sql is an example.

select
	*
from
	(
	select
		row_number() over(partition by id, value1 order by value1) as num,
		id,
		value1
	from
		push_down_multi_column_predicate_through_window_t ) t
where
	abs(id + value1)<4
	and num <= 2;


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
 2024-07-04 21:00:08 +08:00
Mingyu Chen
55636e8035 [test](migrate) move 3 cases from p2 to p0 (#36957) (#37264) 
bp #36957

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
 2024-07-04 20:09:59 +08:00
morrySnow
c7ad1f3d21 [fix](Nereids) simplify window expression should inherit data type (#37061) (#37283) 
pick from master #37061

after window expression rewritten by literal.
literal's data type should same with original window expression.
 2024-07-04 19:19:05 +08:00
morrySnow
e4fb506c20 [fix](Nereids) null type in result set will be cast to tinyint (#37019) (#37281) 
pick from master #37019
 2024-07-04 19:18:35 +08:00
Mingyu Chen
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253) 
bp #37039
 2024-07-04 13:56:05 +08:00
Mingyu Chen
3613413a54 [fix](hive) support find serde info from both tbl properties and serde properties (#37043) (#37188) 
bp #37043
 2024-07-04 13:55:38 +08:00
Jibing-Li
bf3ea1839c [test]Mv external p2 test case to p0. (#37070) (#37140) 
backport: https://github.com/apache/doris/pull/37070
 2024-07-04 11:19:31 +08:00
minghong
26be313d40 [mv](nereids) mv cost related PRs (#35652 #35701 #35864 #36368 #36789 #34970) (#37097) 
## Proposed changes
pick from #35652 #35701 #35864 #36368 #36789 #34970

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-04 09:42:11 +08:00
bobhan1
69aebc2d25 [branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067" (#37226) 
## Proposed changes

picks https://github.com/apache/doris/pull/37067
 2024-07-03 22:42:51 +08:00
zhannngchen
97945af947 [fix](merge-on-write) when full clone failed, duplicate key might occur (#37001) (#37229) 
cherry-pick #37001
 2024-07-03 19:48:10 +08:00
zy-kkk
a9f9113c48 [branch-2.1][test](external)move hive cases from p2 to p0 (#37149) 
pk (#36855)
test_hive_same_db_table_name
test_hive_special_char_partition
test_complex_types
test_wide_table
 2024-07-03 19:44:52 +08:00
Mingyu Chen
e5695e058f [test](migrate) move 2 cases from p2 to p0 (#36935) (#37200) 
bp #36935

Co-authored-by: zhangdong <493738387@qq.com>
 2024-07-03 17:29:01 +08:00
minghong
b3f2bd20e3 [feat](nereids) support explain delete from clause #36782 (#37100) 
## Proposed changes
pick from  #36782

support explain like:
explain delete from T where A=1

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-03 15:08:08 +08:00
Qi Chen
e857680661 [Migrate-Test](multi-catalog) Migrate p2 tests from p2 to p0. (#37175) 
Backport #36989.
 2024-07-03 11:08:49 +08:00
Tiewei Fang
4761090848 [fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF (#37161) 
bp: #35489
 2024-07-03 10:58:08 +08:00
Tiewei Fang
0aeb768bf9 [Fix](export/outfile) Support compression when exporting data to Parquet / ORC. (#37167) 
bp: #36490
 2024-07-03 10:53:57 +08:00
zhangdong
6804ae5827 [enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change (#36513) (#37122) 
pick from https://github.com/apache/doris/pull/36513
 2024-07-03 10:42:03 +08:00
zhangdong
5bb6642b86 [fix](mtmv)mtmv support default key (#36221) (#36601) 
pick : https://github.com/apache/doris/pull/36221
 2024-07-02 23:08:38 +08:00
zhangdong
65375b48fc [enhance](mtmv)support replace materialized view (#36749) (#37147) 
pick: https://github.com/apache/doris/pull/36749
 2024-07-02 23:00:55 +08:00
wuwenchi
e7e1e967cf [test](migrate) move 2 cases from p2 to p0 for 2.1 (#37139) 
pick #37004
 2024-07-02 22:50:53 +08:00
Tiewei Fang
b445c783eb [test](tvf) move p2 tvf tests from p2 to p0 (#37081) (#37152) 
bp: #37081
 2024-07-02 22:38:22 +08:00
Tiewei Fang
74086189d3 [test](tvf) move p2 tvf tests from p2 to p0 (#36871) (#37150) 
bp: #36871
 2024-07-02 22:37:43 +08:00
Tiewei Fang
bd24a8bdd9 [Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values (#37153) 
bp: #36668
 2024-07-02 22:12:17 +08:00
LiBinfeng
c7549acf29 [Fix](Nereids) fix leading with different be instance number (#36613) (#36967) 
cherry-pick: #36613

Problem:
When use different be number to test leading explain shape plan,
physical distribute plan would differ due to different be numbers
Solved:
Disable physical distribute node showing in fix leading cases
 2024-07-02 19:56:17 +08:00
qiye
b63e6ac350 [fix](ES Catalog)Add array types support in esquery function (#36936) (#37054) 
backport #36936
 2024-07-02 17:48:35 +08:00
Ashin Gau
cf86eb8647 [test](migrate) move test_hive_text_complex_type from p2 to p0 (#37007) (#37123) 
bp: #37007
 2024-07-02 17:36:37 +08:00
Sun Chenyang
7443e8fcf2 [cherry-pick](branch-2.1) fix single compaction test p2 #34568 #36881 (#37075) 2024-07-02 15:22:04 +08:00
zhangdong
496024ffb5 [fix](mtmv) mtmv add case and fix bug (#36019) (#36600) 
pick from: #36019
 2024-07-02 14:24:53 +08:00
bobhan1
c66975b244 [branch-2.1] Pick "[Fix](regression) Fix p0 case test_unique_table_auto_inc_concurrent #37048" (#37050) 
## Proposed changes

pick https://github.com/apache/doris/pull/37048
 2024-07-02 11:15:50 +08:00
TengJianPing
6789f5bc80 [fix](null safe equal join) fix coredump if both sides of the conjunct is not nullable #36263 (#37073) 2024-07-02 11:01:55 +08:00
starocean999
6f944549d1 [fix](regression)fix case failure (#37058) 2024-07-02 09:55:18 +08:00