database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,189 Commits 1 Branch 0 Tags
84f0b1fbfea8bd1d52aedf10b6e209f4c8be421c
Commit Graph

9187 Commits

Author SHA1 Message Date
camby
84f0b1fbfe [feature](view) support create or replace view stmt (#40715) (#40915) 
pick #40715 to branch-2.1
 2024-09-19 01:10:43 +08:00
Siyang Tang
7847dad4b8 [enhancement](show) Move storage policy info from show create table to show partitions (#39923) (#40824) 
## Proposed changes

pick: #39923
 2024-09-18 23:35:19 +08:00
morrySnow
5c70fe35ea [fix](Nereids) sink distribute is wrong when table is unpartitioned (#40541) (#40901) 
pick from master #40541
 2024-09-18 15:18:41 +08:00
lihangyu
956b56c664 [pick](short circuit) avoid set cacheId when non prepared execute (#40909) 
#40525
 2024-09-18 14:50:50 +08:00
zy-kkk
f3a42faf40 [branch-2.1][improvement](jdbc catalog) support jdbc external catalog insert stmt in nereids (#40902) 
pick  (#39813)
 2024-09-18 14:02:20 +08:00
morrySnow
7d23a7c587 [fix](planner) throw NPE when insert overwrite partition(*) (#40656) (#40903) 
pick from master #40656
 2024-09-18 13:48:48 +08:00
morrySnow
3336e6acbb [fix](Nereids) avoid bad cast when compute scale for round (#40776) (#40904) 
pick from master #40776

for pass test case, also fix errors in computeResultInFe

computeResultInFe will generate wrong result set if output does not
match between final result and the node executing computeResultInFe
 2024-09-18 13:48:01 +08:00
Jibing-Li
8a575d0891 [improvement](statistics)User admin instead of root to run stats sql. (#40867) (#40898) 
backport: https://github.com/apache/doris/pull/40867
 2024-09-18 13:36:50 +08:00
Jibing-Li
c176c9feb0 [fix](statistics)Fix empty table keep auto analyze bug. (#40811) (#40896) 
backport: https://github.com/apache/doris/pull/40811
 2024-09-18 12:51:44 +08:00
yiguolei
a6e2364f56 [cherrypick](branch21) pick some exception handleing pr in page io (#40883) 
## Proposed changes

pick
https://github.com/apache/doris/pull/39536
https://github.com/apache/doris/pull/39913
https://github.com/apache/doris/pull/39964
https://github.com/apache/doris/pull/40020

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-09-16 08:56:43 +08:00
zhangdong
782973ee77 [fix](auth)Http api check auth (#40688) (#40865) 
pick: https://github.com/apache/doris/pull/40688
 2024-09-15 23:50:54 +08:00
seawinde
fe9f4fdd95 [pick](mtmv)Pick some pr to 21 #39998 #40332 #40485 (#40843) 
## Proposed changes
pr: https://github.com/apache/doris/pull/40485
commitId: 18a374f6

pr: https://github.com/apache/doris/pull/40332
commitId: cd2062ae

pr: https://github.com/apache/doris/pull/39998
commitId: 948fb2ab
 2024-09-15 23:48:33 +08:00
zclllhhjj
0d3b9c3f18 [branch-2.1](load) fix wrong replica number when loading to table with no partition (#40635) (#40881) 
pick https://github.com/apache/doris/pull/40635
 2024-09-15 19:58:53 +08:00
Jibing-Li
b7103bf00c [fix](statistics)Fix drop stats log editlog bug. Catch drop stats exception while truncate table. (#40738) (#40853) 
backport: https://github.com/apache/doris/pull/40738
 2024-09-14 23:08:24 +08:00
Jibing-Li
0b1d517caa [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) (#40838) 
backport: https://github.com/apache/doris/pull/40457
 2024-09-14 13:19:35 +08:00
HappenLee
472a42a80e [update](exec) set ENABLE_NEREIDS_DML_WITH_PIPELINE = true, update from 2.0 to 2.1 (#40825) 
change the dafault value in update to 2.1
 2024-09-14 10:58:07 +08:00
Gabriel
168742deba [followup](branch-2.1) Fuzzy parallel_prepare_threshold (#40793) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-13 22:46:31 +08:00
Calvin Kirs
f9b79c613a [Fix](Job)Replaying logs should not modify the original information of the job (#40474) (#40808) 
…

## Proposed changes
```
        JobExecutionConfiguration jobConfig = new JobExecutionConfiguration();
        jobConfig.setExecuteType(JobExecuteType.INSTANT);
        setJobConfig(jobConfig);
```
- Replaying logs should not modify the original information of the job
- Use the new optimizer to check whether the executed statement is legal

(cherry picked from commit de90051162de7004cf171bbf4d21bd95ff9f3540)

## Proposed changes

Issue Number: #40474
 2024-09-13 20:47:57 +08:00
Sun Chenyang
c2ad828b0c [fix] (Nereids) restrict the execution of match on the scan (#40532) (#40780) 
## Proposed changes

pick from master #40532

<!--Describe your changes.-->
 2024-09-13 20:03:11 +08:00
zy-kkk
9e8e1dac4e [branch-2.1][fix](default value) Avoid default precision value of 0 for default expressions during deserialization (#40806) 
pick  (#40671)

This PR fixes the issue where the precision of default value expressions
was incorrectly set to `0` during deserialization. The `gsonPostProcess`
method no longer assigns a default precision of `0` if the precision is
not provided. Additionally, logic has been added to ensure that
expressions without precision do not incorrectly apply a precision of
`0`.
 2024-09-13 19:59:12 +08:00
Gabriel
3395cd5ce9 [PipelineX](improvement) Prepare tasks in parallel (#40270) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-13 13:34:29 +08:00
Calvin Kirs
76495b6d5c [chore](dependencies)upgrade spring to 5.3.39 (#39863) (#40709) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit a692af226a8292d1b0a409c914d616157b64c82a)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-13 10:39:36 +08:00
morrySnow
51c8b62d1c [opt](Nereids) fix several insert into related issues (#40467) (#40755) 
pick from master #40467

- http_stream TVF should always generate one fragment plan
- http_stream TVF plan should not check root as scan node
- distinguish group_commit TVF with normal insert statement
- index and generate slot should based on type cast base slot
- agg_state could cast from nullable to non-nullable
- colocated and bucket scan range compute should only on scan node
 2024-09-13 10:19:56 +08:00
Calvin Kirs
a7fe2ae901 [branch-2.1][Fix](Job)Replace BlockingWaitStrategy with LiteTimeoutBlockingWaitStrategy to avoid deadlock issues. (#40625) (#40707) 
…

FYI https://issues.apache.org/jira/browse/LOG4J2-1221

- BlockingWaitStrategy is a wait strategy used in the Disruptor
framework that blocks the thread when the ring buffer is full or not yet
available for publishing.

When threads are blocked, they are waiting for space in the ring buffer
to become available, which can lead to potential deadlocks if not
managed properly.
Timeout Handling:

- LiteTimeoutBlockingWaitStrategy provides a timeout for waiting
threads. If the buffer is not ready within the timeout period, the
thread is released, preventing it from being blocked indefinitely.
Reduced Risk of Deadlocks:

- By avoiding indefinite blocking, this strategy reduces the risk of
deadlocks caused by threads waiting on each other. The timeout allows
the system to handle scenarios where resources are temporarily

(cherry picked from commit 087048f2b2f06eae15e80e7ffa209610c9e3f173)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-12 22:42:45 +08:00
morrySnow
967801ca66 [feature](Nereids) support query on ES table in internal catalog (#40449) (#40754) 
pick from master #40449
 2024-09-12 20:01:48 +08:00
morrySnow
fedadbba6e [opt](Nereids) support search from override udfs with same arity (#40432) (#40751) 
pick from master #40432

create alias function f1(int) with parameter(id) as abs(id); create
alias function f1(string) with parameter(id) as substr(id, 2); select
f1('1'); -- bind on f1(string)
select f1(1);   -- bind on f1(int)

test case already existed in P0
 2024-09-12 19:58:49 +08:00
wuwenchi
4b7b43b5ca [bugfix](hive/iceberg)align with Hive insert overwrite table functionality (#39840) (#40724) 
bp #39840
 2024-09-12 19:20:15 +08:00
wuwenchi
23b21fcebf [bugfix](hms)add default value check for hms catalog with dlf (#40336) (#40723) 
## Proposed changes

bp #40336
 2024-09-12 19:19:47 +08:00
wuwenchi
3d2ac60bae [bugfix](hive)Delete the temporarily created folder (#40424) (#40722) 
## Proposed changes

bp: #40424
 2024-09-12 19:18:35 +08:00
starocean999
3484a6cfe3 [fix](nereids)let query be able to forwarded to master (#40119) (#40725) 
pick from master #40119
 2024-09-12 17:09:36 +08:00
starocean999
0f8176dee0 [fix](nereids) build agg for random distributed agg table in bindRelation phase (#40181) (#40702) 
pick from master #40181
 2024-09-12 14:08:50 +08:00
zclllhhjj
e2dc7544dd [branch-2.1](iot) Fix unexpected error when iot auto detect get no data (#40657) (#40691) 
pick https://github.com/apache/doris/pull/40657
 2024-09-12 14:03:43 +08:00
Vallish Pai
3604d63184 [Branch 2.1] backport systable PR (#34384,#40153,#40456,#40455,#40568) (#40687) 
backport
https://github.com/apache/doris/pull/40568
https://github.com/apache/doris/pull/40455
https://github.com/apache/doris/pull/40456
https://github.com/apache/doris/pull/40153
https://github.com/apache/doris/pull/34384

Test result:
2024-09-11 11:00:45.618 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.619 INFO [suite-thread-1] (Suite.groovy:359) -
Execute sql: REVOKE SELECT_PRIV ON
test_partitions_schema_db.duplicate_table FROM partitions_user
2024-09-11 11:00:45.625 INFO [suite-thread-1] (SuiteContext.groovy:299)
- Create new connection for user 'partitions_user'
2024-09-11 11:00:45.632 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
from information_schema.partitions where
table_schema="test_partitions_schema_db" order by
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
2024-09-11 11:00:45.644 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.645 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_partitions_schema in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy
succeed
2024-09-11 11:00:45.652 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:01:10.321 INFO [main] (RegressionTest.groovy:380) -
Success suites:

/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy:
group=default,p0, name=test_partitions_schema
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:459) - All
suites success.
 ____   _    ____ ____  _____ ____
|  _ \ / \  / ___/ ___|| ____|  _ \
| |_) / _ \ \___ \___ \|  _| | | | |
|  __/ ___ \ ___) |__) | |___| |_| |
|_| /_/   \_\____/____/|_____|____/

2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:119) - Test
finished


2024-09-11 11:03:00.712 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select * from
information_schema.table_options ORDER BY
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,TABLE_MODEL,TABLE_MODEL_KEY,DISTRIBUTE_KEY,DISTRIBUTE_TYPE,BUCKETS_NUM,PARTITION_NUM;
2024-09-11 11:03:00.729 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:03:00.731 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_table_options in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy
succeed
2024-09-11 11:03:04.817 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:03:28.741 INFO [main] (RegressionTest.groovy:380) -
Success suites:

/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy:
group=default,p0, name=test_table_options
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:459) - All
suites success.
 ____   _    ____ ____  _____ ____
|  _ \ / \  / ___/ ___|| ____|  _ \
| |_) / _ \ \___ \___ \|  _| | | | |
|  __/ ___ \ ___) |__) | |___| |_| |
|_| /_/   \_\____/____/|_____|____/

2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:119) - Test
finished


*************************** 7. row ***************************
             PartitionId: 18035
           PartitionName: p100
          VisibleVersion: 2
      VisibleVersionTime: 2024-09-11 10:59:28
                   State: NORMAL
            PartitionKey: col_1
Range: [types: [INT]; keys: [83647]; ..types: [INT]; keys: [2147483647];
)
         DistributionKey: pk
                 Buckets: 10
          ReplicationNum: 1
           StorageMedium: HDD
            CooldownTime: 9999-12-31 15:59:59
     RemoteStoragePolicy: 
LastConsistencyCheckTime: NULL
                DataSize: 2.872 KB
              IsInMemory: false
       ReplicaAllocation: tag.location.default: 1
               IsMutable: true
      SyncWithBaseTables: true
            UnsyncTables: NULL
        CommittedVersion: 2
                RowCount: 4
7 rows in set (0.01 sec)

---------

Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
 2024-09-12 11:50:09 +08:00
camby
361a59dec8 [feature](aes_encrypt) support GCM mode for aes_encrypt and aes_decrypt (#40004) (#40672) 
pick #40004 to branch-2.1
 2024-09-11 23:28:28 +08:00
zclllhhjj
bf156d1665 [branch-2.1](DDL) check illegal partition exprs (#40158) (#40651) 
pick https://github.com/apache/doris/pull/40158
 2024-09-11 22:37:55 +08:00
zclllhhjj
52e13c9e6c [branch-2.1](nereids) remove interface Nondeterministic and fix random/random_bytes foldable properties (#39944) (#40650) 
pick https://github.com/apache/doris/pull/39944
 2024-09-11 17:27:52 +08:00
zclllhhjj
d554f600bc [branch-2.1](partition) Support use Auto and Dynamic partition at the same time (#39580) (#40649) 
pick https://github.com/apache/doris/pull/39580
 2024-09-11 15:35:20 +08:00
zclllhhjj
3246baa451 [branch-2.1](function) Refine crypto functions signature to fix wrong result(#40285) (#40648) 
pick https://github.com/apache/doris/pull/40285
 2024-09-11 15:32:19 +08:00
morrySnow
1c27724766 [chore](audit) let line comment work well (#40602) 2024-09-11 15:09:27 +08:00
zhiqiang
ab8f9f7bf6 [fix](histogram) Fix histgrom nullable properties on branch-2.1 (#40638) 
nullable property of histogram on master is changed by
https://github.com/apache/doris/pull/37330
pick it to branch-2.1
related change on 2.1: https://github.com/apache/doris/pull/38608,
#38608 relies on AlwaysNotNullable property.
 2024-09-11 10:08:25 +08:00
yujun
d3dccbc1dd [chore](show partitions) show partitions print commit version #28274 (#40594) 
cherry pick from #28274
 2024-09-10 18:04:24 +08:00
yiguolei
4e453dc1bb Revert "[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457)" (#40616) 
Reverts apache/doris#40540
 2024-09-10 17:17:13 +08:00
Jibing-Li
e43e6e2bba [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) (#40540) 
backport: https://github.com/apache/doris/pull/40457
 2024-09-10 12:55:53 +08:00
zhangdong
db8fb66dba [fix](mtmv)fix nested mtmv not refresh (#40433) (#40560) 
pick: https://github.com/apache/doris/pull/40433
 2024-09-10 11:51:41 +08:00
walter
05f82df1f1 [fix](restore) update is_being_synced properties #40194 (#40556) 
cherry pick from #40194
 2024-09-10 11:15:50 +08:00
hui lai
8eda15ae16 [opt](routine load) support routine load perceived schema change (#39412) (#40508) 
pick #39412

At present, if the table structure changes, the routine load cannot
perceive it. As a long-running load, it should be able to perceive the
changes in the table structure.
 2024-09-10 11:05:58 +08:00
Calvin Kirs
978fcf12b6 [Chore](dependnecies)remove javax.el and upgrade jackson to 2.16.0 (#38843) (#39389) 
…

## Proposed changes
javax.el is the API for Java Expression Language, which provides a
simple and flexible way for Java Web applications to access and
manipulate data. EL expressions are commonly used in JSP pages, but we
are not involved in their use here, so I removed it.
- upgrade jackson to 2.16.0

(cherry picked from commit 33a0ea5565c2e0d7e5909ca3b512fb0b539f05f4)

#38843

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-09 21:58:47 +08:00
walter
68d1064aa9 [chore](persist) reduce truncate table operation log size #40422 (#40555) 
cherry pick from #40422
 2024-09-09 20:01:07 +08:00
abmdocrt
8f37eccbf2 [Cherry-pick](branch-2.1) Pick "[Featrue](default value) Support bitmap_empty default value (#40364)" (#40487) 
## Proposed changes

Pick #40364 

<!--Describe your changes.-->
 2024-09-09 16:57:38 +08:00
Calvin Kirs
2235c1abd3 [Feat](OSS)Decouple COS and OBS dependencies with optional inclusion mechanism (#39831) 
…

improve the flexibility of the project by decoupling direct dependencies
on the hadoop-cos and hadoop-huaweicloud libraries. These changes allow
users to control whether COS and OBS dependencies are included in the
final build, enabling a more customizable setup.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-09 16:50:12 +08:00