database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,167 Commits 1 Branch 0 Tags
b7103bf00c6bc1099bb2cc26b091ee75f11cee34
Commit Graph

7623 Commits

Author SHA1 Message Date
Jibing-Li
b7103bf00c [fix](statistics)Fix drop stats log editlog bug. Catch drop stats exception while truncate table. (#40738) (#40853) 
backport: https://github.com/apache/doris/pull/40738
 2024-09-14 23:08:24 +08:00
Jibing-Li
0b1d517caa [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) (#40838) 
backport: https://github.com/apache/doris/pull/40457
 2024-09-14 13:19:35 +08:00
HappenLee
472a42a80e [update](exec) set ENABLE_NEREIDS_DML_WITH_PIPELINE = true, update from 2.0 to 2.1 (#40825) 
change the dafault value in update to 2.1
 2024-09-14 10:58:07 +08:00
Gabriel
168742deba [followup](branch-2.1) Fuzzy parallel_prepare_threshold (#40793) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-13 22:46:31 +08:00
Calvin Kirs
f9b79c613a [Fix](Job)Replaying logs should not modify the original information of the job (#40474) (#40808) 
…

## Proposed changes
```
        JobExecutionConfiguration jobConfig = new JobExecutionConfiguration();
        jobConfig.setExecuteType(JobExecuteType.INSTANT);
        setJobConfig(jobConfig);
```
- Replaying logs should not modify the original information of the job
- Use the new optimizer to check whether the executed statement is legal

(cherry picked from commit de90051162de7004cf171bbf4d21bd95ff9f3540)

## Proposed changes

Issue Number: #40474
 2024-09-13 20:47:57 +08:00
Sun Chenyang
c2ad828b0c [fix] (Nereids) restrict the execution of match on the scan (#40532) (#40780) 
## Proposed changes

pick from master #40532

<!--Describe your changes.-->
 2024-09-13 20:03:11 +08:00
zy-kkk
9e8e1dac4e [branch-2.1][fix](default value) Avoid default precision value of 0 for default expressions during deserialization (#40806) 
pick  (#40671)

This PR fixes the issue where the precision of default value expressions
was incorrectly set to `0` during deserialization. The `gsonPostProcess`
method no longer assigns a default precision of `0` if the precision is
not provided. Additionally, logic has been added to ensure that
expressions without precision do not incorrectly apply a precision of
`0`.
 2024-09-13 19:59:12 +08:00
Gabriel
3395cd5ce9 [PipelineX](improvement) Prepare tasks in parallel (#40270) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-13 13:34:29 +08:00
morrySnow
51c8b62d1c [opt](Nereids) fix several insert into related issues (#40467) (#40755) 
pick from master #40467

- http_stream TVF should always generate one fragment plan
- http_stream TVF plan should not check root as scan node
- distinguish group_commit TVF with normal insert statement
- index and generate slot should based on type cast base slot
- agg_state could cast from nullable to non-nullable
- colocated and bucket scan range compute should only on scan node
 2024-09-13 10:19:56 +08:00
Calvin Kirs
a7fe2ae901 [branch-2.1][Fix](Job)Replace BlockingWaitStrategy with LiteTimeoutBlockingWaitStrategy to avoid deadlock issues. (#40625) (#40707) 
…

FYI https://issues.apache.org/jira/browse/LOG4J2-1221

- BlockingWaitStrategy is a wait strategy used in the Disruptor
framework that blocks the thread when the ring buffer is full or not yet
available for publishing.

When threads are blocked, they are waiting for space in the ring buffer
to become available, which can lead to potential deadlocks if not
managed properly.
Timeout Handling:

- LiteTimeoutBlockingWaitStrategy provides a timeout for waiting
threads. If the buffer is not ready within the timeout period, the
thread is released, preventing it from being blocked indefinitely.
Reduced Risk of Deadlocks:

- By avoiding indefinite blocking, this strategy reduces the risk of
deadlocks caused by threads waiting on each other. The timeout allows
the system to handle scenarios where resources are temporarily

(cherry picked from commit 087048f2b2f06eae15e80e7ffa209610c9e3f173)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-12 22:42:45 +08:00
morrySnow
967801ca66 [feature](Nereids) support query on ES table in internal catalog (#40449) (#40754) 
pick from master #40449
 2024-09-12 20:01:48 +08:00
morrySnow
fedadbba6e [opt](Nereids) support search from override udfs with same arity (#40432) (#40751) 
pick from master #40432

create alias function f1(int) with parameter(id) as abs(id); create
alias function f1(string) with parameter(id) as substr(id, 2); select
f1('1'); -- bind on f1(string)
select f1(1);   -- bind on f1(int)

test case already existed in P0
 2024-09-12 19:58:49 +08:00
wuwenchi
4b7b43b5ca [bugfix](hive/iceberg)align with Hive insert overwrite table functionality (#39840) (#40724) 
bp #39840
 2024-09-12 19:20:15 +08:00
wuwenchi
23b21fcebf [bugfix](hms)add default value check for hms catalog with dlf (#40336) (#40723) 
## Proposed changes

bp #40336
 2024-09-12 19:19:47 +08:00
wuwenchi
3d2ac60bae [bugfix](hive)Delete the temporarily created folder (#40424) (#40722) 
## Proposed changes

bp: #40424
 2024-09-12 19:18:35 +08:00
starocean999
3484a6cfe3 [fix](nereids)let query be able to forwarded to master (#40119) (#40725) 
pick from master #40119
 2024-09-12 17:09:36 +08:00
starocean999
0f8176dee0 [fix](nereids) build agg for random distributed agg table in bindRelation phase (#40181) (#40702) 
pick from master #40181
 2024-09-12 14:08:50 +08:00
zclllhhjj
e2dc7544dd [branch-2.1](iot) Fix unexpected error when iot auto detect get no data (#40657) (#40691) 
pick https://github.com/apache/doris/pull/40657
 2024-09-12 14:03:43 +08:00
Vallish Pai
3604d63184 [Branch 2.1] backport systable PR (#34384,#40153,#40456,#40455,#40568) (#40687) 
backport
https://github.com/apache/doris/pull/40568
https://github.com/apache/doris/pull/40455
https://github.com/apache/doris/pull/40456
https://github.com/apache/doris/pull/40153
https://github.com/apache/doris/pull/34384

Test result:
2024-09-11 11:00:45.618 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.619 INFO [suite-thread-1] (Suite.groovy:359) -
Execute sql: REVOKE SELECT_PRIV ON
test_partitions_schema_db.duplicate_table FROM partitions_user
2024-09-11 11:00:45.625 INFO [suite-thread-1] (SuiteContext.groovy:299)
- Create new connection for user 'partitions_user'
2024-09-11 11:00:45.632 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
from information_schema.partitions where
table_schema="test_partitions_schema_db" order by
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
2024-09-11 11:00:45.644 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.645 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_partitions_schema in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy
succeed
2024-09-11 11:00:45.652 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:01:10.321 INFO [main] (RegressionTest.groovy:380) -
Success suites:

/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy:
group=default,p0, name=test_partitions_schema
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:459) - All
suites success.
 ____   _    ____ ____  _____ ____
|  _ \ / \  / ___/ ___|| ____|  _ \
| |_) / _ \ \___ \___ \|  _| | | | |
|  __/ ___ \ ___) |__) | |___| |_| |
|_| /_/   \_\____/____/|_____|____/

2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:119) - Test
finished


2024-09-11 11:03:00.712 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select * from
information_schema.table_options ORDER BY
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,TABLE_MODEL,TABLE_MODEL_KEY,DISTRIBUTE_KEY,DISTRIBUTE_TYPE,BUCKETS_NUM,PARTITION_NUM;
2024-09-11 11:03:00.729 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:03:00.731 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_table_options in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy
succeed
2024-09-11 11:03:04.817 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:03:28.741 INFO [main] (RegressionTest.groovy:380) -
Success suites:

/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy:
group=default,p0, name=test_table_options
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:459) - All
suites success.
 ____   _    ____ ____  _____ ____
|  _ \ / \  / ___/ ___|| ____|  _ \
| |_) / _ \ \___ \___ \|  _| | | | |
|  __/ ___ \ ___) |__) | |___| |_| |
|_| /_/   \_\____/____/|_____|____/

2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:119) - Test
finished


*************************** 7. row ***************************
             PartitionId: 18035
           PartitionName: p100
          VisibleVersion: 2
      VisibleVersionTime: 2024-09-11 10:59:28
                   State: NORMAL
            PartitionKey: col_1
Range: [types: [INT]; keys: [83647]; ..types: [INT]; keys: [2147483647];
)
         DistributionKey: pk
                 Buckets: 10
          ReplicationNum: 1
           StorageMedium: HDD
            CooldownTime: 9999-12-31 15:59:59
     RemoteStoragePolicy: 
LastConsistencyCheckTime: NULL
                DataSize: 2.872 KB
              IsInMemory: false
       ReplicaAllocation: tag.location.default: 1
               IsMutable: true
      SyncWithBaseTables: true
            UnsyncTables: NULL
        CommittedVersion: 2
                RowCount: 4
7 rows in set (0.01 sec)

---------

Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
 2024-09-12 11:50:09 +08:00
camby
361a59dec8 [feature](aes_encrypt) support GCM mode for aes_encrypt and aes_decrypt (#40004) (#40672) 
pick #40004 to branch-2.1
 2024-09-11 23:28:28 +08:00
zclllhhjj
bf156d1665 [branch-2.1](DDL) check illegal partition exprs (#40158) (#40651) 
pick https://github.com/apache/doris/pull/40158
 2024-09-11 22:37:55 +08:00
zclllhhjj
52e13c9e6c [branch-2.1](nereids) remove interface Nondeterministic and fix random/random_bytes foldable properties (#39944) (#40650) 
pick https://github.com/apache/doris/pull/39944
 2024-09-11 17:27:52 +08:00
zclllhhjj
d554f600bc [branch-2.1](partition) Support use Auto and Dynamic partition at the same time (#39580) (#40649) 
pick https://github.com/apache/doris/pull/39580
 2024-09-11 15:35:20 +08:00
zclllhhjj
3246baa451 [branch-2.1](function) Refine crypto functions signature to fix wrong result(#40285) (#40648) 
pick https://github.com/apache/doris/pull/40285
 2024-09-11 15:32:19 +08:00
morrySnow
1c27724766 [chore](audit) let line comment work well (#40602) 2024-09-11 15:09:27 +08:00
zhiqiang
ab8f9f7bf6 [fix](histogram) Fix histgrom nullable properties on branch-2.1 (#40638) 
nullable property of histogram on master is changed by
https://github.com/apache/doris/pull/37330
pick it to branch-2.1
related change on 2.1: https://github.com/apache/doris/pull/38608,
#38608 relies on AlwaysNotNullable property.
 2024-09-11 10:08:25 +08:00
yujun
d3dccbc1dd [chore](show partitions) show partitions print commit version #28274 (#40594) 
cherry pick from #28274
 2024-09-10 18:04:24 +08:00
yiguolei
4e453dc1bb Revert "[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457)" (#40616) 
Reverts apache/doris#40540
 2024-09-10 17:17:13 +08:00
Jibing-Li
e43e6e2bba [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) (#40540) 
backport: https://github.com/apache/doris/pull/40457
 2024-09-10 12:55:53 +08:00
zhangdong
db8fb66dba [fix](mtmv)fix nested mtmv not refresh (#40433) (#40560) 
pick: https://github.com/apache/doris/pull/40433
 2024-09-10 11:51:41 +08:00
walter
05f82df1f1 [fix](restore) update is_being_synced properties #40194 (#40556) 
cherry pick from #40194
 2024-09-10 11:15:50 +08:00
hui lai
8eda15ae16 [opt](routine load) support routine load perceived schema change (#39412) (#40508) 
pick #39412

At present, if the table structure changes, the routine load cannot
perceive it. As a long-running load, it should be able to perceive the
changes in the table structure.
 2024-09-10 11:05:58 +08:00
walter
68d1064aa9 [chore](persist) reduce truncate table operation log size #40422 (#40555) 
cherry pick from #40422
 2024-09-09 20:01:07 +08:00
abmdocrt
8f37eccbf2 [Cherry-pick](branch-2.1) Pick "[Featrue](default value) Support bitmap_empty default value (#40364)" (#40487) 
## Proposed changes

Pick #40364 

<!--Describe your changes.-->
 2024-09-09 16:57:38 +08:00
Calvin Kirs
2235c1abd3 [Feat](OSS)Decouple COS and OBS dependencies with optional inclusion mechanism (#39831) 
…

improve the flexibility of the project by decoupling direct dependencies
on the hadoop-cos and hadoop-huaweicloud libraries. These changes allow
users to control whether COS and OBS dependencies are included in the
final build, enabling a more customizable setup.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-09 16:50:12 +08:00
hui lai
9c9827b92e [chore](routine load) make error msg clear if routine load name illegal (#40037) (#40510) 
pick (#40037)
 2024-09-09 15:47:18 +08:00
qiye
5f4d1931ff [feature](ES Catalog)Add FE open API for ES Catalog (#40444) (#40526) 
bp #40444
 2024-09-09 14:25:43 +08:00
hui lai
4b72a4c04e [improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) (#40509) 
pick (#39975)

We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance: Kafka's
data comes in batches of very small data every very short time, which
leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
 2024-09-09 11:36:22 +08:00
zhangdong
01a651e573 [enhance](mtmv) During cache generation, no longer hold the write loc… (#40489) 
…k for mtmv (#40402)

pick: https://github.com/apache/doris/pull/40402
 2024-09-09 11:35:22 +08:00
seawinde
a67f20f073 [opt](mtmv) Support to contain select constant clause when create async materialized view (#40244) (#40435) 
## Proposed changes
commitId: 518a0fc0
pr: https://github.com/apache/doris/pull/40244
 2024-09-09 11:23:15 +08:00
zhangdong
4084ff2c4a [enhance](mtmv)Optimize the speed of obtaining the last update time o… (#40421) 
…f Hive (#40169)

pick: https://github.com/apache/doris/pull/40169
 2024-09-09 11:20:40 +08:00
zhangdong
fc4c2aed64 [enhance](mtmv) Prohibit creating materialized views in external cata… (#40420) 
…logs (#39557)

pick: https://github.com/apache/doris/pull/39557
 2024-09-09 11:16:40 +08:00
zhangdong
ecb75c2e7d [fix](mtmv) Mtmv support set both immediate and starttime (#39573) (#40418) 
pick: https://github.com/apache/doris/pull/39573
 2024-09-09 11:13:51 +08:00
zhangdong
a3eba2aad5 [fix](auth) ordinary users can see the processes of other users (#39747) (#40415) 
pick: https://github.com/apache/doris/pull/39747
 2024-09-09 11:13:18 +08:00
zhangdong
e1e09badfa [chore](mtmv)add plan log for UpdateMvByPartitionCommand (#40072) (#40413) 
pick: https://github.com/apache/doris/pull/40072
 2024-09-09 11:09:58 +08:00
zhangdong
93a2518d12 [fix](auth)Fix some issues with incorrect permission verification (#3… (#40410) 
…9726)

pick: https://github.com/apache/doris/pull/39726
 2024-09-09 11:04:49 +08:00
Siyang Tang
d373ca7da1 [enhancement](schema-change) Support light schema change on hash columns and agg key columns with varchar type to change length (#39319) (#40236) 
## Proposed changes

1. Schema change should rebuild distribution info after modifying
columns, especially distribution columns. Or it may cause dynamic
partition failed when checking distribution columns' equality.
2. Support hash key columns to do light schema change. For unique key or
dup key columns, could not be enabled temporarily due to some historical
reasons. See #39798 .
 2024-09-09 10:55:32 +08:00
zhiqiang
a963709fed [opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 (#40357) 
cherry pick from #39927
 2024-09-09 10:42:19 +08:00
zy-kkk
962c382077 [fix](jdbc catalog) Fix type recognition error when using query tvf to query doris (#40481) 
pick  (#40122)

Using string to match Doris type will not work with query tvf, so use
field matching instead
 2024-09-06 19:30:32 +08:00
qiye
8104b992d1 [fix](ES Catalog)Do not extract doc_values of field with ignore_above setting (#40314) (#40464) 
bp #40314
 2024-09-06 16:25:30 +08:00