e2dc7544dd
[branch-2.1](iot) Fix unexpected error when iot auto detect get no data ( #40657 ) ( #40691 )
...
pick https://github.com/apache/doris/pull/40657
2024-09-12 14:03:43 +08:00
3604d63184
[Branch 2.1] backport systable PR (#34384,#40153,#40456,#40455,#40568) ( #40687 )
...
backport
https://github.com/apache/doris/pull/40568
https://github.com/apache/doris/pull/40455
https://github.com/apache/doris/pull/40456
https://github.com/apache/doris/pull/40153
https://github.com/apache/doris/pull/34384
Test result:
2024-09-11 11:00:45.618 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.619 INFO [suite-thread-1] (Suite.groovy:359) -
Execute sql: REVOKE SELECT_PRIV ON
test_partitions_schema_db.duplicate_table FROM partitions_user
2024-09-11 11:00:45.625 INFO [suite-thread-1] (SuiteContext.groovy:299)
- Create new connection for user 'partitions_user'
2024-09-11 11:00:45.632 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
from information_schema.partitions where
table_schema="test_partitions_schema_db" order by
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
2024-09-11 11:00:45.644 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.645 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_partitions_schema in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy
succeed
2024-09-11 11:00:45.652 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:01:10.321 INFO [main] (RegressionTest.groovy:380) -
Success suites:
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy:
group=default,p0, name=test_partitions_schema
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:459) - All
suites success.
____ _ ____ ____ _____ ____
| _ \ / \ / ___/ ___|| ____| _ \
| |_) / _ \ \___ \___ \| _| | | | |
| __/ ___ \ ___) |__) | |___| |_| |
|_| /_/ \_\____/____/|_____|____/
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:119) - Test
finished
2024-09-11 11:03:00.712 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select * from
information_schema.table_options ORDER BY
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,TABLE_MODEL,TABLE_MODEL_KEY,DISTRIBUTE_KEY,DISTRIBUTE_TYPE,BUCKETS_NUM,PARTITION_NUM;
2024-09-11 11:03:00.729 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:03:00.731 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_table_options in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy
succeed
2024-09-11 11:03:04.817 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:03:28.741 INFO [main] (RegressionTest.groovy:380) -
Success suites:
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy:
group=default,p0, name=test_table_options
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:459) - All
suites success.
____ _ ____ ____ _____ ____
| _ \ / \ / ___/ ___|| ____| _ \
| |_) / _ \ \___ \___ \| _| | | | |
| __/ ___ \ ___) |__) | |___| |_| |
|_| /_/ \_\____/____/|_____|____/
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:119) - Test
finished
*************************** 7. row ***************************
PartitionId: 18035
PartitionName: p100
VisibleVersion: 2
VisibleVersionTime: 2024-09-11 10:59:28
State: NORMAL
PartitionKey: col_1
Range: [types: [INT]; keys: [83647]; ..types: [INT]; keys: [2147483647];
)
DistributionKey: pk
Buckets: 10
ReplicationNum: 1
StorageMedium: HDD
CooldownTime: 9999-12-31 15:59:59
RemoteStoragePolicy:
LastConsistencyCheckTime: NULL
DataSize: 2.872 KB
IsInMemory: false
ReplicaAllocation: tag.location.default: 1
IsMutable: true
SyncWithBaseTables: true
UnsyncTables: NULL
CommittedVersion: 2
RowCount: 4
7 rows in set (0.01 sec)
---------
Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com >
2024-09-12 11:50:09 +08:00
361a59dec8
[feature](aes_encrypt) support GCM mode for aes_encrypt and aes_decrypt ( #40004 ) ( #40672 )
...
pick #40004 to branch-2.1
2024-09-11 23:28:28 +08:00
bf156d1665
[branch-2.1](DDL) check illegal partition exprs ( #40158 ) ( #40651 )
...
pick https://github.com/apache/doris/pull/40158
2024-09-11 22:37:55 +08:00
52e13c9e6c
[branch-2.1](nereids) remove interface Nondeterministic and fix random/random_bytes foldable properties ( #39944 ) ( #40650 )
...
pick https://github.com/apache/doris/pull/39944
2024-09-11 17:27:52 +08:00
d554f600bc
[branch-2.1](partition) Support use Auto and Dynamic partition at the same time ( #39580 ) ( #40649 )
...
pick https://github.com/apache/doris/pull/39580
2024-09-11 15:35:20 +08:00
3246baa451
[branch-2.1](function) Refine crypto functions signature to fix wrong result( #40285 ) ( #40648 )
...
pick https://github.com/apache/doris/pull/40285
2024-09-11 15:32:19 +08:00
1c27724766
[chore](audit) let line comment work well ( #40602 )
2024-09-11 15:09:27 +08:00
ab8f9f7bf6
[fix](histogram) Fix histgrom nullable properties on branch-2.1 ( #40638 )
...
nullable property of histogram on master is changed by
https://github.com/apache/doris/pull/37330
pick it to branch-2.1
related change on 2.1: https://github.com/apache/doris/pull/38608 ,
#38608 relies on AlwaysNotNullable property.
2024-09-11 10:08:25 +08:00
d3dccbc1dd
[chore](show partitions) show partitions print commit version #28274 ( #40594 )
...
cherry pick from #28274
2024-09-10 18:04:24 +08:00
4e453dc1bb
Revert "[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. ( #40457 )" ( #40616 )
...
Reverts apache/doris#40540
2024-09-10 17:17:13 +08:00
e43e6e2bba
[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. ( #40457 ) ( #40540 )
...
backport: https://github.com/apache/doris/pull/40457
2024-09-10 12:55:53 +08:00
db8fb66dba
[fix](mtmv)fix nested mtmv not refresh ( #40433 ) ( #40560 )
...
pick: https://github.com/apache/doris/pull/40433
2024-09-10 11:51:41 +08:00
05f82df1f1
[fix](restore) update is_being_synced properties #40194 ( #40556 )
...
cherry pick from #40194
2024-09-10 11:15:50 +08:00
8eda15ae16
[opt](routine load) support routine load perceived schema change ( #39412 ) ( #40508 )
...
pick #39412
At present, if the table structure changes, the routine load cannot
perceive it. As a long-running load, it should be able to perceive the
changes in the table structure.
2024-09-10 11:05:58 +08:00
978fcf12b6
[Chore](dependnecies)remove javax.el and upgrade jackson to 2.16.0 ( #38843 ) ( #39389 )
...
…
## Proposed changes
javax.el is the API for Java Expression Language, which provides a
simple and flexible way for Java Web applications to access and
manipulate data. EL expressions are commonly used in JSP pages, but we
are not involved in their use here, so I removed it.
- upgrade jackson to 2.16.0
(cherry picked from commit 33a0ea5565c2e0d7e5909ca3b512fb0b539f05f4)
#38843
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-09 21:58:47 +08:00
68d1064aa9
[chore](persist) reduce truncate table operation log size #40422 ( #40555 )
...
cherry pick from #40422
2024-09-09 20:01:07 +08:00
8f37eccbf2
[Cherry-pick](branch-2.1) Pick "[Featrue](default value) Support bitmap_empty default value ( #40364 )" ( #40487 )
...
## Proposed changes
Pick #40364
<!--Describe your changes.-->
2024-09-09 16:57:38 +08:00
2235c1abd3
[Feat](OSS)Decouple COS and OBS dependencies with optional inclusion mechanism ( #39831 )
...
…
improve the flexibility of the project by decoupling direct dependencies
on the hadoop-cos and hadoop-huaweicloud libraries. These changes allow
users to control whether COS and OBS dependencies are included in the
final build, enabling a more customizable setup.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-09 16:50:12 +08:00
9c9827b92e
[chore](routine load) make error msg clear if routine load name illegal ( #40037 ) ( #40510 )
...
pick (#40037 )
2024-09-09 15:47:18 +08:00
5f4d1931ff
[feature](ES Catalog)Add FE open API for ES Catalog ( #40444 ) ( #40526 )
...
bp #40444
2024-09-09 14:25:43 +08:00
4b72a4c04e
[improve](routine load) delay schedule EOF tasks to avoid too many small transactions ( #39975 ) ( #40509 )
...
pick (#39975 )
We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance: Kafka's
data comes in batches of very small data every very short time, which
leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.
To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
2024-09-09 11:36:22 +08:00
01a651e573
[enhance](mtmv) During cache generation, no longer hold the write loc… ( #40489 )
...
…k for mtmv (#40402 )
pick: https://github.com/apache/doris/pull/40402
2024-09-09 11:35:22 +08:00
a67f20f073
[opt](mtmv) Support to contain select constant clause when create async materialized view ( #40244 ) ( #40435 )
...
## Proposed changes
commitId: 518a0fc0
pr: https://github.com/apache/doris/pull/40244
2024-09-09 11:23:15 +08:00
4084ff2c4a
[enhance](mtmv)Optimize the speed of obtaining the last update time o… ( #40421 )
...
…f Hive (#40169 )
pick: https://github.com/apache/doris/pull/40169
2024-09-09 11:20:40 +08:00
fc4c2aed64
[enhance](mtmv) Prohibit creating materialized views in external cata… ( #40420 )
...
…logs (#39557 )
pick: https://github.com/apache/doris/pull/39557
2024-09-09 11:16:40 +08:00
ecb75c2e7d
[fix](mtmv) Mtmv support set both immediate and starttime ( #39573 ) ( #40418 )
...
pick: https://github.com/apache/doris/pull/39573
2024-09-09 11:13:51 +08:00
a3eba2aad5
[fix](auth) ordinary users can see the processes of other users ( #39747 ) ( #40415 )
...
pick: https://github.com/apache/doris/pull/39747
2024-09-09 11:13:18 +08:00
e1e09badfa
[chore](mtmv)add plan log for UpdateMvByPartitionCommand ( #40072 ) ( #40413 )
...
pick: https://github.com/apache/doris/pull/40072
2024-09-09 11:09:58 +08:00
93a2518d12
[fix](auth)Fix some issues with incorrect permission verification (#3… ( #40410 )
...
…9726)
pick: https://github.com/apache/doris/pull/39726
2024-09-09 11:04:49 +08:00
d373ca7da1
[enhancement](schema-change) Support light schema change on hash columns and agg key columns with varchar type to change length ( #39319 ) ( #40236 )
...
## Proposed changes
1. Schema change should rebuild distribution info after modifying
columns, especially distribution columns. Or it may cause dynamic
partition failed when checking distribution columns' equality.
2. Support hash key columns to do light schema change. For unique key or
dup key columns, could not be enabled temporarily due to some historical
reasons. See #39798 .
2024-09-09 10:55:32 +08:00
a963709fed
[opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 ( #40357 )
...
cherry pick from #39927
2024-09-09 10:42:19 +08:00
962c382077
[fix](jdbc catalog) Fix type recognition error when using query tvf to query doris ( #40481 )
...
pick (#40122 )
Using string to match Doris type will not work with query tvf, so use
field matching instead
2024-09-06 19:30:32 +08:00
8104b992d1
[fix](ES Catalog)Do not extract doc_values of field with ignore_above setting ( #40314 ) ( #40464 )
...
bp #40314
2024-09-06 16:25:30 +08:00
88750e7e5d
[cherry-pick](branch-2.1) Pick "[Fix](group commit) Fix table not found fault when disable group commit ( #39731 )" ( #40323 )
...
<!--Describe your changes.-->
## Proposed changes
Pick #39731
<!--Describe your changes.-->
2024-09-05 21:39:15 +08:00
0928c9c6ed
[fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input #40267 ( #40358 )
...
cherry pick from #40267
2024-09-05 19:51:01 +08:00
961d2c9af5
[feat](job)Internal job cancellation immediately and the strong association with the STARTS parameter ( #36805 ) ( #38110 )
...
…
## Proposed changes
For internal tasks, such as MTMV, the start time may already be set, or
the time may be adjusted immediately.
<!--Describe your changes.-->
(cherry picked from commit 904a6c0fc1a804520285533de874fe4d0ffff2c1)
## Proposed changes
Issue Number: close #36805
<!--Describe your changes.-->
2024-09-05 16:28:35 +08:00
52393f829d
[cherry-pick](branch-2.1) Fix some group commit fault ( #40319 )
...
## Proposed changes
Pick #39986 #40120
<!--Describe your changes.-->
2024-09-05 15:26:59 +08:00
40d10bdb0a
Revert "Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value"" ( #40411 )
...
Reverts apache/doris#40359
2024-09-05 14:59:01 +08:00
c6771da240
[fix](nereids) fix bug for A>n, where A.max is infinity #39936 ( #40368 )
...
## Proposed changes
pick #39936
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-05 08:33:04 +08:00
08fb91f161
Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value" ( #40359 )
...
Reverts apache/doris#40283
2024-09-04 17:55:25 +08:00
b4beec8ea8
[fix](OrcWriter) fix be core when upgrading BE without upgrading FE ( #40303 )
...
bp: #40282
2024-09-04 10:24:41 +08:00
db06c678a3
[Fix](planner) legacy planner repeat group by has grouping report error ( #40281 ) ( #40305 )
...
cherry-pick #40281 to branch-2.1
2024-09-04 08:50:21 +08:00
3cf5d15b49
[fix](fe) avoid setting the value of batch_size excessively high ( #35941 ) ( #40312 )
...
Issue Number: close #xxx
pick #35941
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-03 19:49:26 +08:00
c49b345342
[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value ( #40283 )
...
## Proposed changes
Pick #40104 #40197
<!--Describe your changes.-->
2024-09-03 11:19:05 +08:00
92752b90e7
[feature](metacache) add system table catalog_meta_cache_statistics #40155 ( #40210 )
...
bp #40155
2024-09-02 23:23:35 +08:00
1527099e35
[opt](Nereids) forbid one step topn with distribute as child ( #40066 ) ( #40278 )
...
pick from master #40066
2024-09-02 22:26:26 +08:00
a5efdc6085
[improve](statistics)Clean expired TableStatsMeta. ( #39779 ) ( #40253 )
...
backport: https://github.com/apache/doris/pull/39779
2024-09-02 22:13:42 +08:00
8ff068eb7c
[feature](statistics) Support get row count for JDBC external table. ( #38889 ) ( #40248 )
...
backport: https://github.com/apache/doris/pull/38889
2024-09-02 16:39:42 +08:00
98e039ae31
[pick](Variant) truncate key when column is variant type when crea… ( #40238 )
...
#39988
2024-09-02 13:54:04 +08:00
