database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,138 Commits 1 Branch 0 Tags
23b21fcebfde3f305d7e4efd310cb5fadc5e9d90
Commit Graph

20138 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wuwenchi
23b21fcebf [bugfix](hms)add default value check for hms catalog with dlf (#40336) (#40723) 
## Proposed changes

bp #40336
 2024-09-12 19:19:47 +08:00
wuwenchi
3d2ac60bae [bugfix](hive)Delete the temporarily created folder (#40424) (#40722) 
## Proposed changes

bp: #40424
 2024-09-12 19:18:35 +08:00
camby
ed76b0c5f5 [fix](compile) failed on MacOS because size_t is not uint64_t on MacOS (#39296) (#40720) 
pick #39296 to branch-2.1

Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
 2024-09-12 18:47:59 +08:00
starocean999
3484a6cfe3 [fix](nereids)let query be able to forwarded to master (#40119) (#40725) 
pick from master #40119
 2024-09-12 17:09:36 +08:00
yagagagaga
b192e89ec1 [chore](conf) Specify UTF8 as the default charset. (#40669) 
## Proposed changes

pick from  #39521

<!--Describe your changes.-->
 2024-09-12 14:35:00 +08:00
starocean999
0f8176dee0 [fix](nereids) build agg for random distributed agg table in bindRelation phase (#40181) (#40702) 
pick from master #40181
 2024-09-12 14:08:50 +08:00
zclllhhjj
e2dc7544dd [branch-2.1](iot) Fix unexpected error when iot auto detect get no data (#40657) (#40691) 
pick https://github.com/apache/doris/pull/40657
 2024-09-12 14:03:43 +08:00
Vallish Pai
3604d63184 [Branch 2.1] backport systable PR (#34384,#40153,#40456,#40455,#40568) (#40687) 
backport
https://github.com/apache/doris/pull/40568
https://github.com/apache/doris/pull/40455
https://github.com/apache/doris/pull/40456
https://github.com/apache/doris/pull/40153
https://github.com/apache/doris/pull/34384

Test result:
2024-09-11 11:00:45.618 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.619 INFO [suite-thread-1] (Suite.groovy:359) -
Execute sql: REVOKE SELECT_PRIV ON
test_partitions_schema_db.duplicate_table FROM partitions_user
2024-09-11 11:00:45.625 INFO [suite-thread-1] (SuiteContext.groovy:299)
- Create new connection for user 'partitions_user'
2024-09-11 11:00:45.632 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
from information_schema.partitions where
table_schema="test_partitions_schema_db" order by
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
2024-09-11 11:00:45.644 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.645 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_partitions_schema in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy
succeed
2024-09-11 11:00:45.652 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:01:10.321 INFO [main] (RegressionTest.groovy:380) -
Success suites:

/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy:
group=default,p0, name=test_partitions_schema
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:459) - All
suites success.
 ____   _    ____ ____  _____ ____
|  _ \ / \  / ___/ ___|| ____|  _ \
| |_) / _ \ \___ \___ \|  _| | | | |
|  __/ ___ \ ___) |__) | |___| |_| |
|_| /_/   \_\____/____/|_____|____/

2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:119) - Test
finished


2024-09-11 11:03:00.712 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select * from
information_schema.table_options ORDER BY
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,TABLE_MODEL,TABLE_MODEL_KEY,DISTRIBUTE_KEY,DISTRIBUTE_TYPE,BUCKETS_NUM,PARTITION_NUM;
2024-09-11 11:03:00.729 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:03:00.731 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_table_options in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy
succeed
2024-09-11 11:03:04.817 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:03:28.741 INFO [main] (RegressionTest.groovy:380) -
Success suites:

/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy:
group=default,p0, name=test_table_options
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:459) - All
suites success.
 ____   _    ____ ____  _____ ____
|  _ \ / \  / ___/ ___|| ____|  _ \
| |_) / _ \ \___ \___ \|  _| | | | |
|  __/ ___ \ ___) |__) | |___| |_| |
|_| /_/   \_\____/____/|_____|____/

2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:119) - Test
finished


*************************** 7. row ***************************
             PartitionId: 18035
           PartitionName: p100
          VisibleVersion: 2
      VisibleVersionTime: 2024-09-11 10:59:28
                   State: NORMAL
            PartitionKey: col_1
Range: [types: [INT]; keys: [83647]; ..types: [INT]; keys: [2147483647];
)
         DistributionKey: pk
                 Buckets: 10
          ReplicationNum: 1
           StorageMedium: HDD
            CooldownTime: 9999-12-31 15:59:59
     RemoteStoragePolicy: 
LastConsistencyCheckTime: NULL
                DataSize: 2.872 KB
              IsInMemory: false
       ReplicaAllocation: tag.location.default: 1
               IsMutable: true
      SyncWithBaseTables: true
            UnsyncTables: NULL
        CommittedVersion: 2
                RowCount: 4
7 rows in set (0.01 sec)

---------

Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
 2024-09-12 11:50:09 +08:00
camby
361a59dec8 [feature](aes_encrypt) support GCM mode for aes_encrypt and aes_decrypt (#40004) (#40672) 
pick #40004 to branch-2.1
 2024-09-11 23:28:28 +08:00
zclllhhjj
bf156d1665 [branch-2.1](DDL) check illegal partition exprs (#40158) (#40651) 
pick https://github.com/apache/doris/pull/40158
 2024-09-11 22:37:55 +08:00
zzzxl
ebe031c019 [fix](inverted index) Fix match_regexp to correctly handle empty string patterns (#40659) 
https://github.com/apache/doris/pull/39503
 2024-09-11 18:10:33 +08:00
zclllhhjj
52e13c9e6c [branch-2.1](nereids) remove interface Nondeterministic and fix random/random_bytes foldable properties (#39944) (#40650) 
pick https://github.com/apache/doris/pull/39944
 2024-09-11 17:27:52 +08:00
qiye
8708fae420 [fix](ES Catalog)Support parse single value for array column (#40614) (#40660) 
bp #40614
 2024-09-11 17:26:48 +08:00
zclllhhjj
d554f600bc [branch-2.1](partition) Support use Auto and Dynamic partition at the same time (#39580) (#40649) 
pick https://github.com/apache/doris/pull/39580
 2024-09-11 15:35:20 +08:00
zclllhhjj
279c58fbc7 [branch-2.1](auto-partition) Re-add deduplication to auto partition rpc (#40580) (#40652) 
pick https://github.com/apache/doris/pull/40580
 2024-09-11 15:34:17 +08:00
zclllhhjj
3246baa451 [branch-2.1](function) Refine crypto functions signature to fix wrong result(#40285) (#40648) 
pick https://github.com/apache/doris/pull/40285
 2024-09-11 15:32:19 +08:00
yujun
66421c4270 [chore](regression test) mv test_report_version_missing to p1 #40184 (#40646) 
cherry pick from #40184
 2024-09-11 15:28:23 +08:00
Sun Chenyang
86647df45b [fix] (inverted index) fix error result in complex compound expr (#40630) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-11 15:27:40 +08:00
morrySnow
1c27724766 [chore](audit) let line comment work well (#40602) 2024-09-11 15:09:27 +08:00
zhiqiang
b0ab02bcf4 [fix](scanner) Fix deadlock when scanner submit failed #40495 (#40640) 
cherry pick from #40495
 2024-09-11 10:15:29 +08:00
zhiqiang
ab8f9f7bf6 [fix](histogram) Fix histgrom nullable properties on branch-2.1 (#40638) 
nullable property of histogram on master is changed by
https://github.com/apache/doris/pull/37330
pick it to branch-2.1
related change on 2.1: https://github.com/apache/doris/pull/38608,
#38608 relies on AlwaysNotNullable property.
 2024-09-11 10:08:25 +08:00
Gabriel
d6f459903d [opt](pipeline) Distribute data evenly for passthrough local exchange (#40637) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-11 09:58:14 +08:00
zhiqiang
725886d365 [fix](regression-test) Fix scanner profile test failed occasionally #40566 (#40639) 
cherry pick from #40566
 2024-09-11 09:56:31 +08:00
zhannngchen
023317e8a0 [fix](ut) fix SegmentTest and SegmentMoWTest asan core (#40287) (#40622) 
## Proposed changes

pick #40287
 2024-09-10 18:37:00 +08:00
yujun
d3dccbc1dd [chore](show partitions) show partitions print commit version #28274 (#40594) 
cherry pick from #28274
 2024-09-10 18:04:24 +08:00
yiguolei
4e453dc1bb Revert "[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457)" (#40616) 
Reverts apache/doris#40540
 2024-09-10 17:17:13 +08:00
Vallish Pai
6f239ebcdd Branch 2.1 clang format error fix (#40609) 
## Proposed changes

bp #40581
 2024-09-10 16:29:20 +08:00
zhiqiang
7d80630757 [opt](scanner profile) More counter for scanner #40144 (#40571) 
cherry pick from #40144
 2024-09-10 16:08:55 +08:00
Jibing-Li
e43e6e2bba [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) (#40540) 
backport: https://github.com/apache/doris/pull/40457
 2024-09-10 12:55:53 +08:00
zhiqiang
6961c95eca [chore](error msg) More convenient error msg when function not found. #40296 (#40570) 
cherry pick from #40296
 2024-09-10 12:03:12 +08:00
zhangdong
db8fb66dba [fix](mtmv)fix nested mtmv not refresh (#40433) (#40560) 
pick: https://github.com/apache/doris/pull/40433
 2024-09-10 11:51:41 +08:00
walter
05f82df1f1 [fix](restore) update is_being_synced properties #40194 (#40556) 
cherry pick from #40194
 2024-09-10 11:15:50 +08:00
hui lai
8eda15ae16 [opt](routine load) support routine load perceived schema change (#39412) (#40508) 
pick #39412

At present, if the table structure changes, the routine load cannot
perceive it. As a long-running load, it should be able to perceive the
changes in the table structure.
 2024-09-10 11:05:58 +08:00
airborne12
185353e890 [Fix](inverted index) gc TEMP colum when next_batch in segment iterator (#40563) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-10 09:56:03 +08:00
Calvin Kirs
978fcf12b6 [Chore](dependnecies)remove javax.el and upgrade jackson to 2.16.0 (#38843) (#39389) 
…

## Proposed changes
javax.el is the API for Java Expression Language, which provides a
simple and flexible way for Java Web applications to access and
manipulate data. EL expressions are commonly used in JSP pages, but we
are not involved in their use here, so I removed it.
- upgrade jackson to 2.16.0

(cherry picked from commit 33a0ea5565c2e0d7e5909ca3b512fb0b539f05f4)

#38843

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-09 21:58:47 +08:00
Xinyi Zou
354967c09f [branch-2.1](memory) pick reserve memory and workload group (#40543) 
1. pick #38494
2. pick #39862
3. remove vdata_stream_test, master has been removed
 2024-09-09 21:16:06 +08:00
walter
68d1064aa9 [chore](persist) reduce truncate table operation log size #40422 (#40555) 
cherry pick from #40422
 2024-09-09 20:01:07 +08:00
lihangyu
f69063ea87 [Fix](Variant) use uinque id to access column reader (#39841) (#40269) 
#39841
#40295
 2024-09-09 18:01:12 +08:00
abmdocrt
8f37eccbf2 [Cherry-pick](branch-2.1) Pick "[Featrue](default value) Support bitmap_empty default value (#40364)" (#40487) 
## Proposed changes

Pick #40364 

<!--Describe your changes.-->
 2024-09-09 16:57:38 +08:00
Calvin Kirs
2235c1abd3 [Feat](OSS)Decouple COS and OBS dependencies with optional inclusion mechanism (#39831) 
…

improve the flexibility of the project by decoupling direct dependencies
on the hadoop-cos and hadoop-huaweicloud libraries. These changes allow
users to control whether COS and OBS dependencies are included in the
final build, enabling a more customizable setup.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-09 16:50:12 +08:00
bobhan1
44a7efff4f [branch-2.1] Picks "[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322" (#40383) 
picks https://github.com/apache/doris/pull/40322
 2024-09-09 16:32:24 +08:00
hui lai
9c9827b92e [chore](routine load) make error msg clear if routine load name illegal (#40037) (#40510) 
pick (#40037)
 2024-09-09 15:47:18 +08:00
qiye
314f6ae823 [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) (#40521) 
bp #40385
 2024-09-09 14:29:21 +08:00
qiye
5f4d1931ff [feature](ES Catalog)Add FE open API for ES Catalog (#40444) (#40526) 
bp #40444
 2024-09-09 14:25:43 +08:00
hui lai
4b72a4c04e [improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) (#40509) 
pick (#39975)

We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance: Kafka's
data comes in batches of very small data every very short time, which
leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
 2024-09-09 11:36:22 +08:00
zhangdong
01a651e573 [enhance](mtmv) During cache generation, no longer hold the write loc… (#40489) 
…k for mtmv (#40402)

pick: https://github.com/apache/doris/pull/40402
 2024-09-09 11:35:22 +08:00
Kaijie Chen
facce8b4d5 [fix](move-memtable) multi replica tables should tolerate minority failures (#38003) (#40477) 
backport #38003
 2024-09-09 11:30:46 +08:00
seawinde
c32d9a129a [test](mtmv) SSB mv rewrite test use little data set for test performance (#40188) (#40437) 
## Proposed changes

commitId: 0baa9366
pr: https://github.com/apache/doris/pull/40188
 2024-09-09 11:23:47 +08:00
seawinde
a67f20f073 [opt](mtmv) Support to contain select constant clause when create async materialized view (#40244) (#40435) 
## Proposed changes
commitId: 518a0fc0
pr: https://github.com/apache/doris/pull/40244
 2024-09-09 11:23:15 +08:00
zhangdong
2c588e2738 [fix](auth)fix case should grant after create view (#40108) (#40488) 
pick: https://github.com/apache/doris/pull/40108
 2024-09-09 11:21:52 +08:00