Commit Graph

21393 Commits

Author SHA1 Message Date
7b579c75c6 [fix](ES Catalog)Make sure ES meta is synced before using (#46781) (#47711)
bp #46781
2025-02-11 21:00:08 +08:00
e2f9b436a4 [fix](nereids)Add catalog/db/table filter info in SchemaScanNode (#46864) (#47759)
backport: https://github.com/apache/doris/pull/46864
2025-02-11 19:05:27 +08:00
964c940131 [fix](coordinator) Fix wrong recvrId in fragment contains BHJ (#47727) (#47729)
In Coordinator, a shuffle map consists of `recvrId` in each instance.
For example, if 3 BEs exist in a cluster, for a shuffled hash join, we
get 3 maps for a fragment sent to each BE:
BE0: {0:0, 1:1}
BE1: {2:0, 3:1}
BE2: {4:0, 5:1}
In this example, parallelism is 2. Keys in shuffle map indicate the
global shuffle id and the values indicate the instance id in current BE.
In Coordinator, the `recvrId` is the global shuffle id of each instance
so we may get a wrong result if it is wrong.

This bug is caused by `recvrId` set by a BHJ fragment. If a fragment
contains both BHJ and SHJ, `recvrId` should be set by SHJ and BHJ should
be ignored.
2025-02-11 18:49:34 +08:00
6a3c630ee0 branch-2.1: [chore](binlog) Save properties in CreateTableRecord #47551 (#47639)
Cherry-picked from #47551

Co-authored-by: walter <maochuan@selectdb.com>
2025-02-11 16:57:20 +08:00
c8ccada010 branch-2.1: [Fix](memory) Add try catch block for Segment::load_pk_index_and_bf… (#47742)
pick https://github.com/apache/doris/pull/47715
2025-02-11 16:56:13 +08:00
2f7e0f0b76 Revert "[branch-2.1]downgrade resource tag when there is not queryabl… (#47746)
revert #45387
2025-02-11 16:51:07 +08:00
9eb6c2fa60 [fix](jni) pass krb5 conf to jni (#47679)
### What problem does this PR solve?

Related PR: #47299

Problem Summary:

This pick part of #47299, only related to `jni-util.cpp`.
To pass the krb5.conf config to Jni env
2025-02-10 22:27:11 +08:00
6ceb010195 branch-2.1: [fix](mv)Fix direct select mv pre agg bug. #47658 (#47700)
Cherry-picked from #47658

Co-authored-by: James <lijibing@selectdb.com>
2025-02-10 21:45:00 +08:00
470030b878 [feat](clone) Speed clone tablet via batch small file downloading #45061 (#45218)
cherry pick from #45061
2025-02-10 19:38:40 +08:00
f4e92d75f5 [branch-2.1] fix select * from variables system table (#34529) (#47660)
pick #34529

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Co-authored-by: xueweizhang <zxw520blue1@163.com>
2025-02-09 14:14:06 +08:00
8ff4ae879e branch-2.1: [fix](nereids) fix compare ipv4 / ipv6 always equals (#47514)
### What problem does this PR solve?

fix compare ipv4 and ipv6 always equals.

for example, `select cast('127.0.0.1' as ipv4) = cast('192.168.10.10' as
ipv4)'` will return 1, but it should return 0;

Issue Number: close #xxx
2025-02-09 04:43:07 +08:00
3f250e55ce branch-2.1: [Enhancement](regression-test)Add regression test for database properties backup and restore #41925 (#47643)
Cherry-picked from #41925

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-02-09 04:24:05 +08:00
e47b398123 branch-2.1: [chore](binlog) Add download binlog related configs to BE #47412 (#47585)
cherry pick from #47412
2025-02-08 16:23:23 +08:00
e681c055a4 [chore](binlog) Log download binlog cost (#47360) (#47575)
cherry-pick #47360
2025-02-08 16:22:23 +08:00
b4f00be528 branch-2.1: [fix](restore) Make the DirMoveTask idempotent. #47313 (#47584)
cherry pick from #47313
2025-02-08 14:28:38 +08:00
3ec723f2cb branch-2.1: [fix](prepared statement) fix protocol with TIME datatype #47389 (#47543)
Cherry-picked from #47389

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2025-02-08 13:00:49 +08:00
800288b7fe [branch-2.1](FoldFE) pick special date arithmetics of #37376 and #37766 (#47616)
pick https://github.com/apache/doris/pull/37376 and
https://github.com/apache/doris/pull/37766
2025-02-08 11:29:52 +08:00
45c9daa063 branch-2.1:[fix](docker) Starting thirdpaty script with only the res… (#47592) 2025-02-08 10:11:32 +08:00
9d3ee27e55 branch-2.1: [fix](restore) Add a local snapshot lock to protect snapshot dir #47279 (#47574)
cherry pick from #47279
2025-02-08 09:49:22 +08:00
226f848ad8 branch-2.1: [fix](hive docker)Table partition_location_1 miss data #47539 (#47559)
Cherry-picked from #47539

Co-authored-by: Thearas <gaozifeng@selectdb.com>
2025-02-07 11:21:47 +08:00
0cd2557882 branch-2.1: [Fix]Fix double-free when memtable agg #47523 (#47532)
Cherry-picked from #47523

Co-authored-by: wangbo <wangbo@selectdb.com>
2025-02-06 13:11:05 +08:00
209ddb374e branch-2.1: [chore](io) Add debug log for critical file operations #46770 (#46859)
cherry pick from #46770
2025-02-06 09:41:45 +08:00
328d23a825 branch-2.1: [fix](case)rm hms event use_meta_chache is false case. #47457 (#47492)
Cherry-picked from #47457

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-02-05 18:04:46 +08:00
5240e6fd36 branch-2.1: [regression-test](case) audit log case #47456 (#47485)
Cherry-picked from #47456

Co-authored-by: shuke <shuke@selectdb.com>
2025-02-05 11:43:19 +08:00
13e88b8a15 branch-2.1: [Fix](test) test_cumu_compaction_with_delete case should be nonConcurrent #47372 (#47448)
Cherry-picked from #47372

Co-authored-by: abmdocrt <lianyukang@selectdb.com>
2025-01-27 08:48:13 +08:00
2334b2a513 [fix](hive) Incorrect location conversion for target path for 2.1 (#47162) (#47343)
bp: #47162
2025-01-26 23:56:04 +08:00
af55eba242 branch-2.1: [opt](hive docker)Exit on creating table failed #47390 (#47453) 2025-01-26 17:28:20 +08:00
803d3a1545 Revert "[fix](date_function) fix str_to_date function return wrong microsecond issue" (#47441) 2025-01-26 12:53:11 +08:00
Pxl
631d1f43cf [Chore](profile) add some profile on ReaderInit #45556 (#47427)
pick from #45556
2025-01-26 10:51:08 +08:00
0e13ee3db6 branch-2.1: [fix](restore) Release snapshots after the FINISHED state is persisted #47277 (#47286)
Cherry-picked from #47277

Co-authored-by: walter <maochuan@selectdb.com>
2025-01-25 11:53:48 +08:00
2ae03d7e02 branch-2.1: [Bug](function) fix wrong result when case when have more than 256 conditions and the… #47179 (#47392)
Cherry-picked from #47179

Co-authored-by: Pxl <xl@selectdb.com>
2025-01-25 10:04:50 +08:00
7c9d64d79a [opt](iceberg docker)Add health check for iceberg rest container (#46767) (#47422) 2025-01-25 09:04:27 +08:00
5f2438aeab branch-2.1: [opt](docker)Add healthy check for ES and Kafka #47362 (#47414)
Cherry-picked from #47362

Co-authored-by: Thearas <gaozifeng@selectdb.com>
2025-01-25 09:00:50 +08:00
dd7affb8f9 branch-2.1 [Fix](ShortCircuite) fix point query crash with prepared statement when encounter delete sign (#47227)
cherry-pick from #47178
2025-01-25 08:33:06 +08:00
92db307697 branch-2.1: [fix](edit_log) throw exception when replay alter constraint on catalog table #47151 (#47233)
Cherry-picked from #47151

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-01-25 08:32:01 +08:00
407d04fab5 branch-2.1: [opt](docker)Replace healthy container with --wait #47357 (#47421)
Cherry-picked from #47357

Co-authored-by: Thearas <gaozifeng@selectdb.com>
2025-01-25 08:31:15 +08:00
3a9612d97e [Bug](fix) Try to prevent the bug of brpc issue 2146 (#47432) 2025-01-24 21:38:08 +08:00
3b7cc67ec2 branch-2.1: [fix](func) Fix precision loss in ST_GeometryFromWKB coordinate parsing #46661 (#47263)
Cherry-picked from #46661

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
2025-01-24 17:34:37 +08:00
ad6737796f branch-2.1: [fix](binlog) Fix table not gc binlog meta/records #46981 (#47257)
Cherry-picked from #46981

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-01-24 17:32:58 +08:00
8925a390d6 [fix](date_function) fix str_to_date function return wrong microsecond issue (#47252) 2025-01-24 17:32:23 +08:00
ffb9c863d9 branch-2.1: [fix](mtmv)Release snapshots to avoid keeping references in MTMVTask #47149 (#47259)
Cherry-picked from #47149

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-24 17:31:25 +08:00
2c476753d8 branch-2.1: [fix](cache) fix sql cache throw npe in cloud mode #47221 (#47267)
Cherry-picked from #47221

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
2025-01-24 17:30:37 +08:00
701aec6b21 branch-2.1: [opt](jsonb) add ut for the jsonb parser #47181 (#47388)
Cherry-picked from #47181

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-01-24 17:29:33 +08:00
13964bb1ee branch-2.1: [fix](sql) Wrong result of tosql for drop column with rollup #47153 (#47206)
pick: https://github.com/apache/doris/pull/47153
2025-01-24 17:28:27 +08:00
4c0f3d7631 branch-2.1: [fix](ccr) Wrong tosql for alt tbl enable feature #46620 (#47212)
pick: https://github.com/apache/doris/pull/46620
2025-01-24 17:18:34 +08:00
511adb8ea3 branch-2.1:[Opt](regression) Add more test cases to external communit… (#47383)
### What problem does this PR solve?
bp https://github.com/apache/doris/pull/46897
Add more test cases to external community pipeline
2025-01-24 16:35:49 +08:00
baaf026e82 [fix](hive docker)Reserve host port for hive2 namenode and datanode (#47262) (#47354)
Problem Summary:

The [External hive

CI](http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/612304?buildTab=log&linesState=3650&logView=flowAware)
failed because of `namenode` error( 50070 port already in used), docker
logs:
```txt
2025-01-21T04:22:37.955682469Z java.net.BindException: Port in use: 0.0.0.0:50070
2025-01-21T04:22:37.955686106Z 	at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:940)
2025-01-21T04:22:37.955689402Z 	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:876)
2025-01-21T04:22:37.955692708Z 	at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
2025-01-21T04:22:37.955697828Z 	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:760)
2025-01-21T04:22:37.955701444Z 	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)
2025-01-21T04:22:37.955704831Z 	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819)
2025-01-21T04:22:37.955708237Z 	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803)
2025-01-21T04:22:37.955711674Z 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)
2025-01-21T04:22:37.955715090Z 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)
2025-01-21T04:22:37.955718446Z Caused by: java.net.BindException: Address already in use
2025-01-21T04:22:37.955722013Z 	at sun.nio.ch.Net.bind0(Native Method)
2025-01-21T04:22:37.955725460Z 	at sun.nio.ch.Net.bind(Net.java:433)
2025-01-21T04:22:37.955729227Z 	at sun.nio.ch.Net.bind(Net.java:425)
2025-01-21T04:22:37.955733074Z 	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
2025-01-21T04:22:37.955736600Z 	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
2025-01-21T04:22:37.955740197Z 	at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
2025-01-21T04:22:37.955743884Z 	at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:934)
2025-01-21T04:22:37.955747391Z 	... 8 more
2025-01-21T04:22:37.961686454Z 25/01/21 04:22:37 INFO util.ExitUtil: Exiting with status 1
```

The best choice is avoid the services using server port at range
`/proc/sys/net/ipv4/ip_local_port_range` (32768-60999). But since the
namenode [hardcode exposing port `50070` in docker
image](https://hub.docker.com/layers/bde2020/hadoop-datanode/2.0.0-hadoop2.7.4-java8/images/sha256-5623fca5e36d890983cdc6cfd29744d1d65476528117975b3af6a80d99b3c62f),
so we add the port to `net.ipv4.ip_local_reserved_ports` and introduce a
new flags `--reserve-ports` to control it (default false, because not
everyone want to modify system reserved ports).

Change-Id: I03a81e9931cb555695199436b6f0517cccf83588
2025-01-24 16:12:03 +08:00
3aad9e5f67 [opt](oceanbase docker)Use LTS docker image and print unhealthy docker logs (#46647) (#47349)
### What problem does this PR solve?

Problem Summary:
Oceanbase container sometimes start failed.
<img width="653" alt="image"

src="https://github.com/user-attachments/assets/d95c66cf-7e04-4179-a565-9b9dd8b87128"
/>

We do two things:
1. Print last 100 lines docker logs of unhealthy container for debugging
2. Upgrade Oceanbase docker image to the newest `4.2.1-lts`, since it is
7 months newer than `4.2.1`, more stable
2025-01-24 11:22:02 +08:00
896fd7d6bc branch-2.1: [Opt](test) export/outfile case supports multi-kerberos env #47189 (#47239)
Cherry-picked from #47189

Co-authored-by: zgxme <zhenggaoxiong@selectdb.com>
2025-01-23 15:19:44 +08:00
50b3303385 branch-2.1:[fix](docker) Start kerberos docker correctly (#47315)
### What problem does this PR solve?

Wrongly started kerberos twice that may cause container name conflict.

Related PR: https://github.com/apache/doris/pull/46858
2025-01-22 17:55:37 +08:00