Commit Graph

299 Commits

Author SHA1 Message Date
bc6af178b3 branch-2.1: [fix](parquet)Fix data column and null map column not equal when reading Parquet complex type cross-page data #47734 (#48039)
Cherry-picked from #47734

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-02-19 15:21:13 +08:00
fb31586612 branch-2.1: [test](jdbc catalog) add more jdbc catalog extreme test (#47799)
cherry-pick (#47525)
2025-02-14 17:03:49 +08:00
3ec900e878 branch-2.1:[opt](jdbc catalog) Compatible with higher ClickHouse JDBC Driver versions (#47832)
cherry-pick from (#46026)
2025-02-13 16:55:25 +08:00
3aad9e5f67 [opt](oceanbase docker)Use LTS docker image and print unhealthy docker logs (#46647) (#47349)
### What problem does this PR solve?

Problem Summary:
Oceanbase container sometimes start failed.
<img width="653" alt="image"

src="https://github.com/user-attachments/assets/d95c66cf-7e04-4179-a565-9b9dd8b87128"
/>

We do two things:
1. Print last 100 lines docker logs of unhealthy container for debugging
2. Upgrade Oceanbase docker image to the newest `4.2.1-lts`, since it is
7 months newer than `4.2.1`, more stable
2025-01-24 11:22:02 +08:00
6c72bfe082 branch-2.1: [fix](test) fix unstable test after some test migration #46834 (#46852)
Cherry-picked from #46834

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-13 18:01:03 +08:00
94856b19a2 [fix](hive)fix select count(*) hive full acid tb opt error. (#46732) (#46804)
bp #46732
2025-01-12 08:55:10 +08:00
04b840713b branch-2.1: [fix](jdbc catalog) Use factory methods to return mapped types instead (#46698)
cherry-pick #46623
2025-01-09 18:12:03 +08:00
4d0037a928 branch-2.1: [fix](ES catalog)Fix query long value exception with doc_value #46554 (#46581)
Cherry-picked from #46554

Co-authored-by: qiye <luen@selectdb.com>
2025-01-08 15:26:58 +08:00
012848060b branch-2.1: [fix](test) Fix mysql catalog check db test #46263 (#46343)
Cherry-picked from #46263

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-01-04 10:45:26 +08:00
a6fc852ce7 branch-2.1: [opt](catalog) use table in db object return get db #46211 (#46229)
Cherry-picked from #46211

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: morningman <yunyou@selectdb.com>
2025-01-01 08:00:49 +08:00
6c9167d9d9 branch-2.1: [test](doc) add job-scheduler example in doris's doc to regression test #42904 (#45968)
Cherry-picked from #42904

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
2024-12-30 10:44:58 +08:00
5425e4686b branch-2.1: [fix](auth) fix use database stmt access unauthorized catalog #45720 (#45978)
Cherry-picked from #45720

Co-authored-by: Petrichor <xiaowenli@selectdb.com>
2024-12-30 10:43:28 +08:00
a380f5d222 [enchement](utf8)import enable_text_validate_utf8 session var (#45537) (#46070)
bp #45537
2024-12-28 10:05:03 +08:00
02519c37ea branch-2.1: [test](mysql catalog) fix mysql jdbc catalog test case (#45988) 2024-12-26 15:06:24 +08:00
bceebfcf38 branch-2.1: [test](jdbc catalog) add some jdbc query tvf test (#45991)
cherry-pick #45938
2024-12-26 15:05:50 +08:00
1396f5a950 [2.1][improvement](jdbc catalog) Optimize JdbcCatalog case mapping stability (#43256)
cherry-pick #41510
2024-12-26 14:57:56 +08:00
289d621faa [improvement](information_schema)Show view definition in information_schema.views. (#45857) (#45930)
backport: https://github.com/apache/doris/pull/45857
2024-12-26 10:11:13 +08:00
303557ac70 [fix](hive)fix hive insert only translaction table. (#45753)
### What problem does this PR solve?
bp #44001 , but no hive4 acid table.

Problem Summary:
1. Fixed the issue that when reading insert translaction only tables,
there was no acid check, which caused multiple data reads (i.e., reading
data from the previous base_n).
2. Forbidden to create, insert data, and delete aicd tables.
2024-12-22 21:23:21 +08:00
19c0e89da7 [enchement](iceberg)support read iceberg partition evolution table. (#45367) (#45569)
cherry-pick #45367

Co-authored-by: daidai <changyuwei@selectdb.com>
2024-12-20 08:56:51 +08:00
d6c629d293 branch-2.1: [fix](tvf) Tvf supports to parse the enclose character in csv files #45407 (#45570)
Cherry-picked from #45407

Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
2024-12-19 16:23:44 +08:00
7d32e4f71f branch-2.1: [Fix](ORC) Not push down fixed char type in orc reader #45484 (#45525)
cherry-pick #45484
2024-12-19 14:06:00 +08:00
855e9a508c [fix](catalog) opt the count pushdown rule for iceberg/paimon/hive scan node (#44038) (#45564)
bp #44038
2024-12-18 09:54:56 +08:00
b881157d52 branch-2.1: [regression](paimon) test Paimon's connectivity of cloud filesystems #45268 (#45378)
Cherry-picked from #45268

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-12-14 19:22:16 -08:00
667f5e6e6a [feat](iceberg)Supports using rest type catalog to read tables in unity catalog for 2.1 (#43525) (#45217)
bp: #43525
2024-12-12 00:49:36 -08:00
d4a6fd1850 Revert #43255 & #44615 (#45096)
Revert "branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for
OR-connected Predicate #43255 (#44438)"
Revert "[fix](orc) check all the cases before build_search_argument
(#44615) (#44801)"
2024-12-06 21:14:13 +08:00
dceaf97381 branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 (#44438)
Cherry-picked from #43255

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-22 22:52:53 +08:00
ec068049c7 branch-2.1: [fix](meta-cache) fix refreshOnlyCatalogCache when use_meta_cache = false #44363 (#44461)
Cherry-picked from #44363

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-22 22:41:10 +08:00
702abbff0f [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#44239)
bp #42004

Co-authored-by: kaka11chen <kaka11.chen@gmail.com>
2024-11-22 11:01:41 +08:00
fbf573f8c8 [fix](hive)fix hive catalog miss partition that have special characters. #42906 (#44152)
cherry pick from #42906

---------

Co-authored-by: daidai <2017501503@qq.com>
2024-11-21 14:11:22 +08:00
83b74827aa branch-2.1: [fix](iceberg)Fix count(*) error with dangling delete problem #44039 (#44101)
Cherry-picked from #44039

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-11-19 17:19:25 +08:00
161d76328d branch-2.1: [fix](iceberg)Delete the data files when deleting the table #43441 (#44138)
Cherry-picked from #43441

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-11-18 18:00:13 +08:00
b120e895f1 branch-2.1: [fix](test) fix mysql jdbc catalog test out #43867 (#43916)
Cherry-picked from #43867

Co-authored-by: zy-kkk <zhongyongkang@selectdb.com>
2024-11-16 16:29:58 +08:00
a1ff02288f branch-2.1: [fix](hive) support query hive view created by spark (#43553)
Cherry-picked from #43530

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
Co-authored-by: morningman <yunyou@selectdb.com>
2024-11-11 23:28:53 +08:00
7de7541ec6 branch-2.1: [fix](ES Catalog)Only like on keyword can be applied to wildcard query (#43400)
Cherry-picked from #41176

Co-authored-by: qiye <jianliang5669@gmail.com>
2024-11-07 22:43:42 +08:00
09f2516c7c [fix](test) fix some test cases #43217 (#43216)
bp #43217
2024-11-04 22:15:30 +08:00
98d3db03b1 [fix](regression-test) fix test_hive_serde_prop #42886 (#43098)
cherry pick from #42886

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-11-01 23:11:45 +08:00
c573351e4e [fix](tvf) fix FE cannot start when replay alter view from tvf (#42866)
bp: #40872
2024-10-31 14:15:48 +08:00
cc30a7e78e [fix](test) fix some unstable external p0 test cases (#42685) (#42943)
cherry-pick #42685

Co-authored-by: daidai <2017501503@qq.com>
2024-10-31 12:36:03 +08:00
cdd32d9582 [enhance](hive) support reading hive table with OpenCSVSerde #42257 (#42940)
cherry pick from #42257

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-31 11:12:07 +08:00
4a62d9e44b Revert "[2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool" (#42481)
Reverts apache/doris#42255

We have found that after closing the connection pool, there will be
class loading problems and connection release problems for some data
sources. We will remove this function first and re-add it after solving
and testing it completely.
2024-10-25 19:37:36 +08:00
2defa90be7 [test](ES Catalog)Add mapping _routing test case (#42074) (#42282)
## Proposed changes

bp #42074
2024-10-23 10:14:12 +08:00
157d67e7ca [enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 (#42273)
cherry pick from #42200

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-22 23:56:57 +08:00
bde8e2d474 [2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool (#42255)
pick (#41992)

We initially introduced jdbc connection pool to improve the connection
performance of jdbc catalog, but we always found that connection pool
would bring some unexpected errors, so we chose to add a catalog
property: `enable_connection_pool` to choose whether to enable the jdbc
connection pool of jdbc catalog, and the default false.However, the
created catalog will still open the connection pool when it is upgraded,
and only the newly created catalog will be false

And we conducted performance tests on this, the performance loss is
within the expected range.

- Enable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM mysql.test.test
limit 1;' --create-schema=mysql --delimiter=";" --verbose
Benchmark
        Average number of seconds to run all queries: 0.008 seconds
        Minimum number of seconds to run all queries: 0.004 seconds
        Maximum number of seconds to run all queries: 0.133 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1

- Disable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM
mysql_no_pool.test.test limit 1;' --create-schema=mysql --delimiter=";"
--verbose
Benchmark
        Average number of seconds to run all queries: 0.054 seconds
        Minimum number of seconds to run all queries: 0.047 seconds
        Maximum number of seconds to run all queries: 0.184 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1
2024-10-22 23:28:28 +08:00
c1d2b8d548 [2.1][improvement](jdbc catalog) Disallow non-constant type conversion pushdown and implicit conversion pushdown (#42242)
pick (#42102)

Add a variable `enable_jdbc_cast_predicate_push_down`, the default value
is false, which prohibits the pushdown of non-constant predicates with
type conversion and all predicates with implicit conversion. This change
can prevent the wrong predicates from being pushed down to the Jdbc data
source, resulting in query data errors, because the predicates with cast
were not correctly pushed down to the data source before. If you find
that the data is read correctly and the performance is better before
this change, you can manually set this variable to true

```
| Expression                                          | Can Push Down |
|-----------------------------------------------------|---------------|
| column type equals const type                       | Yes           |
| column type equals cast const type                  | Yes           |
| cast column type equals const type                  | No            |
| cast column type equals cast const type             | No            |
| column type not equals column type                  | No            |
| column type not equals cast const type              | No            |
| cast column type not equals const type              | No            |
| cast column type not equals cast const type         | No            |

```
2024-10-22 17:27:29 +08:00
47ff6f1300 [fix](OrcReader) fix the issue that orc_reader can not read DECIMAL(0,0) type of orc file #41795 (#42220)
cherry pick from #41795

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-10-22 10:10:25 +08:00
e713b92321 [fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113 (#42222)
cherry pick from #42113

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-22 09:43:29 +08:00
084434e25c [Test](tvf) add regression tests for testing orc reader #41606 #42188 (#42120)
cherry pick from #42031 #42188

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: TieweiFang <ftw2139@163.com>
2024-10-21 21:31:18 +08:00
bbd4970ed8 [feature](jdbc catalog) support gbase jdbc catalog #41027 #41587 (#42123)
cherry pick from #41027 #41587

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-10-21 16:52:23 +08:00
a32ad0b1f7 [cherry-pick](branch-2.1) support reading brotli compressed parquet file (#42162)
pick pr: https://github.com/apache/doris/pull/41875
2024-10-21 16:48:09 +08:00
a150d160ea [fix](jdbc catalog) fix and add mysql and doris extremum test #41679 (#42122)
cherry pick from #41679

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-10-21 16:39:40 +08:00