Commit Graph

161 Commits

Author SHA1 Message Date
7ce746654a [test](jdbc) add doris and sqlserver jdbc catalog test case (#26656) 2023-11-10 10:32:09 +08:00
49cffd0bc9 [fix](JdbcCatalog) fix that the predicate column name does not have back quote when querying the JDBC appearance (#26479) 2023-11-10 09:54:39 +08:00
8434389358 [fix](jdbc) fix clickhouse catalog arr nullable and add case (#26639) 2023-11-09 19:32:05 +08:00
22bf2889e5 [feature](tvf)(jni-avro)jni-avro scanner add complex data types (#26236)
Support avro's enum, record, union data types
2023-11-09 13:58:49 +08:00
f6b7046a6e [fix](regression-test) add tests for jdbc catalog (#26608) 2023-11-09 11:59:35 +08:00
5bcf6bfd46 [fix](jdbc catalog) fix mysql zero date (#26569) 2023-11-08 21:41:56 +08:00
daeca7d414 [feature](doris compose) support multiple disk types and other improvement (#25924) 2023-11-06 23:23:09 +08:00
80f654ec2a [Fix](statistics)Fix analyze min max sql syntax error. #26240 2023-11-02 09:22:32 +08:00
78204f7c92 [Fix](statistics)Fix external couldn't analyze database bug (#26025) 2023-10-31 11:32:47 +08:00
501c6096dd Revert "[Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)" (#26069)
This reverts commit 3beba1764c01b6712b108556433c96429c59cc45.
2023-10-29 12:45:32 +08:00
3beba1764c [Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)
Add tpcds sf100 hive shapes.

Disable query64 temporarily because it is not same with emr cluster after collecting metadata by analyze table xxx.
And the root cause need to analyze, will enable in future PR.
2023-10-27 18:39:29 +08:00
6a62835f7e [fix](docker) Add docker ignore files and change docker default sourc… (#25696) 2023-10-27 11:23:21 +08:00
c86fad7cbd [Fix](orc-reader) Fix orc decimal128 scale issue. (#25977) 2023-10-26 08:50:18 -05:00
e7a3cb079b [Enhance](regression)docker hive s3 file address is determined based on the configuration (#25905)
docker hive s3 file address is determined based on the configuration custom_settings.env
2023-10-26 11:58:33 +08:00
e783ef716f [fix](multi-table) fix unknown source slot descriptor when load multi table (#25762) 2023-10-25 21:52:01 +08:00
10f1957379 [feature](docker)add docker-iceberg init tables (#25424)
Add some init tables for docker-iceberg.
2023-10-24 19:29:57 +08:00
7de3d9882c [regresstion-test](jdbc catalog)Mariadb compatible test (#25664) 2023-10-23 11:51:03 +08:00
0533911275 [Enhance](regression)change external default docker list (#25573) 2023-10-18 02:56:10 -05:00
ce18f1148a [improvement](catalog)compatible with paimon 0.5 (#24985)
compatible with paimon 0.5
add p0 for paimon,need set enablePaimonTest=true
2023-10-17 22:07:13 +08:00
b946521a56 [enhancement](regression-test) add single stream multi table case (#25360) 2023-10-14 10:59:50 +08:00
1c3ecbbae9 [docker] [fix] add kafka log collector (#25326)
add kafka log collector
2023-10-12 15:23:10 +08:00
73c3e3ab55 [Feature](x-load) support config min replica num for loading data (#21118) 2023-10-11 21:07:35 +08:00
f680a2141d [enhancement](regression-test) add routine load json case (#25253) 2023-10-11 19:43:08 +08:00
b7ac95a970 [enhancement](regression-test) open routine load regression test by default and add data check (#25122) 2023-10-11 10:03:16 +08:00
7e9ffad933 [fix](ES catalog)Doris cannot parse ES date field without time zone (#24864)
1. Add support for Doris to parse ES date field without time zone info. eg: `2023-04-17T23:01:18.151`, this time will be treated as UTC time, since ES assumes that the time zone for time fields without time zones is UTC.
2. Change local time zone convertion from system local time zone to session variable time zone.
2023-10-08 19:28:08 +08:00
541f48a754 [feature](es-catalog) add include_hidden_indexin order to get the hidden index. (#24826) 2023-10-08 14:35:08 +08:00
fddef8b473 [fix](es-catalog)fix error when querying the index ,elasticsearch version 8.9.1 (#24839)
Issue Number: close #24833
2023-10-08 10:19:45 +08:00
b9496b2a8f [feature](docker) regression test support run suite in docker (#24508) 2023-09-24 21:44:16 +08:00
74bba4bdaf [enhancement](regression-test) Add routine load case (#24536) 2023-09-22 14:55:01 +08:00
f5c07ad6da [enhance](external)add tools for generate orc and parquet file (#24713) 2023-09-22 11:13:58 +08:00
6c19e106ad [fix](rest catalog)support set region for s3 (#24566)
Use REST Catalog to access S3 and support setting up regions:
```
CREATE CATALOG iceberg_rest_s3 PROPERTIES (
    "type"="iceberg",
    "iceberg.catalog.type"="rest",
    "uri" = "http://127.0.0.1:8181",
    "s3.endpoint" = "http://127.0.0.1:8010",
    "s3.access_key" = "admin",
    "s3.secret_key" = "password",
    "s3.region" = "us-east-1"
);
```
2023-09-21 20:19:43 +08:00
2a260be10c [improvement](jdbc catalog) when lower_case_table_names of jdbc catalog properties is set to true, use the real table name to query the jdbc data source (#24520) 2023-09-20 17:47:11 +08:00
1c142309a6 [refactor](jdbc catalog) refactor JdbcFunctionPushDownRule (#23826)
1. Change from using string matching function to using Expr matching
2. Replace the `nvl` function with `ifnull` when pushed down to MySQL
3. Adapt ClickHouse's `from_unixtime` function to push down
4. Non-function filtering can still be pushed down when `enable_func_pushdown` is set to false
2023-09-15 22:16:07 +08:00
dc0c39f1d8 [Enhance](external)change hive docker to host network and add hive case (#24401)
1. Change the external hive docker network mode from the bridge mode to the host mode to support the external test of the multi-node doris cluster
2. Added more hive test data in various formats
3. Added a test case with hive
2023-09-15 17:46:24 +08:00
11afd321cb [fix](es catalog) fix issue with select and insert from es catalog core (#24318)
Issue Number: close #24315

The root cause of this issue is that Elasticsearch's long type allows inserting floats and strings. Doris did not handle these cases when doing type conversion. The current strategy is to take the integer before the decimal point if a float or string is found.
2023-09-13 23:07:31 +08:00
3d6d40db33 [docker] Add kafka relate case (#24180)
Add kafka relate case
2023-09-11 19:41:21 +08:00
a664bc5fca [enhancement](jdbc catalog ) support postgresql partition table (#23744) 2023-09-03 10:52:01 +08:00
657e927d50 [fix](json)Fix the bug that read json file Out of bounds access (#23411) 2023-09-02 01:11:37 +08:00
4c00b1760b [feature](partial update) Support partial update for broker load (#22970) 2023-08-29 14:41:01 +08:00
d19dcd6bc1 [improve](jdbc catalog) support sqlserver uniqueidentifier data type (#23297) 2023-08-28 10:30:10 +08:00
a7675243d9 [fix](jdbc catalog) fix adaptation to Oracle special character / table names (#23080)
The changes of this PR for JdbcOracleClient are as follows:

#### bug fixes:
  1. Fix the problem that if there is an approximate table name for Schema synchronization with a table name with `/` characters, the synchronization Column will be confused
  2. Fix the NPE problem of metadata synchronization after enabling lower_case_table_names configuration

#### improvement:
  1. Modify the method of synchronizing Oracle User to Doris Database mapping, use `metadata.getSchemas` instead of `SELECT DISTINCT OWNER FROM all_tables`
  2. When synchronizing metadata, change `null` at the catalog level to `conn.getcatalog`
2023-08-22 15:25:42 +08:00
51db11ed0b [improve](jdbc catalog) Add a variable to accommodate the final keyword in ClickHouse Jdbc Catalog queries (#23282) 2023-08-22 12:13:36 +08:00
41ff48f838 [regresstion][external]fix case test_show_where and es_query 0811 (#22898) 2023-08-12 19:41:55 +08:00
23094a01d4 [fix](test) load data inpath will remove the data in hdfs (#22908)
Load data from hdfs in hive will move the source directory into table's location directory, leading the error like Can not get first file, please check uri in tvf test.
2023-08-12 15:12:00 +08:00
de5603da6b [regresstion][external]fix jdbc cases fail external 0809 (#22761)
fix jdbc cases fail external 0809
2023-08-10 15:23:30 +08:00
3eeca7ee55 [enhance](regresstion case)add external group mark 0727 (#22287)
* add external group mark 0727

* add external pipeline regression conf
 0727

* update pipeline regression config 0727

* open es config from docker 0727
2023-07-28 17:11:19 +08:00
6f1c03c766 [fix](jdbc_catalog) fix int and bigint in mysql view when use doris catalog (#22251) 2023-07-27 16:50:42 +08:00
4f6a3c5bf0 [feature](catalog) support clob type in oracle jdbc catalog (#21532) 2023-07-27 15:49:15 +08:00
cf677b327b [fix](jdbc catalog) Fixed mappings with type errors for bool and tinyint(1) (#22089)
First of all, mysql does not have a boolean type, its boolean type is actually tinyint(1), in the previous logic, We force tinyint(1) to be a boolean by passing tinyInt1isBit=true, which causes an error if tinyint(1) is not a 0 or 1, Therefore, we need to match tinyint(1) according to tinyint instead of boolean, and this change will not affect the correctness of where k = 1 or where k = true queries
2023-07-25 22:45:22 +08:00
beec0e9169 [Improvement](tablet clone) impr tablet sched speed and fix tablet sched failed too many times (#21856) 2023-07-18 23:25:22 +08:00