database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
18,065 Commits 1 Branch 0 Tags
31a7060dbdb88c8e81be00aada4a2df35eca625c
Commit Graph

8101 Commits

Author SHA1 Message Date
wuwenchi
31a7060dbd [testcase](hive)add exception test for hive txn (#33278) 
Issue #31442
#32726

1. add LocalDfsFileSystem to manipulate local files.
2. add HMSCachedClientTest to analog HMS services.
3. add test for rollback commit.
 2024-04-12 10:38:48 +08:00
wuwenchi
e11db3f050 [feature](hive)support ExternalTransaction for writing exteral table (#32726) 
Issue #31442

Add `TransactionManager` and `Transaction`. 

```
public interface Transaction {
    void commit() throws UserException;
    void rollback();
}
public interface TransactionManager {
    long begin();
    void commit(long id) throws UserException;
    void rollback(long id);
    Transaction getTransaction(long id);
}
```
`TransactionManager` is used to manage all external transactions:
The application layer should manage the entire transaction through this `TransactionManager`, like:
```
transactionManager.commit();
transactionManager.rollback();
```

`Transaction` is an interface. You can implement this interface according to the specific content, such as `HMSTransaction` currently implemented, iceberg that may be implemented in the future, etc.
 2024-04-12 10:38:12 +08:00
slothever
f0ac21e231 [feature](external) process tbl/db exist when create/drop db/tbl (#33119) 
Issue Number: #31442
 2024-04-12 10:36:43 +08:00
slothever
7a05396cd1 [feature](multi-catalog)support catalog name when create/drop db (#33116) 
Issue Number: #31442
 2024-04-12 10:36:18 +08:00
slothever
01b21da82d [feature](insert)add hive insert plan ut and remove redundant fields (#33051) 
add hive insert sink plan UT case
remove some deprecated code
 2024-04-12 10:30:08 +08:00
slothever
07f296734a [regression](insert)add hive DDL and CTAS regression case (#32924) 
Issue Number: #31442

dependent on #32824

add ddl(create and drop) test
add ctas test
add complex type test
TODO:
bucketed table test
truncate test
add/drop partition test
 2024-04-12 10:24:23 +08:00
slothever
716c146750 [fix](insert)fix hive external return msgs and exception and pass all columns to BE (#32824) 
[fix](insert)fix hive external return msgs and exception and pass all columns to BE
 2024-04-12 10:23:52 +08:00
meiyi
9ada38327b [feature](txn insert) txn insert support insert into select (#31666) 2024-04-12 10:11:22 +08:00
wuwenchi
bd364897d4 [feature](hive/iceberg)add doris's version in table properties (#32774) 
issue #31442
when create a external table, we can add doris's version in table's properties.
 2024-04-12 10:02:31 +08:00
slothever
b98d225183 [fix](insert)fix hive table sink type coercion and unify coercion (#32762) 
Issue Number: #31442
 2024-04-12 10:02:09 +08:00
slothever
3343322965 [fix](insert)fix conversion of doris type to hive type (#32735) 
#31442

create table
fix doris to hive type, use primitiveType to check doris type.
 2024-04-12 10:01:30 +08:00
slothever
70489fe749 [fix](insert)fix hive table sink write path (#32587) 
issue: #31442

fix hive table sink write path to hdfs://${hdfs_root}/tmp/.doris_staging/${user}
 2024-04-12 10:00:48 +08:00
slothever
c68b353017 [feature][insert]add FE UT and support CTAS for external table (#32525) 
1. add FE ut for create hive table
2. support external CTAS:

> source table:
```
mysql> show create table hive.jz3.test;

CREATE TABLE `test`(
  `id` int COMMENT 'col1',
  `name` string COMMENT 'col2')
PARTITIONED BY (
 `dt` string,
 `dtm` string)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  'hdfs://HDFS8000871/usr/hive/warehouse/jz3.db/test'
TBLPROPERTIES (
  'transient_lastDdlTime'='1710837792',
  'file_format'='orc')
```


> create unpartitioned target table
```
mysql> create table hive.jz3.ctas engine=hive as select * from hive.jz3.test;
mysql> show create table ctas;

CREATE TABLE `ctas`(
  `id` int COMMENT '',
  `name` string COMMENT '',
  `dt` string COMMENT '',
  `dtm` string COMMENT '')
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  'hdfs://HDFS8000871/usr/hive/warehouse/jz3.db/ctas'
TBLPROPERTIES (
  'transient_lastDdlTime'='1710860377')

```


> create partitioned target table
```
mysql> create table hive.jz3.ctas1 engine=hive partition by list (dt,dtm) () as select * from hive.jz3.test;
mysql> show create table hive.jz3.ctas1;

CREATE TABLE `ctas1`(
  `id` int COMMENT '',
  `name` string COMMENT '')
PARTITIONED BY (
 `dt` string,
 `dtm` string)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  'hdfs://HDFS8000871/usr/hive/warehouse/jz3.db/ctas1'
TBLPROPERTIES (
  'transient_lastDdlTime'='1710919070')
```
 2024-04-12 09:58:49 +08:00
slothever
36a1bf1d73 [feature][insert]Adapt the create table  statement to the nereids sql (#32458) 
issue: #31442

1. adapt  create table statement from doris  to hive
2. fix insert overwrite for table sink

> The doris create hive table statement:

```
mysql> CREATE TABLE buck2(
    ->     id int COMMENT 'col1',
    ->     name string COMMENT 'col2',
    ->     dt string COMMENT 'part1',
    ->     dtm string COMMENT 'part2'
    -> ) ENGINE=hive
    -> COMMENT "create tbl"
    -> PARTITION BY LIST (dt, dtm) ()
    -> DISTRIBUTED BY HASH (id) BUCKETS 16
    -> PROPERTIES(
    ->     "file_format" = "orc"
    -> );
```

> generated  hive create table statement:

```
CREATE TABLE `buck2`(
  `id` int COMMENT 'col1',
  `name` string COMMENT 'col2')
PARTITIONED BY (
 `dt` string,
 `dtm` string)
CLUSTERED BY (
  id)
INTO 16 BUCKETS
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  'hdfs://HDFS8000871/usr/hive/warehouse/jz3.db/buck2'
TBLPROPERTIES (
  'transient_lastDdlTime'='1710840747',
  'doris.file_format'='orc')

```
 2024-04-12 09:57:37 +08:00
feiniaofeiafei
dc8da9ee89 [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog (#33528) 
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog

* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog

* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog

* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog

---------

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
 2024-04-11 21:43:01 +08:00
zclllyybb
3d66723214 [branch-2.1](auto-partition) pick auto partition and some more prs (#33523) 2024-04-11 17:12:17 +08:00
zhannngchen
ff38e7c497 [log](chore) print isBad in Replica::toString() (#33427) 2024-04-11 09:31:50 +08:00
Jibing-Li
b5a84f7d23 Fix alter column stats without min max value deserialize failure. (#33406) 2024-04-11 09:31:50 +08:00
Pxl
3070eda58c [Bug](load) fix stream load file on hll type mv column (#33373) 
fix stream load file on hll type mv column
 2024-04-11 09:31:50 +08:00
morrySnow
f35dd3fc35 [chore](test) let some case suitable for legacy planner and nereids (#33352) 2024-04-11 09:31:50 +08:00
yiguolei
a38b97fbdd [bugfix](profile) should use backend ip:heartbeat port as key during merge profile (#33368) 2024-04-11 09:31:50 +08:00
feiniaofeiafei
2708641bee [Fix]fix insert overwrite non-partition table null pointer exception (#33205) 
fix legacy planner bug when insert overwrite non-partition table.
 2024-04-11 09:31:50 +08:00
wangbo
38b2e58d59 [Improvement](executor)cancel query when a query is queued (#33339) 2024-04-11 09:31:50 +08:00
abmdocrt
326eee5d04 [Fix](schema change) Fix schema change fault when add complex type column (#31824) 
Problem: An error is encountered when executing a schema change on a unique table to add a column with a complex type, such as bitmap, as documented in https://github.com/apache/doris/issues/31365

Reason: The error arises because the schema change logic erroneously applies an aggregation check for new columns across all table types, demanding an explicit aggregation type declaration. However, unique and duplicate tables inherently assume default aggregation types for newly added columns, leading to this misstep.

Solution: The schema change process for introducing new columns needs to distinguish between table types accurately. For unique and duplicate tables, it should automatically assign the appropriate aggregation type, which, for the purpose of smooth integration with subsequent processes, should be set to NONE.
 2024-04-11 09:31:50 +08:00
Pxl
3081fc584d [Improvement](runtime-filter) support sync join node build side's size to init bloom runtime filter (#32180) 
support sync join node build side's size to init bloom runtime filter
 2024-04-11 09:31:50 +08:00
morrySnow
75f497976c [opt](Nereids) auto fallback when query unsupport table type (#33357) 2024-04-10 16:24:13 +08:00
morrySnow
ecbd92204d [fix](Nereids) variant push down not work on slot without table (#33356) 2024-04-10 16:23:41 +08:00
TengJianPing
0e262ba0e4 [improvement](spill) improve cancel of spill and improve log printing (#33229) 
* [improvement](spill) improve cancel of spill and improve log printing

* fix
 2024-04-10 16:23:20 +08:00
谢健
045dd05f2a [fix](Nereids): don't transpose agg and join if join is mark join (#33312) 2024-04-10 16:23:20 +08:00
Pxl
6462264e77 [Improvement](materialized-view) adjust priority of materialized view match rule (#33305) 
adjust priority of materialized view match rule
 2024-04-10 16:23:04 +08:00
minghong
4eee1a1f0d [fix](nereids) make runtime filter targets in fixed order (#33191) 
* make runtime filter targets in fixed order
 2024-04-10 16:22:39 +08:00
yujun
159ebc76e7 [fix](npe) fix kafka be id npe (#33151) 2024-04-10 16:22:27 +08:00
meiyi
741d4ff97e [fix](group commit) Fix syntax error when insert into table which column names contain keyword (#33322) 2024-04-10 16:22:09 +08:00
meiyi
4079a7b6ab [fix](txn insert) Fix txn insert into values for sequence column or column name is keyword (#33336) 2024-04-10 16:21:31 +08:00
Luwei
29777bc3a8 [fix](fe)reduce memory usage in alter (#32810) (#33474) 
Co-authored-by: kylinmac <kylinmac@163.com>
 2024-04-10 16:04:50 +08:00
morrySnow
d1099852b5 [fix](Nereids) partial update generate column in wrong way (#33326) 
intro by PR #31461
 2024-04-10 16:02:54 +08:00
morrySnow
f31e273ae8 [fix](Nereids) variant column prune push down failed on variant literal (#33328) 2024-04-10 16:02:54 +08:00
morrySnow
93b20f0cc4 [chore](Nereids) create policy always allow fallback (#33226) 2024-04-10 16:01:58 +08:00
morrySnow
bcc819ddd9 [fix](Nereids) array_range not support amount without unit (#33231) 2024-04-10 16:01:58 +08:00
morrySnow
b8d4a87703 [chore](Nereids) load command always could fallback (#33233) 2024-04-10 16:00:53 +08:00
camby
14c5247fb7 [feature](replica) support force set replicate allocation for olap tables (#32916) 
Add a config to force set replication allocation for all OLAP tables and partitions.
 2024-04-10 16:00:15 +08:00
Pxl
2092a862fc [Bug](materialized-view) fix wrong result when salias name same with base slot on mv (#33198) 
fix wrong result when salias name same with base slot on mv
 2024-04-10 16:00:05 +08:00
wangbo
2785269d36 [Improvement](executor)Add BypassWorkloadGroup to pass query queue #33101 2024-04-10 15:56:41 +08:00
yiguolei
16f8afc408 [refactor](coordinator) split profile logic and instance report logic (#32010) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-04-10 15:51:32 +08:00
morrySnow
96867ff3fd [fix](Nereids) support update without filter (#33214) 2024-04-10 15:26:09 +08:00
Xinyi Zou
b696909775 [fix](plsql) Fix plsql variable initialization (#33186) 2024-04-10 15:26:09 +08:00
morrySnow
edd1701963 [fix](Nereids) convert agg state type failed in some cases (#33208) 2024-04-10 15:26:09 +08:00
feiniaofeiafei
5e59c09a60 [Fix](nereids) modify the binding aggregate function in order by (#32758) 
modify the bind logical to make the order by has same behavior with mysql when sort child is aggregate.
when an order by Expr has aggregate function, all slots in this order by Expr should bind the LogicalAggregate non-AggFunction outputs first, then bind the LogicalAggregate Child
e.g.
select 2*abs(sum(c1)) as c1, c1,sum(c1)+c1 from t_order_by_bind_priority group by c1 order by sum(c1)+c1 asc;
in this sql, the two c1 in order by all bind to the c1 in t_order_by_bind_priority
 2024-04-10 15:26:09 +08:00
feiniaofeiafei
67bb519613 [Fix](nereids) forward the user define variables to master (#33013) 2024-04-10 15:26:08 +08:00
LiBinfeng
6798a24a27 [Enhencement](Nereids) reduce child output rows if agg child is literal (#32188) 
with group by:
select max(1) from t1 group by c1; -> select 1 from (select c1 from t1 group by c1);
without group by:
select max(1) from t1; -> select max(1) from (select 1 from t1 limit 1) tmp;
 2024-04-10 15:26:08 +08:00