Commit Graph

13948 Commits

Author SHA1 Message Date
771b8b5bec [fix](case) Update analyze_stats.groovy (#25146) 2023-10-10 12:51:29 +08:00
7434f80300 [pipelineX](refactor) Refactor pending finish dependency (#25181) 2023-10-10 11:56:02 +08:00
880d0d7e70 [Bug](pipeline) Support the auto partition in pipeline load (#25176) 2023-10-10 11:51:12 +08:00
6ad22721cb fix: ubsan compile bug (#25199) 2023-10-10 11:46:33 +08:00
39669c6df2 [feature](pipelineX) add runtimefliter in pipelineX multicast sink (#25120) 2023-10-10 10:41:08 +08:00
59dee6b235 [fix](Nereids) support string cast to complex type (#25154) 2023-10-10 10:26:33 +08:00
3a29bb4bc5 [fix](doc) spelling error for colocate join #25053 (#25202)
Issue: 25053

Translation text not cleaned up
2023-10-10 10:10:55 +08:00
f5b826b66d [fix](mark join) mark join column should be nullable (#24910) 2023-10-10 10:10:36 +08:00
e2be5fafa9 [case](regresstest) update query for parquet/orc with array/map nested type and insert into (#24746) 2023-10-10 10:07:22 +08:00
b8621364d2 [FIX](serde)fix scale with decimalv2 in mysql writer which get real scale #25190 2023-10-10 09:09:57 +08:00
90ad48cdb7 [feature](pipelineX) add node id and profilev2 in pipelineX (#25084) 2023-10-10 09:09:26 +08:00
b58010c48e [fix](export) BufferWritable must be committed before deconstruct (#25185)
F20231009 16:03:47.659968 3342535 string_buffer.hpp:48] Check failed: _now_offset == 0
*** Check failure stack trace: ***
@ 0x561a6f8e21e6 google::LogMessage::SendToLog()
@ 0x561a6f8de7b0 google::LogMessage::Flush()
@ 0x561a6f8e2a29 google::LogMessageFatal::~LogMessageFatal()
@ 0x561a4a409233 doris::vectorized::BufferWritable::~BufferWritable()
@ 0x561a6e202853 doris::vectorized::VCSVTransformer::write()
@ 0x561a6e1f19ba doris::vectorized::VFileResultWriter::_write_file()
@ 0x561a6e1f1522 doris::vectorized::VFileResultWriter::append_block()
@ 0x561a6e121bed

The error will occur in DEBUG mode, and doing export will invalid data.
It has been covered by baidu case.
2023-10-09 22:39:45 +08:00
5e8aef4756 [feature](Nereids) fold weeks_sub/add on fe (#25155)
support folding weeks_sub/add on fe
2023-10-09 21:52:44 +08:00
53b46b7e6c [FIX](filter) update for filter_by_select logic (#25007)
this pr is aim to update for filter_by_select logic and change delete limit

only support scala type in delete statement where condition
only support column nullable and predict column support filter_by_select logic, because we can not push down non-scala type to storage layer to pack in predict column but do filter logic
2023-10-09 21:27:40 +08:00
37247ac449 [opt](Nereids) add two args signature to trim family functions (#25169) 2023-10-09 07:17:52 -05:00
08e7a7b932 [feat](optimizer) Scale sample stats with ratio to make it more precise (#25079)
Since Doris support query specific tablet only, so we don't depend on tableSample to do sample, instead use grammar: TABLET(id) to do so. In OlapAnalyzeTask, we calculate which tablets would be hit and set theirs id in it, so we could get how many rows actually queried and furthur we could get the scale up ratio here
2023-10-09 07:01:59 -05:00
400b9f2f97 [Enhancement](log) Improve Safety and Robustness of Log4j Configuration (#24861) 2023-10-09 06:44:37 -05:00
f8eb36158a [fix](Nereids) alias function support arithmetic functions (#25162) 2023-10-09 19:04:47 +08:00
977d119545 [fix](Insert select tvf) fix NPE because tvf do not have catalog name (#25149) 2023-10-09 18:02:43 +08:00
d02ef36631 [opt](Nereids) match predicate support array as first arg (#25172) 2023-10-09 04:17:27 -05:00
4de3df6a46 [refactor](column) remove unused method and column definitions (#25152)
remove unused method and column definitions
using primitive type in predicate column to check datev1 and datev2
2023-10-09 17:14:35 +08:00
263631e983 [improvement](meta) Infer the column name when create view if the column is expression (#24990)
## Proposed changes

Infer the column name when create view if the column is expression

## Further comments
expr column name infer strategy as following:
|      expr       |                example                    |           column name(before)             | Inferred column name(if position is 2)  |
|  -------------  | ---------------------------------------   | ------------------------------            | --------------------------------------  |
| function        | dayofyear()                               | dayofyear()                               | __dayofyear_1                           |
| cast            | cast(1 as bigint)                         | CAST(1 AS BIGINT)                         | __cast_1                                |
| anylyticExpr    | min()                                     | min()                                     | __min_1                                 |
| predicate       | 1 in (1,2,3,4)                            | 1 IN (1, 2, 3, 4)                         | __in_predicate_1                        |
| literal         | 1 or 'string_var_name'                    | 1 or 'string_var_name'                    | __literal_1                             |
| arithmeticExpr  | &                                         | ... & ...                                 | __arithmetic_expr_1                     |
| identifier      | a or b                                    | a or b                                    | a or b                                  |
| case            | CASE WHEN remark = 's' THEN 1 ELSE 2 END  | CASE WHEN remark = 's' THEN 1 ELSE 2 END  | __case_1                                |
| window          | min(timestamp) OVER (...)                 | min(timestamp) OVER(...)                  | __min_1                                 |


SQL for example:
```sql
CREATE VIEW v1 AS 
SELECT 
  error_code,
  1, 
  'string', 
  now(), 
  dayofyear(op_time), 
  cast (source AS BIGINT), 
  min(`timestamp`) OVER (
    ORDER BY 
      op_time DESC ROWS BETWEEN UNBOUNDED PRECEDING
      AND 1 FOLLOWING
  ), 
  1 > 2,
  2 + 3,
  1 IN (1, 2, 3, 4), 
  remark LIKE '%like', 
  CASE WHEN remark = 's' THEN 1 ELSE 2 END,
  TRUE | FALSE 
FROM 
  db_test.table_test1
```

the output column name is as following:
```
error_code
__literal_1
__literal_2
__now_3
__dayofyear_4
__cast_expr_5
__min_6
__binary_predicate_7
__arithmetic_expr_8
__in_predicate_9
__like_predicate_10
__case_expr_11
__arithmetic_expr_12
```
2023-10-09 04:14:01 -05:00
79fa1d1640 [enhancement](regression-test) add stream load json case (#25168) 2023-10-09 16:40:39 +08:00
af707e5244 [pipelineX](fix) fix external table scan operator (#25166) 2023-10-09 16:33:27 +08:00
e1b9854f90 [bugfix](thirdparty) Upgrade aws s3 sdk to prevent mem leak (#25106)
During the use of the AWS S3 SDK, we found that there is a memory leak. According to the official issue, upgrading the SDK should resolve the issue.
2023-10-09 16:08:50 +08:00
fbbaf929ee [chore](workflow) 2/2, auto trigger "License Check" and "BE Code Formatter" instead of after approval (#25101)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-10-09 15:57:30 +08:00
320709b9ff [opt](Nereids) support like and regexp function (#25148) 2023-10-09 02:55:57 -05:00
7ceb029a17 [Fix](statistics)Fix alter column stats data size is always 0 bug (#24891)
Fix alter column stats data size is always 0 bug.
2023-10-09 15:48:11 +08:00
0bf954ba05 [fix](Nereids) unique table support bitmap column (#25160) 2023-10-09 02:39:11 -05:00
4f7fad5498 [fix](Nereids) properties parser should return map (#25150) 2023-10-09 02:32:56 -05:00
d7b6fe57df [Bug](java-udf) fix java-udf memory leak (#25151) 2023-10-09 15:10:56 +08:00
cdba4c4775 [fix](Nereids) deep copier generate wrong slot for TVF (#25156) 2023-10-09 14:52:36 +08:00
d34ab7accc [fix](Nereids) bind sink should use full base schema (#25153) 2023-10-09 01:40:57 -05:00
1a6c2a3e84 [fix](Nereids) miss BoolLiteral when convert literal to Nereids' one (#25159) 2023-10-09 01:39:09 -05:00
78878e2d56 [chore](workflow) 1/2, remove required checks ("License Check", "Clang Formatter") temporally (#25134)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-10-09 14:35:38 +08:00
490563cfdb [fix](Nereids) support empty array and map literal (#25145) 2023-10-09 01:23:51 -05:00
b41ec6a8a4 [feature](Nereids): Pushdown LimitDistinct Through Join (#25113)
Push down limit-distinct through left/right outer join or cross join.

such as select t1.c1 from t1 left join t2 on t1.c1 = t2.c1 order by t1.c1 limit 1;
2023-10-09 14:19:22 +08:00
5a55e47acd [Enhancement](Load) stream tvf support two phase commit (#23800) 2023-10-09 14:15:56 +08:00
e4100b4ebd [bugfix](DDL) Fix the bug of incorrect partition policy setting (#25021) 2023-10-09 11:28:48 +08:00
0f59f49768 [bugfix](policy) Forbid creating policy with same name with different resource name (#25025) 2023-10-09 11:27:49 +08:00
febc5ed1a5 min max rf type (#25129)
extend min-max rf to support 3 types:

n<A
m>A
n<A<m,
in which m/n are constants and A is column.
this pr only contains thrift definition.
2023-10-09 11:09:07 +08:00
ffaa145728 align node id in explain with nereids node id (#25068)
it is painful to align node in `explain` and node in `explain physical plan`, since they use two different sets of node IDs.
This pr makes 'explain' command use node IDs of their correspond node in 'explain physical plan'
2023-10-09 10:11:36 +08:00
aa1704c50b [doc](data-model) update data-model doc (#24941) 2023-10-08 21:08:16 -05:00
683546f71f [fix](catalog)fix use regex parse partition may cause backtracking (#24876) 2023-10-08 21:04:10 -05:00
6d8391e8b5 [fix](load)fix use regex split partition may cause backtracking (#24903) 2023-10-08 20:56:26 -05:00
f41b6a5fc3 [minor](doc) update the doc for docker env and custom_lib dir (#25088)
1. Update the doc for `apache/doris:build-env-for-2.0`
2. Update the doc for `custom_dir`
2023-10-09 09:50:31 +08:00
7af4be1ee3 [fix](mysqldb) Fix mysqldb upgrade (#25111)
If user has database with same name mysql, will introduce problem when doing checkpoint.

Solution:

Add check for this situation, if duplicate, exit and print log info to prevent damage of metadata;
Add fe config field: mysqldb_replace_name to make things correct if user already has mysql db.
Related pr: #23087 #22868
2023-10-09 09:40:56 +08:00
9e31cb26bb [fix](parse_url) fix parse_url is not working in some case to extract the HOST (#25040)
Issue Number: close #24452
2023-10-09 00:14:58 +08:00
fe167da2ff [fix](hudi) hbase-2.5.5 conflict with hudi (#25136)
PR https://github.com/apache/doris/pull/24606 has updated hbase version to 2.5.5, but it conflict with hudi, causing error like:
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: Failed to get hudi partitions
	at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:1021) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:696) ~[doris-fe.jar:1.2-SNAPSHOT]
...
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.fs.FilterFileSystem.getConf(FilterFileSystem.java:524) ~[hadoop-common-3.3.6.jar:?]
	at org.apache.hadoop.hbase.io.hfile.ReaderContext.<init>(ReaderContext.java:53) ~[hbase-server-2.5.5.jar:2.5.5]
	at org.apache.hadoop.hbase.io.hfile.ReaderContextBuilder.build(ReaderContextBuilder.java:106) ~[hbase-server-2.5.5.jar:2.5.5]
```
2023-10-08 23:39:46 +08:00
451e299151 [Opt](performance) Optimize timeround with minute / second (#25073) 2023-10-08 23:14:23 +08:00