Commit Graph

13073 Commits

Author SHA1 Message Date
c7b9eb5f9c [enhancement](bitmap)support bitmap type for non-key column in unique table (#23228) 2023-08-23 14:21:22 +08:00
527293aa41 [refactor](dynamic table) remove dynamic table (#23298) 2023-08-23 14:15:14 +08:00
d8a31405c4 [refactor](Nereids): remove Penalty (#23321) 2023-08-23 13:31:01 +08:00
ba882dea21 [pipelineX](dependency) Build DAG between pipelines (#23355) 2023-08-23 13:21:32 +08:00
450b90c082 [Bug](compile) fix compiling error due to merge conflicts (#23351)
fix compiling error due to merge conflicts
2023-08-23 11:57:57 +08:00
78849f6e64 [refactor](predicate) refactor function parse_to_predicte (#23343)
Refactored the interface of the parse_to_predicte function, we will be utilizing this interface to support the decomposition of variant columns.

The "column" parameter might represent a column resulting from the decomposition of a variant column. Instead of using a "unique_id" for identification, we are utilizing a "path" to denote this column.
2023-08-23 11:52:20 +08:00
14296ee87f [fix](window_function) wrong order by range (#23346) 2023-08-23 11:23:00 +08:00
ab9b29f11f [fix](multi-catalog)fix broker and tvf params (#23252)
fix fs specifics, broker and tvf params, and NPE
2023-08-23 10:33:24 +08:00
a82c448910 [test](fix) fix test_clickhouse_jdbc_catalog.groovy (#23342) 2023-08-23 10:31:54 +08:00
35d0c9e71e [refactor](nereids) Refactor stats collection framework (#22963)
* remove auto analyze grammer
* refactor ResultRow
2023-08-23 10:05:57 +08:00
78c6b115c3 [fix](planner)avg function need support large int param (#23254)
* [fix](planner)avg function need support large int param
2023-08-23 10:05:08 +08:00
a5382043d7 [chore](third-party) Fix errors while building brpc (#23339)
close #23340
2023-08-23 09:13:06 +08:00
391355eeae [pipelineX](agg) Support streaming agg (#23341) 2023-08-23 08:38:12 +08:00
Pxl
8ed4045df9 [Chore](primitive-type) remove VecPrimitiveTypeTraits (#22842) 2023-08-23 08:37:40 +08:00
d9993a7790 [doc](multi-catalog)add gcs broker load faq (#23229)
add gcs broker load faq

When the '_' is used  at buckt name, will report an error:  Missing required property: uri

the gcs bucket not match the s3 bucket name format: https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/bucketnamingrules.html

we'd better not use  '_' in bucket.

<!--Describe your changes.-->
2023-08-22 22:48:31 +08:00
Pxl
e6d20f842c [Bug](compile) fix compile failed on function case (#23335) 2023-08-22 22:10:53 +08:00
5c2fae7ce5 [pipeline](exec) Refactor the table sink code in remove unless code (#23223)
Refactor the table sink code in remove unless code
2023-08-22 20:42:14 +08:00
2ad46c5826 [fix](show) show load warning support load v2 (#22759) 2023-08-22 20:08:19 +08:00
eeec26d68d [refactor](load) rename flush_memtable_and_wait to flush_async (#23204) 2023-08-22 20:07:50 +08:00
1609b6cbf2 [pipelineX](sort) Support sort operator (#23322) 2023-08-22 19:36:50 +08:00
6124eea55e [Fix](regression-test)fix unstable test_default_limit case. (#23270)
[Fix](regression-test)fix unstable test_default_limit case. #23270
2023-08-22 18:59:48 +08:00
8819d73abd [fix](be) fix the crash when there is no tzfile in docker env (#23071) 2023-08-22 18:56:36 +08:00
6643d147c6 forbid: nereids_scalar_fn_Array (#23328) 2023-08-22 18:50:34 +08:00
Pxl
1a1f86486d [Improvement](function) opt for case when (#23068)
opt for case when
2023-08-22 18:31:40 +08:00
0b51e6d8e1 [refractor](FunctionArrayIndex) make the codes more simple 2023-08-22 17:48:59 +08:00
9a7d4f906b [enhance](Tools): update tpcds tools #23308
Update tpcds tools: add sf1000 & sf10000 related sql and config.
2023-08-22 17:10:50 +08:00
5184246804 [fix](case) disable unstable case (#23314)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-08-22 16:34:04 +08:00
a4e041ea55 [improve](alter-job) Add a config for forbiding doing alter job (#23294) 2023-08-22 16:28:36 +08:00
4a690f8b13 fix compile error with gcc12 (#23190) 2023-08-22 16:05:03 +08:00
bcdb481374 [refactor](fragment) refactor non pipeline fragment executor (#23281)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-08-22 16:00:34 +08:00
820d328ad7 [feature](Nereids) support handle query in fe (#23263)
1. fix select variable with alias parsed as two level alias expression.
2. support handle query in fe
2023-08-22 15:28:29 +08:00
a7675243d9 [fix](jdbc catalog) fix adaptation to Oracle special character / table names (#23080)
The changes of this PR for JdbcOracleClient are as follows:

#### bug fixes:
  1. Fix the problem that if there is an approximate table name for Schema synchronization with a table name with `/` characters, the synchronization Column will be confused
  2. Fix the NPE problem of metadata synchronization after enabling lower_case_table_names configuration

#### improvement:
  1. Modify the method of synchronizing Oracle User to Doris Database mapping, use `metadata.getSchemas` instead of `SELECT DISTINCT OWNER FROM all_tables`
  2. When synchronizing metadata, change `null` at the catalog level to `conn.getcatalog`
2023-08-22 15:25:42 +08:00
5b05d5167d [FIX](regress-test) fix local_tvf with complex-type in cluster (#23299)
* fixed

* update

* fixed
2023-08-22 15:25:10 +08:00
da2eb69eba [test](Nereids) add array scalar function test cases (#23303) 2023-08-22 15:05:28 +08:00
9d2e23b1aa [fix](parquet) A row of complex type may be stored across more pages (#23277)
A row of complex type may be stored across two(or more) pages, and the parameter `align_rows` indicates that whether the reader should read the remaining value of the last row in previous page.
2023-08-22 14:47:10 +08:00
fbccd5cfeb disable nereids_delete_mow_partial_update (#23273) 2023-08-22 14:46:51 +08:00
6c8af92175 [fix])(nereids)Support select catalog.db.table.column from xxx for nereids planner. #23221
Nereids doesn't support select table.* from table, this pr is to fix this bug.
Support three layer qualifier. (catalog.database.table)
2023-08-22 13:58:25 +08:00
5ff7b57fc1 [fix](parquet) parquet reader confuses logical/physical/slot id of columns (#23198)
`ParquetReader` confuses logical/physical/slot id of columns. If only reading the scalar types, there's nothing wrong, but when reading complex types, `RowGroup` and `PageIndex` will get wrong statistics. Therefore, if the query contains complex types and pushed-down predicates, the probability of the result set is incorrect.
2023-08-22 13:35:29 +08:00
51db11ed0b [improve](jdbc catalog) Add a variable to accommodate the final keyword in ClickHouse Jdbc Catalog queries (#23282) 2023-08-22 12:13:36 +08:00
b471862dba [Fix](regression-test) fix es regression test (#23160) 2023-08-22 11:52:37 +08:00
7450bd504d [improvement](scan) Reduce time to fetch function calls (#23098) 2023-08-22 11:24:55 +08:00
438010a214 [fix](merge-on-write) process error of delete bitmap calculation (#23285) 2023-08-22 10:53:50 +08:00
2a5aed46f7 add CloudCanal en doc (#22731) 2023-08-22 10:49:54 +08:00
5d9678700c [feature](Nereids) support select tablets with nereids optimizer (#23164) 2023-08-22 10:14:27 +08:00
8f48acaab1 [refactor](nereids) convert session var name "beNumForTest" #23255
this var is used for test only. So keep "for_test" as suffix.
2023-08-22 10:12:07 +08:00
0d7a61ae8c [fix](load) fix duplicate register of memtable writer in memory limiter (#23205) 2023-08-22 10:05:17 +08:00
f842a5cf37 [fix](case) comment out orthogonal_bitmap_* (#23259)
comment out orthogonal_bitmap_*
2023-08-22 10:02:53 +08:00
b670dd0db7 [feature](Nereids) support array type (#22851)
FEATURE:
1. enable array type in Nereids
2. support generice on function signature
3. support array and map type in type coercion and type check
4. add element_at and element_slice syntax in Nereids parser

REFACTOR:
1. remove AbstractDataType

BUG FIX:
1. remove FROM from nonReserved keyword list

TODO:
1. support lambda expression
2. use Nereids' way do function type coercion
3. use castIfnotSame when do implict cast on BoundFunction
4. let AnyDataType type coercion do same thing as function type coercion
5. add below array function
- array_apply
- array_concat
- array_filter
- array_sortby
- array_exists
- array_first_index
- array_last_index
- array_count
- array_shuffle shuffle
- array_pushfront
- array_pushback
- array_repeat
- array_zip
- reverse
- concat_ws
- split_by_string
- explode
- bitmap_from_array
- bitmap_to_array
- multi_search_all_positions
- multi_match_any
- tokenize
2023-08-22 09:47:55 +08:00
4e63aa052c [fix](session-var) fix forward parallel fragment instance num, and forward some session by the way (#23234) 2023-08-22 09:00:29 +08:00
ae48de70d6 [improve](jdbc catalog) Improve the speed of writing to postgresql databases (#23279)
reWriteBatchedInserts (boolean) Default false
This will change batch inserts from insert into foo (col1, col2, col3) values (1, 2, 3) into insert into foo (col1, col2, col3) values (1, 2, 3), (4, 5, 6) this provides 2-3x performance improvement

from https://jdbc.postgresql.org/documentation/use/#reWriteBatchedInserts

delete useless parameters useCursorFetch
2023-08-21 23:46:27 +08:00