Commit Graph

5298 Commits

Author SHA1 Message Date
f0babfdcf8 print image's version when it is higher than FEConstants.meta_version (#10908) 2022-07-16 19:26:47 +08:00
f78db1d773 release memory allocated in agg function in vec stream load (#10739)
release memory allocated in agg function in vec stream load

When a load is cancelled, memory allocated by agg functions should
be freeed.
2022-07-16 15:32:53 +08:00
75ca21dafa [Bug] handle null map right in vectorized load (#10883) 2022-07-16 14:18:38 +08:00
00c9455f16 [fix](array-type) fix arrow column to doris array column (#10855)
* support merge array column, while convert from arrow column to doris array column

* fix typo

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-07-16 11:49:42 +08:00
b5b6e1e12c [regression] add ssb flat sf 1 test (#10894) 2022-07-16 11:40:45 +08:00
2d5ea56a06 [regression] rm unwanted sleep (#10893) 2022-07-16 06:00:42 +08:00
6a7487a302 [improvement]Remove the page button on the System page (#10900) 2022-07-16 06:00:08 +08:00
ba1c527a23 [improvement](arrow) Avoid parse timezone for each datetime value (#10869)
* [improvement](arrow) Avoid parse timezone for each datetime value

Convert arrow batch to doris block is too slow when there are datetime values.
Because we call `TimezoneUtils::find_cctz_time_zone` for each values.

After modify, the tpch-100 q1 with external table cost from 40s -> 9s

Co-authored-by: morningman <morningman@apache.org>
2022-07-15 21:19:36 +08:00
7be2ef79ed array column support read by rowids (#10886)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-07-15 21:19:02 +08:00
b1711e94b7 (unique-key-merge-on-write) Add tablet lookup interface, DSIP-18[3/4] (#10820)
Add lookup_row_key interface for tablet and segment.
2022-07-15 20:49:28 +08:00
06ecf8bdc5 [Bug](vec compaction) fix compaction core with sequence column (#10845)
Block reader ignores sequence column but rowset writer should write this column, will core in set_source_column row_num DCHECK. 

Sequence column works across rowsets, so compaction can not discard it and should keeps it altime. 

Co-authored-by: yixiutt <yixiu@selectdb.com>
2022-07-15 20:39:50 +08:00
23dab4cbf9 [enhancement] (SSB & TPC-H) fix test scripts, cover having PASSWORD situation (#10885) 2022-07-15 20:07:47 +08:00
c8e13caf63 [docs] Fixed description about GROUPING_ID in GROUPING SETS DESIGN. (#10805) 2022-07-15 19:59:28 +08:00
6751e5b23c [fix](alter)(tablet-scheduler) fix unexpected exception with compaction_too_slow message when add rollup for olap table (#10827) 2022-07-15 19:59:00 +08:00
dc6fbcce14 [feature-wip] (datev2) modify datev2 format in memory (#10873)
* [feature-wip] (datev2) modify datev2 format in memory

* update
2022-07-15 19:57:38 +08:00
401203da6a [feature](code-data) move cold data to object storage without losing any feature(FE) (#10693)
Co-authored-by:platonekosama@gmail.com
2022-07-15 18:00:48 +08:00
cc84cfcc0e [feature-wip](array-type) add function array_remove (#10385)
Description:

array_remove function remove all elements in array which is equal to the target.
2022-07-15 17:57:49 +08:00
1275aa905a fix sc regression test (#10879) 2022-07-15 17:53:11 +08:00
97861f517a Revert "[chore][nereids] Bump the version of antlr4 to 4.10.1 (#10780)" (#10876)
This reverts commit b4927a8f151c60357387302723fa808e523d17e3.
2022-07-15 17:05:08 +08:00
ecdcc883bc [docs]Replenish-streamload-document (#10844) 2022-07-15 15:56:54 +08:00
dd70feb35f [docs]update flink-connector-doc (#10875)
Co-authored-by: wudi <>
2022-07-15 15:53:15 +08:00
ad4751972c [feature-wip] Support in predicate for datev2 type (#10810) 2022-07-15 14:32:40 +08:00
c4426aa7e2 [doc] Improve the alter table part of the documentation (#10872) 2022-07-15 14:20:46 +08:00
cc0f334cb7 [Bug] (UT) fix parsing microseconds ut (#10842) 2022-07-15 14:15:08 +08:00
82251a6bab [refactor] some refactor of delete predicates (#10816) 2022-07-15 14:13:34 +08:00
dafd41824b [fix](unique-key-merge-on-write) Fix missing delete bitmap introduced by DSIP-018 (#10548) (#10863)
Missing delete bitmap for copy constructor of `TabletMeta(TabletMeta&)`.
2022-07-15 11:58:49 +08:00
33e9d5b2da [enhance](test): remove some System.out.println in UT. (#10859) 2022-07-15 11:16:24 +08:00
39325bd666 [Bug](tablet) Fix bug that segment files are removed as trash but tablet meta is normal (#10721) 2022-07-15 11:07:16 +08:00
b4927a8f15 [chore][nereids] Bump the version of antlr4 to 4.10.1 (#10780) 2022-07-15 10:43:05 +08:00
18348a83ad [chore][compile] fix java udf compile error (#10841) 2022-07-14 20:51:44 +08:00
505758c76b [BUG] (decimalv3) fix FE UTs (#10834) 2022-07-14 19:24:50 +08:00
364c8733fa fix light schema change coredump (#10828) 2022-07-14 15:43:15 +08:00
41f71f3ade [regression] add ssb sf1 test (#10831)
Co-authored-by: stephen <hello-stephen@qq.com>
2022-07-14 15:03:40 +08:00
ca5dbb1bcc Fix olap scan node normalize_in_and_eq_predicate infinite loop bug. (#10817) 2022-07-14 14:54:57 +08:00
799300c475 [website]1.1 release formulation (#10838) 2022-07-14 14:43:33 +08:00
190daee9f3 [doc] Fixed doc typo for materialized views 2022-07-14 14:16:45 +08:00
d245ab76cc [improvement]Use uint32 instead of size_t to reduce agg key's length (#10832) 2022-07-14 14:11:55 +08:00
575a1cb173 [Release] add download links for v1.1 (#10836) 2022-07-14 14:01:52 +08:00
13e9cb146f [feature-wip](unique-key-merge-on-write) Add option to enable unique-key-merge-on-write, DSIP-018[5/1] (#10814)
* Add option in FE

* add opt in be

* some fix

* update

* fix code style

* fix typo

* fix typo

* update

* code format
2022-07-14 12:10:58 +08:00
d1573e1a4a [improvement]Use phmap for aggregation with serialized key (#10821) 2022-07-14 11:26:09 +08:00
e361eb385e [vectorized][udf] improvement java-udaf with group by clause (#10296)
save for file about udaf
add bool _destory_deserialize
update some code according reviewer
change destroy all data at once
2022-07-14 11:23:42 +08:00
3d52bff8d1 [improvement]output query_id when be core dumped. (#10822) 2022-07-14 10:55:28 +08:00
3b46242483 [feature-wip] Optimize Decimal type (#10794)
* [feature-wip](decimalv3) support decimalv3

* [feature-wip] Optimize Decimal type

Co-authored-by: liaoxin <liaoxinbit@126.com>
2022-07-14 10:50:50 +08:00
bb0d023abd [docs] Change the incubator fields before and after Doris' Graduation (#10482)
Change the incubator fields before and after Doris' Graduation
2022-07-14 10:48:20 +08:00
077ec4b114 [bug](multi-catalog) empty hadoop configuration when reading iceberg table (#10793) 2022-07-14 10:18:59 +08:00
4f4ce4674a [fix](array) regression test fix for array (#10815) 2022-07-13 21:21:34 +08:00
e78cca1009 (Refactor)[Nereids] Combine operator and plan (#10786)
in #9755, we split plan into plan & operator, but in subsequent development, we found the rule became complex and counter intuition: 
1. we must create an operator instance, then wrap a plan by the operator type.
2. relational algebra(operator) not contains children 

e.g.
```java
logicalProject().then(project -> {
    List<NamedExpression> boundSlots =
        bind(project.operator.getProjects(), project.children(), project);
    LogicalProject op = new LogicalProject(flatBoundStar(boundSlots));
    // wrap a plan
    return new LogicalUnaryPlan(op, project.child());
})
```

after combine operator and plan, the code become to:
```java
logicalProject().then(project -> {
    List<NamedExpression> boundSlots =
        bind(project.getProjects(), project.children(), project);
    return new LogicalProject(flatBoundStar(boundSlots), project.child());
})
```

Originally, we thought it would be convenient for `Memo.copyIn()` after split plan & operator, because Memo don't known how to re-new the plan(assembling child plan in the children groups) by the plan type. So plan must provide the `withChildren()` abstract method to assembling children. The less plan type, the lower code cost we have(logical/physical with leaf/unary/binary plan, about 6 plans, no concrete plan e.g. LogicalAggregatePlan). 

But the convenient make negative effect that difficult to understand, and people must known the concept then can develop some new rules, and rule become ugly. So we combine the plan & operator, make the rule as simple as possible, the negative effect is we must overwrite some withXxx for all concrete plan, e.g. LogicalAggregate, PhysicalHashJoin.
2022-07-13 19:05:15 +08:00
56b55563c6 [feature-wip](unique-key-merge-on-write) add bloom filter index for primary key, DSIP-018[1.2] (#10706) 2022-07-13 18:58:45 +08:00
ad7702f06e add flink local debug log dep doc (#10806)
add flink local debug log dep doc
2022-07-13 17:43:42 +08:00
def59a686e [improvement]output fetal log to stderr (#10789) 2022-07-13 16:34:37 +08:00