71aedb994e
[refactor][doc] add data-table doc ( #8927 )
2022-04-09 19:18:44 +08:00
1ee8633e5e
[fix](account) use LOG.info instead of LOG.debug ( #8911 )
...
This complements (#8849 )
2022-04-09 19:18:13 +08:00
5706679e08
[fix] fix the problem that using tsan to compile，BE will stack overflow when start ( #8904 )
...
Currently TSAN can only be compiled using CLang, not GCC.
And when compiling with -o0, stack overflow occurs at startup, issue #8868 .
A function definition will be reported missing at compile time, the file provided in PR #8665 is required.
2022-04-09 19:17:28 +08:00
f28ad36c02
[test][improvement] support execute multiple sql in sql file ( #8902 )
...
regression testing framework support execute multiple sql in sql file
2022-04-09 19:15:53 +08:00
ce6b5169c2
[fix](join) Fix error bucket num get in bucket shuffle join in dynamic partition ( #8891 )
2022-04-09 19:11:44 +08:00
2c1c7f40b6
[refactor][doc] Add data backup, data restore and data delete recovery ( #8865 )
...
1.Add data backup doc,
2.add data restore doc,
3.add data delete recovery doc
2022-04-09 19:04:57 +08:00
1de0ea2dc4
[refactor][doc] Added documentation for advanced usage section ( #8826 )
...
1.Materialized view
2.Schema Change
3.Dynamic Partition
4.Bucket Shuffle Join
5.Colocation Join
6.Runtime Filter
7.partition cache
8.Orthogonal BITMAP calculation
9.Variable
10.Time zone
11.File Manager
2022-04-09 19:03:43 +08:00
a290104966
[fix](routine load) Routine load task doesn't reallocate when previous BE is down. ( #8824 )
...
if previous be is not alive, should assigned another available BE instead.
2022-04-09 19:02:55 +08:00
0f10f84075
[refactor][doc] Add update-delete documentation ( #8821 )
2022-04-09 19:02:16 +08:00
ddf7ef9327
[improvement](join) update broadcast join cost algorithm ( #8695 )
...
broadcast join cost is used compressed data size currently.
The amount of memory used may be significantly more than estimated.
This patch:
1. add a compressed ratio to broadcast join cost and set to 5 according to the experience.
2. add a new session variable `auto_broadcast_join_threshold` to limit memory used by broadcast in bytes, the default value is 1073741824(1GB)
2022-04-09 19:00:27 +08:00
2059e88d43
[fix][doc] remove non-exist outfile.md ( #8913 )
2022-04-08 23:24:10 +08:00
453485abfb
[Bug] Fix some bugs(rewrite rule/symbol transport) of like predicate ( #8770 )
2022-04-08 14:32:09 +08:00
c5718928df
[feature-wip](array-type) support explode and explode_outer table function ( #8766 )
...
explode(ArrayColumn) desc:
> Create a row for each element in the array column.
explode_outer(ArrayColumn) desc:
> Create a row for each element in the array column. Unlike explode, if the array is null or empty, it returns null.
Usage example:
1. create a table with array column, and insert some data;
2. open enable_lateral_view and enable_vectorized_engine;
```
set enable_lateral_view = true;
set enable_vectorized_engine=true;
```
3. use explode_outer
```
> select * from array_test;
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 3 | NULL | NULL |
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
+------+------+--------+
> select k1,explode_column from array_test LATERAL VIEW explode_outer(k3) TempExplodeView as explode_column;
+------+----------------+
| k1 | explode_column |
+------+----------------+
| 1 | 1 |
| 1 | 2 |
| 2 | NULL |
| 4 | NULL |
| 3 | NULL |
+------+----------------+
```
4. explode usage example. explode return empty rows while the ARRAY is null or empty
```
> select k1,explode_column from array_test LATERAL VIEW explode(k3) TempExplodeView as explode_column;
+------+----------------+
| k1 | explode_column |
+------+----------------+
| 1 | 1 |
| 1 | 2 |
+------+----------------+
```
2022-04-08 12:11:04 +08:00
bd0a3369b7
[fix] check disk capacity before writing data ( #8887 )
...
1. We forgot to check disk capacity when writing data.
2. TODO: the user specified disk capacity is not used now. We need to find a way to use it.
3. Avoid print too much compaction log when there is not suitable version for compaction.
2022-04-08 11:29:49 +08:00
f854f0e83e
remove unreadable char in comment ( #8909 )
2022-04-08 09:26:53 +08:00
3dd6b42781
[fix](datax) Fix the problem of keyword error when importing datax ( #8893 )
2022-04-08 09:20:54 +08:00
dbbc6549bd
[feature](vectorized) support vexplode_bitmap ( #8890 )
2022-04-08 09:20:26 +08:00
fa8e4ec2f0
[fix] Disable cast operation of object type ( #8882 )
...
Disable cast between string and object type(bitmap, hll, quantile_state)
2022-04-08 09:13:56 +08:00
69ab1f8681
[doc] add doc of fe dev in vscode ( #8875 )
2022-04-08 09:12:43 +08:00
3f04220d49
[typo] Fix typo in function.cpp ( #8873 )
2022-04-08 09:09:19 +08:00
e3daa9580a
[Fix](Lateral View) The Error expr type when exploding a function result of inline view ( #8851 )
...
Fixed #8850
The column in inline view maybe a function instead of slotRef.
So when this column is used as the input of explode function,
it can't be converted to slotRef.
The correct way is to treat it as an Expr and extract the required slotRef for materialization.
For example:
```
with d as (select k1+k1 as k1_plus from table)
select k1_plus from d explode_split(k1_plus, ",")
```
FnExp: SlorRef<k1_plus>
SubstituteFnExpr: functionCallExpr<k1+k1>
originSlotRefList: SlotRef<k1>
2022-04-08 09:08:55 +08:00
318feb01f3
[improvement](account) support to account management sql ( #8849 )
...
Add [IF EXISTS] support to following statements:
- CREATE [IF NOT EXISTS] USER
- CREATE [IF NOT EXISTS] ROLE
- DROP [IF EXISTS] USER
- DROP [IF EXISTS] ROLE
2022-04-08 09:08:08 +08:00
0b98d78664
[improvement](hll) Optimize Hyperloglog ( #8829 )
...
In meituan, pr #6625 was revert due to the oom probleam.
currently, we are trying to modify the old hyperloglog, based on pr #8555 , we did some works.
via some test, we find it better than old hll, and better than apache:master hll.
Changes summary:
- use SIMD max tp speed up heavy function _merge_registers
- use phmap::flat_hash_set rather than std::set
- replace std::max
- other small changes
2022-04-08 09:06:08 +08:00
b88bf73ca7
[refactor][doc] Added doc for compilation, deployment and data export ( #8776 )
2022-04-08 09:04:03 +08:00
519305cb22
[feature-wip] (memory tracker) (step4) Switch TLS mem tracker to separate more detailed memory usage ( #8669 )
...
Based on #8605 , Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.
2022-04-08 09:02:26 +08:00
7fb4b6a6e2
[chore](tsan) add file mremap_fallback for tsan ( #8665 )
2022-04-08 09:01:53 +08:00
24bb9810b4
[doc](manager) Add space list documents ( #8658 )
...
Add space list and access control document. Remove some pictures to reduce the size of source code.
2022-04-08 09:01:23 +08:00
d51545a952
[fix](ut)(memory-leak) Fix be asan ut failed and hdfs file reader memory leak ( #8905 )
2022-04-08 00:07:00 +08:00
2a25b90cb3
[Test] Fix explode test and build fail ( #8885 )
2022-04-07 14:23:57 +08:00
32bba15e34
[refactor][fix] remove useless import in Config.java ( #8878 )
2022-04-07 11:40:05 +08:00
c9cb07a270
[typo](doc)Update upgrade.md ( #8866 )
2022-04-07 11:36:39 +08:00
02be8176c3
[fix] access parallel_flat_hash_map via thread safely methods ( #8854 )
...
Iterator of parallel_flat_hash_map is not thread safely, so
we should use if_contains instead.
2022-04-07 11:35:59 +08:00
64d18364db
[improvement](restore) set table property 'dynamic_partition.enable' to false after restore ( #8852 )
...
when restore table with dynamic partition properties, 'dynamic_partition.enable' is set to the backup time value.
but Doris could not turn on dynamic partition automatically when restore.
So we cloud see table never do dynamic partition with dynamic_partition.enable is set to 'true'.
2022-04-07 11:34:01 +08:00
ce50c4d826
[feature](diagnose) support "ADMIN DIAGNOSE TABLET" stmt ( #8839 )
...
`ADMIN DIAGNOSE TABLET tablet_id`
This statement makes it easier to quickly diagnose the status of a tablet.
See "ADMIN-DIAGNOSE-TABLET.md" for details
```
mysql> admin diagnose tablet 10196;
+----------------------------------+------------------------------+------------+
| Item | Info | Suggestion |
+----------------------------------+------------------------------+------------+
| TabletExist | Yes | |
| TabletId | 10196 | |
| Database | default_cluster:db1: 10192 | |
| Table | tbl1: 10194 | |
| Partition | tbl1: 10193 | |
| MaterializedIndex | tbl1: 10195 | |
| Replicas(ReplicaId -> BackendId) | {"10197":10002} | |
| ReplicasNum | OK | |
| ReplicaBackendStatus | Backend 10002 is not alive. | |
| ReplicaVersionStatus | OK | |
| ReplicaStatus | OK | |
| ReplicaCompactionStatus | OK | |
+----------------------------------+------------------------------+------------+
```
2022-04-07 11:30:03 +08:00
ca4055244e
[fix](storage) Fix core bug of convert to predicate column ( #8833 )
...
recurrent:
When `enable_low_cardinality_optimize = true`, for the TPCH dataset, using the following SQL query will Core
```sql
select count(*) from lineitem where l_comment = 'ously even exc';
```
This SQL will trigger the execution of `ColumnDictionary::convert_to_predicate_column_if_dictionary`, and `res->reserve(_codes.size())` is problematic because the current `_codes.size()` is smaller than its reserve value, so inserting a value into `PredicateColumn` will Core.
2022-04-07 11:29:26 +08:00
e72ccfd80c
[Refactor][httpv2]remove http v1 code ( #8848 )
...
http v2 has been actually tested in production, and it is completely replaceable to have http code. In order to simplify code maintenance, remove the previous http part of the code
2022-04-07 08:38:29 +08:00
98cab78320
[refactor](schema_hash) remove schema_hash since every tablet id in be is unique ( #8574 )
2022-04-07 08:37:45 +08:00
57638ae43d
[Refactor][Doc]Add part of the document content for data import ( #8772 )
2022-04-07 08:37:11 +08:00
e53c90fbef
min and max window function bug fix ( #8822 )
...
[Fix bug] min and max window function bug fix #8822
2022-04-07 08:36:33 +08:00
319f1f634a
[fix](ut) fix fe run CreateTableAsSelectStmtTest ,UserPropertyTest, ProjectPlannerFunctionTest and AggregateTest failed ( #8838 )
2022-04-06 15:23:49 +08:00
f90a1a1919
[fix](ut)(compile) Fix ut failure at functions_geo and compilation bug ( #8843 )
2022-04-05 21:30:40 +08:00
03c5d5d677
fix some error on build.sh && fix build fail with clang on runtime_profile ( #8748 )
2022-04-05 15:52:53 +08:00
d07b49247e
rm sequential file ( #8713 )
...
[refactor]remove sequential file reader from env
2022-04-04 17:49:06 +08:00
fcefed7c1c
[Bug][Vectorized] Fix core bug of segment vectorized ( #8800 )
...
* [Bug][Vectorized] Fix core bug of segment vectorized
1. Read table with delete condition
2. Read table with default value HLL/Bitmap Column
* refactor some code
Co-authored-by: lihaopeng <lihaopeng@baidu.com >
2022-04-03 19:50:25 +08:00
33736e45fa
[fix](table-function) Fixed unreasonable nullable conversion ( #8818 )
2022-04-03 11:02:35 +08:00
a8417e6c8b
[fix](restore) fix restore issue when meta version is too low ( #8816 )
...
When restore snapshot from 0.13 to master, the restore job is pending for long time.
However, we get error "Could not set meta version to 93 since it is lower than minimum required version 100" in log.
We should cancel restore job once get that error.
2022-04-03 10:56:23 +08:00
78b85414d6
[fix](debug) get_hash_value_fvn DCHECK failed ( #8811 )
...
* fix_get_hash_value_fvn
* fix compile
2022-04-03 10:55:15 +08:00
eed4908790
[chore](deps) upgrade spring to 2.6.2 to 2.6.6 ( #8802 )
2022-04-03 10:52:31 +08:00
f3c6ddf651
[feature](function) Support geolocation functions on vectorized engine ( #8790 )
2022-04-03 10:50:54 +08:00
586bec79f5
[fix](storage) Fix query result error due to find code by bound ( #8787 )
...
Problem recurrence
SSB single table `lineorder_flat`, the query SQL is as follows:
```sql
SELECT
sum(LO_REVENUE),
(LO_ORDERDATE DIV 10000) AS year,
P_BRAND
FROM lineorder_flat
WHERE P_BRAND >= 'MFGR#22211111' AND P_BRAND <= 'MFGR#22281111' AND S_REGION = 'ASIA' and (LO_ORDERDATE DIV 10000) = 1992
GROUP BY
year,
P_BRAND
ORDER BY
year,
P_BRAND;
```
when `enable_low_cardinality_optimize=false`, query result：
```sql
+-------------------+------+-----------+
| sum(`LO_REVENUE`) | year | P_BRAND |
+-------------------+------+-----------+
| 65423264312 | 1992 | MFGR#2222 |
| 66936772687 | 1992 | MFGR#2223 |
| 64047191934 | 1992 | MFGR#2224 |
| 65744559138 | 1992 | MFGR#2225 |
| 66993045668 | 1992 | MFGR#2226 |
| 67411226147 | 1992 | MFGR#2227 |
| 69390885970 | 1992 | MFGR#2228 |
+-------------------+------+-----------+
```
when `enable_low_cardinality_optimize=true`, query result：
```sql
+-------------------+------+-----------+
| sum(`LO_REVENUE`) | year | P_BRAND |
+-------------------+------+-----------+
| 66936772687 | 1992 | MFGR#2223 |
| 64047191934 | 1992 | MFGR#2224 |
| 65744559138 | 1992 | MFGR#2225 |
| 66993045668 | 1992 | MFGR#2226 |
| 67411226147 | 1992 | MFGR#2227 |
| 69390885970 | 1992 | MFGR#2228 |
+-------------------+------+-----------+
```
One line less than the correct result.
The reason is that 'MFGR#22211111' is not in the dictionary, so get the boundary code (`find_code_by_bound` method), but there is a bug here.
2022-04-03 10:38:14 +08:00
