Now we make wrong for decimal parse from string
if given string precision is bigger than defined decimal precision, we will return a overflow error, but only digit part is bigger than typed digit length , we should return overflow error when we traverse given string to decimal value
Jemalloc heap profile follows libgcc's way of backtracing by default.
rewrites dl_iterate_phdr will cause Jemalloc to fail to run after enable profile.
TODO, two solutions:
- Jemalloc specifies GNU libunwind as the prof backtracing way, but my test failed,
--enable-prof-libunwind not work: --enable-prof-libunwind not work jemalloc/jemalloc#2504
- ClickHouse/libunwind solves Jemalloc profile backtracing, but the branch of ClickHouse/libunwind
has been out of touch with GNU libunwind and LLVM libunwind, which will leave the fate to others.
The error message was not good for not exist column while show column stats:
```
MySQL [hive.tpch100]> show column stats `lineitem` (l_extendedpric);
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: null
```
This pr show a meaningful message:
```
mysql> show column stats `lineitem` (l_extendedpric);
ERROR 1105 (HY000): errCode = 2, detailMessage = Column: l_extendedpric not exists
```
use three new plan node to represent defer materialize of TopN.
Example:
```
-- SQL
select * from t1 order by c1 limit 10;
-- PLAN
+------------------------------------------+
| Explain String |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink |
| --PhysicalDeferMaterializeTopN |
| ----PhysicalDistribute |
| ------PhysicalDeferMaterializeTopN |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
1. add more logs and make error messages more clear
2. sleep a while between retry analyze
3. make concurrency of sync analyze configurable
4. Ignore internal columns like delete sign to save resources
Fix error when reading empty map values in parquet. The `offsets.back()` doesn't not equal the number of elements in map's key column.
### How does this happen
Map in parquet is stored as repeated group, and `repeated_parent_def_level` is set incorrectly when parsing map node in parquet schema.
```
the map definition in parquet:
optional group <name> (MAP) {
repeated group map (MAP_KEY_VALUE) {
required <type> key;
optional <type> value;
}
}
```
### How to fix
Set the `repeated_parent_def_level` of key/value node as the definition level of map node.
`repeated_parent_def_level` is the definition level of the first ancestor node whose `repetition_type` equals `REPEATED`. Empty array/map values are not stored in doris column, so have to use `repeated_parent_def_level` to skip the empty or null values in ancestor node.
For instance, considering an array of strings with 3 rows like the following:
`null, [], [a, b, c]`
We can store four elements in data column: `null, a, b, c`
and the offsets column is: `1, 1, 4`
and the null map is: `1, 0, 0`
For the `i-th` row in array column: range from `offsets[i - 1]` until `offsets[i]` represents the elements in this row, so we can't store empty array/map values in doris data column. As a comparison, spark does not require `repeated_parent_def_level`, because the spark column stores empty array/map values , and use anther length column to indicate empty values. Please reference: https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
Furthermore, we can also avoid store null array/map values in doris data column. The same three rows as above, We can only store three elements in data column: `a, b, c`
and the offsets column is: `0, 0, 3`
and the null map is: `1, 0, 0`
The original logic is to first deserialize the ColumnString into a HashSet (insert the deserialized elements into the hashset), and then traverse all the HashSet elements into the target HashSet during the merge phase.
After optimization, when deserializing, elements are directly inserted into the target HashSet, thereby reducing unnecessary hashset insert overhead.
In one of our internal query tests, 30 hashsets were merged in second phase aggregation(the average cardinality is 1,400,000), and the cardinality after merging is 42,000,000. After optimization, the MergeTime dropped from 5s965ms to 3s375ms.
Problem:
When create view with join in table partitions, an error would rise like "Unknown column"
Example:
CREATE VIEW my_view AS SELECT t1.* FROM t1 PARTITION(p1) JOIN t2 PARTITION(p2) ON t1.k1 = t2.k1;
select * from my_view ==> errCode = 2, detailMessage = Unknown column 'k1' in 't2'
Reason:
When create view, we do tosql first in order to persistent view sql. And when doing tosql of table reference, partition key
word was removed to keep neat of sql string. But here when we remove partition keyword it would regarded as an alias.
So "PARTITION" keyword can not be removed.
Solved:
Add “PARTITION” keyword back to tosql string.
