validate supported data types checks if a project node's output contains any unsupported data types like array, map, etc in nereids. So this validation should run before EliminateUnnecessaryProject rule
sql : select bitmap_empty() from d_table where true;
should always use base index instead of any mv, because the conjuncts is constant (true) and use none of the column from any mv
Currently, the session variable for Split size will not take effect after the file splits are cached.
1. This PR is to cache file for Hive Table, instead of cache file splits. And split the file every time using the current split size.
2. Use self splitter by default.
In the current implementation of the function of dynamically add and drop inverted index, there is a problem that the inverted index information of historical data is out of date after compaction on the base tablet.
In the future, I will submit PRs to solve this problem. Now, temporarily add or drop inverted index by the directly schema change logic
could reproduced by:
CREATE TABLE t (
name varchar(128)
) ENGINE=OLAP
UNIQUE KEY(name)
DISTRIBUTED BY HASH(name) BUCKETS 1;
insert into t values('abc');
SELECT cd
FROM
(SELECT cast(now() as string) cd FROM t) t1
JOIN
(select cast(now() as string) td from t GROUP BY now()) t2
ON t1.cd = t2.td;
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: null
(affects tpch q14/7/9)
1. equation estimation confidence level
For equation, if any side is almost unique, its estimation confidence is high, we call it trustable condition.
if a join contains more than one un-trustable condition, we only use the one whose selectivity is biggest in order to avoid error propagation.
2. like expression estimation factor: 0.2
give a separate default shrink ratio for like operator, default ratio is 0.2
3. disable fat-child-penalty
set HEAVY_OPERATOR_PUNISH_FACTOR=1
this change affect tpch q15. This factor should be adaptive to the implementation of BE.
#18015 enables stream load profile log, however be will encounter rpc fail when loading tpch data(see #18291). This is because when `is_report_success` is true, be will reportExecStatus to fe, but fe cannot find QueryInfo in `coordinatorMap`, thus it will return error to be.
1. Support the show load warnings for mysql load to get the detail error message.
2. Fix fillByteBufferAsync not mark the load as finished in same data load
3. Fix drain data only in client mode.
we map date/datetime/V2 to double. this map reserves date order, but it does not reserve range length.
For example, from 1990-01-01 to 1991-01-01, there are 12 months. for filter `A < 1990-02-01`, the selectivity
should be `1/12`.
if we compute this filter by their corresponding double value,
`sel = (19900201 - 19900101) / (19910101 - 19900101) = 100/10000 = 1/100`
the error is about 10 times.
This pr aims to fix this error.
Describe your changes.
Solution:
convert double to its corresponding dataType(date/datev2), then compute the range length with respect to its datatype.
1. create a project node to adjust the output column position when a mv is selected in olap scan node
2. pass SlotReference's column info when call Alias's toSlot() method
3. should compare plan's logical properties when compare two plans after rewrite
1. Organize http documents
2. Add http interface authentication for FE
3. **Support https interface for FE**
4. Provide authentication interface
5. Add http interface authentication for BE
6. Support https interface for BE
HiveMetastoreCache type for file split was Hadoop InputSplit. In this pr, change it to Doris defined Split
This change could avoid convert it every time.
Also fix the explain verbose result return -1 for split file length.
Add St_Angle/St_azimuth function：
St_Angle：
Enter three point, which represent two intersecting lines. Returns the angle between these lines. Point 2 and point 1 represent the first line and point 2 and point 3 represent the second line. The angle between these lines is in radians, in the range [0, 2pi). The angle is measured clockwise from the first line to the second line.
`
mysql> SELECT ST_Angle(ST_Point(1, 0),ST_Point(0, 0),ST_Point(0, 1));
+----------------------------------------------------------------------+
| st_angle(st_point(1.0, 0.0), st_point(0.0, 0.0), st_point(0.0, 1.0)) |
+----------------------------------------------------------------------+
| 4.71238898038469 |
+----------------------------------------------------------------------+
1 row in set (0.04 sec)
`
St_azimuth：
Enter two point, and returns the azimuth of the line segment formed by points 1 and 2. The azimuth is the angle in radians measured between the line from point 1 facing true North to the line segment from point 1 to point 2.
`
mysql> SELECT st_azimuth(ST_Point(0, 0),ST_Point(1, 0));
+----------------------------------------------------+
| st_azimuth(st_point(0.0, 0.0), st_point(1.0, 0.0)) |
+----------------------------------------------------+
| 1.5707963267948966 |
+----------------------------------------------------+
1 row in set (0.04 sec)
add regression-test for like this:
mysqldump -h127.0.0.1 -P9030 -uroot --no-tablespaces --databases > /backup/mysqldump/test.db
To prevent errors Unknown table 'column_statistics' in information_schema (1109), the table information_schema.column_statistics was added.
in this pr, we add a new algorithm to estimate semi/anti join row count.
In original alg., we reduce row count from cross join. usually, this is not good.
for example, L left semi join R on L.a=R.a
suppose L is larger than R, and ndv(L.a) < ndv(R.a)
the estimated row count is rowcount(R) * rowcount(L) / ndv(R.a). in most cases, the estimated row count is larger than rowcount(L).
in new alg, we use ndv(R.a)/originalNdv(R.a) to estimate result rowCount. the basic idea is as following:
1. Suppose ndv(R.a) reduced from m to n.
2. Assume that the value space of L.a is the same as R.a if R.a is not filtered.(this assumption is also hold in original alg.)
regard `L left join R` as a filter applied on L, that is, if L.a is in R.a, then this tuple stays in result.
R.a shrinks to m/n, so L.a also shrinks to m/n
1. Fix value idx in bool rle decoder
2. Iceberg table support datetimev2(3). In the previous version, we converted hive timestamp to datetimev2(0) default.
1. Introduce hadoop libhdfs
2. For Linux-X86 platform, use the hadoop libhdfs
3. For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform
Co-authored-by: adonis0147 <adonis0147@gmail.com>
sql like the format (q1, q2, q3 is a query):
``` sql
(q1)
UNION ALL (q2)
UNION ALL (q3)
ORDER BY keys
```
cannot be parsed by nereids, because order will be recognized as an alias of query, we add queryOrganization to avoid it.
