* [Bug](point query) checkAndSetPointQuery before checkEnableTwoPhaseRead
1. checkEnableTwoPhaseRead rely on thr short circuit flag
2. add more metric to display lookup profile
* fix rebase
#18976 introduced merge small IO facility to optimize performance, and used by parquet reader.
This PR support this facility in orc reader. Current ORC reader implementation need to reposition parent present stream when reading lazy columns in lazy materialization facility. So let it works by removing `DCHECK_GE(offset, cached_data.end_offset)`.
Fix errors when inserting string/date/datetime values into SQLServer:
ERROR 1105 (HY000): errCode = 2, detailMessage = (172.21.0.101)[INTERNAL_ERROR]UdfRuntimeException: JDBC executor sql has error:
CAUSED BY: SQLServerException: Invalid column name '2021-10-30'.
When using double quotes enclose string values, it will be parsed as column name, so we should enclose string values with single quotes.
We can not use the string as the variable key to use in the hint.
Before this PR
mysql> SET enable_nereids_planner=true;
Query OK, 0 rows affected (0.01 sec)
mysql> set enable_fallback_to_original_planner=false;
Query OK, 0 rows affected (0.10 sec)
mysql> explain select /*+ SET_var("enable_nereids_planner" = "false") */ 1;
ERROR 1105 (HY000): Exception, msg: Nereids cannot parse the SQL, and fallback disabled. caused by:
no viable alternative at input 'select /*+ SET_var("enable_nereids_planner"'(line 1, pos 27)
After this PR
mysql> SET enable_nereids_planner=true;
Query OK, 0 rows affected (0.01 sec)
mysql> set enable_fallback_to_original_planner=false;
Query OK, 0 rows affected (0.10 sec)
mysql> select /*+ SET_var("enable_nereids_planner" = "false") */ 1;
+------+
| 1 |
+------+
| 1 |
+------+
1 row in set (0.00 sec)
Describe your changes.
Support the string for the hint key in the Parser.
Before change, when doing optimize use Nereids planner, input will serialize to memory first. And when bug happen, it would be dump to minidump file when catching the exception.
We found that serialization process will cause the performance when statistic message too large or when optimization time be small enough.
So the user minidump using should change to ONLY YOU OPEN MINIDUMP SWITCH(set enable_minidump=true;) can you use it.
1. Before this PR if rowset does not contain column which should be read for related SlotDescriptor will call `insert_default` to column, but it's not this real defautl value.Real default value relevant information should be provided by the frontend side.
2. Support fetch when light schema change is not enabled, but disable for AGG or UNIQUE MOR model
When the postgresql bit type size is 1, it reads as a java.lang.boolean via jdbc, and if we match against string,
it will display true or false. But the normal display should be a number,
so when I detect that the size of bit is 1, I will match it with boolean
When FE is deployed on a virtual machine and CN is deployed on k8s, FE needs to use a proxy IP to communicate with
CN nodes, and BE cannot resolve the proxy IP from the local network card.
We change the previous verification rules and obey the IP assigned by the master
1. Fix duplicate '/' in front-end request URI.
2. When the FileSystemSeparator is '\\', replace '\\' as '/'
Co-authored-by: labuladuo <labuladuo@douyu.tv>
Suppose three queries are executed in a resource group with a memory_limit of 8G, and they consume memory of query_a = 3G, query_b = 3G, and query_c = 3G. The total memory used is counted as 9G when the resource group GC is executed, which exceeds the resource group limit and cancels query_a.
When the resource group is next GC, the memory of query_a may not be freed yet, and it will be counted again in the total memory consumed by that resource group, which again exceeds the resource group limit and cancels query_b.
From the user's perspective, it is fine to execute query_a and query_b at the same time, but executing query_ a, query_b and query_c will be cancelled for two queries, which is not as expected.
This pr skips the queries that are cancelled when counting the memory used by the resource group. If this causes the process memory to grow, the process gc will handle it.
/home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:895:21: runtime error: load of value 423208544, which is not a valid value for type 'doris::ReaderType'
/home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_decimal.cpp:260:33: runtime error: load of misaligned address 0x7fa3348b301c for type 'int64_t' (aka 'long'), which requires 8 byte alignment
/home/zcp/repo_center/doris_master/doris/be/src/olap/block_column_predicate.cpp:82:24: runtime error: variable length array bound evaluates to non-positive value 0
/home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_string.h:225:26: runtime error: null pointer passed as argument 2, which is declared to never be null
Support the operator `PartitionTopN`, which can partition first and do the topn operation later in each partition. It used in the following case
```
-- Support push the filter down to the window and generate the PartitionTopN.
-- The plan change from `window -> filter` to `partitionTopN -> window -> filter`.
explain select * from (select * , row_number() over(partition by b order by a) as num from t ) tt where num <= 10;
-- Support push the limit down to the window and generate the PartitionTopN.
-- The plan change from `window -> limit` to `partitionTopN -> window -> limit `.
explain select row_number() over(partition by b order by a) as num from t limit 10;
-- Support push the topn down to the window and generate the PartitionTopN.
-- The plan change from `window -> topn` to `partitionTopN -> window -> topn `.
explain select row_number() over(partition by b order by a) as num from t order by num limit 10;
```
The FE part detail design:
1. Add the following rewrite rules:
- PUSHDOWN_FILTER_THROUGH_WINDOW
- PUSH_LIMIT_THROUGH_PROJECT_WINDOW
- PUSH_LIMIT_THROUGH_WINDOW
- PUSHDOWN_TOP_N_THROUGH_PROJECTION_WINDOW
- PUSHDOWN_TOP_N_THROUGH_WINDOW
2. Add the PartitionTopN node(LogicalPlan/ PhysicalPlan/ TranslatorPlan)
3. For the rewrite plan, there are several requests that need to meet:
- For the `Filter` part, only consider `</ <=/ =` conditions. And the filter conditions will be stored.
- For the `Window` part, we only support one window function. And we support the `row_number`, `rank`, `dense_rank` window functions. And the `partition by` key and `order by` key can not be empty at the same time. The `Window Frame` should be `UNBOUNDED to CURRENT`.
4. For the `PhysicalPartitionTopN`, the requested property is `Any`and the output property is its children's property.
That's the main details that are very important. For the other part, you can directly check the code.
Issue Number #18646
BE Part #19708
If the user manually removed a hive partition (remove the partition dir through hdfs), doris will failed to query the hive
table with an error message get file split failed for table. That is because the Hive metadata still contains the removed partition.
This pr is to fix this bug. Skip the not exist dirs.
The variable logType in ExternalCatalog is not persistent to disk, after refresh, it will be set to NULL and cause NPE. This pr is to fix the bug.
Also, remove the old type variable in ExternalCatalog, use logType instead.
Support collect hive external table statistics by running sql against hive table.
By running sql, we could collect all the statistics collected for Olap table, including the min, max value of String column.
With 3 BE (16 core, 64 GB), it cost less than 2 minutes to collect TPCH 100GB statistics for all columns of all tables.
Also less than 2 minutes to collect all columns statistics for SSB 100GB tables.
1. reduce s3 buffer pool's ctor cost
2. before this pr, if one s3 file writer return err when calling append or close function, the caller will not call abort function which result in one confusing DCHECK failed like the following picture
