Using cache to store external table columns, doesn't persist uniq id for external columns anymore.
So use -1 as column id for ES external table.
Avoid non-master FE trying to get uniq id problem. The problem will cause non-master FE fail to write bdbje.
1.Support in bitmap syntax, like 'where k1 in (select bitmap_column from tbl)';
2.Support bitmap runtime filter. Generate a bitmap filter using the right table bitmap and push it down to the left table storage layer for filtering.
Problem:
We got following error frequently while SELECT xxx INTO OUTFILE:
ERROR 1064 (HY000): RpcException, msg: Fail to write to broker, broker:TNetworkAddress(hostname=a.b.c.d, port=8111) failed:write() send(): Broken pipe
Reason:
we cache broker thrift client in BE;
thrift client check connect isOpen only return cached flag, not care the real socket is opened or closed;
after we get client from cache, the socket may already closed, then pwrite will failed.
How to fix:
Other interfaces such as open and close, will reopen and retry again, but pwrite do not retry.
As there are write offset inside pwrite, and the broker(server) side also will check the write offset, it is safe to retry pwrite.
in the previous, the result is:
```
mysql> select array_position([1, null], null);
+--------------------------------------+
| array_position(ARRAY(1, NULL), NULL) |
+--------------------------------------+
| NULL |
+--------------------------------------+
1 row in set (0.02 sec)
```
but after this commit, the result become:
```
mysql> select array_position([1, null], null);
+--------------------------------------+
| array_position(ARRAY(1, NULL), NULL) |
+--------------------------------------+
| 2 |
+--------------------------------------+
1 row in set (0.02 sec)
```
The run length of null map is saved as `uint16_t`. Previously, the run length of null map was
limited by `batch_size` in the `ParquetReader`, by setting `batch_size = std::min(batch_size, (size_t)USHRT_MAX)`.
It works well when the batch size is less than `USHRT_MAX`.
However, [Lazy read](https://github.com/apache/doris/pull/13917) will merge empty batches until reading
a non-empty batch or reaching the EOF of a row group, so the `batch_size` may be greater than `USHRT_MAX`
in non-predicate columns.
In addition, even if the `batch_size` does not exceed `USHRT_MAX`, the adjacent batches may also make
the run length exceed the `USHRT_MAX` in `ColumnSelectVector::get_next_run`.
This pr contribute:
- support explain CTE;
- refine CTE, fix the bug: reuse the same analyzed plan which LogicalOlapScan has the same relationId;
- change EliminateAliasNode to LogicalSubQueryAliasToLogicalProject and move to the top of rewrite stage, so we can simply observe the analyzed plan by the LogicalSubQueryAlias with alias;
- job traverse left child first, so the ExprId growth from left child to right child.
in #14482, we implement the feature to keep specific number of meta with same name in catalog recycle bin.
But it will cause meta replay bug.
Because every time we drop db/table/partition, it will try to erase a certain number of meta with same name.
And when replay "drop" edit log, it will do same thing. But the number of meta to erase it based on current config value,
not persist in edit log, so it will cause inconsistency with "drop" and "replay drop".
In this PR, I move the "erase meta with same name" logic to the daemon thread of catalog recycle bin.
In #14533 , we run BE UT workflows periodically to share the cache with brand new pull requests. However, we don't need to save the cache when the unit tests doesn't run, otherwise it may occupy huge cache space and some useful caches will be evicted by GitHub.
This PR is to make sharing hash table for broadcast more robust:
Add a session variable to enable/disable this function.
Do not block the hash join node's close function.
Use shared pointer to share hash table and runtime filter in broadcast join nodes.
The Hash join node that doesn't need to build the hash table will close the right child without reading any data(the child will close the corresponding sender).
Issue Number: close#13615
The main work:
implement grouping sets/ cube/ rollup.
fix if function Infinite loop problem.
Support for isNull transitions to legacy optimizers.
1. in dateV2, we adjust the dir structure to avoid creating a tpch-1G database
2. use `drop table XXX` to replace `delete * from XXX where key>0`
3. remove explain cases, because
- the explain string itself is variable, and the case is hard to maintain
- it is original planner explain, not nereids
Currently, we build the third-party libraries and release them automatically (See https://github.com/apache/doris-thirdparty/pull/13). We must make sure that the changes for third-party libraries are valid.
Previously, we use "Date" type for cooldownTime in StoragePolicy.
But the serialization method of Date type in Gson is different in java8 and java11, which may cause inconsistent meta error.
This PR use Long to save cooldownTime.
And notice that in FE, the cooldownTime is saved in milliseconds, and in BE, it is saved in seconds.
Origin code using Set to store hms external table partition columns,
which couldn't guarantee the order of the columns.
This could cause the column name and column type doesn't match.
Using List instead of Set to fix the problem.
