Currently, the compiled output of BE mainly consists of two binaries:
palo_be and meta_tool, which are both around 1.6G in size.
However, the debug information is only needed for debugging purposes.
So I separate the debug info from binaries.
After BE is built, the debug info file will be saved in `be/lib/debug_info/` dir.
`palo_be` and `meta_tool`'s size decrease to about 100MB
This is optional, and default is disabled.
To enable it, use:
`STRIP_DEBUG_INFO=ON sh build.sh`
WindowFunctionLastData::add should keep the last value,
but current implementation keeps the first one.
Obviously, this code is copied from WindowFunctionFirstData::add.
Early Design Documentation: https://shimo.im/docs/DT6JXDRkdTvdyV3G
Implement a new way of memory statistics based on TCMalloc New/Delete Hook,
MemTracker and TLS, and it is expected that all memory new/delete/malloc/free
of the BE process can be counted.
1.
The methods in the IndexChannel are called back in the RpcClosure in the NodeChannel.
However, this callback may occur after the whole task is finished (e.g. due to network latency),
and by that time the IndexChannel may have been destructured, so we should not call
the IndexChannel methods anymore, otherwise the BE will crash.
Therefore, we use the `_is_closed` variable and `_closed_lock` to ensure that the RPC callback function
will not call the IndexChannel's method after the NodeChannel is closed.
2.
Do not add IndexChannel to the ObjectPool.
Because when deconstruct IndexChannel, it may call the deconstruction of NodeChannel.
And the deconstruction of NodeChannel maybe time consuming(wait rpc finished).
But the ObjectPool will hold a SpinLock to destroy the objects, so it may cause CPU busy.
The current BloomFilter runs vectorization predicate evaluate, but `evaluate_vec` interface is not implemented, so the RuntimeFilter does not play a role after it is pushed down to the storage layer.
And BF predicate computation cannot be automatically vectorized, thus making BloomFilter run short-circuit logic.
For SSB Q2.1，`enable_storage_vectorization = true;`
```
test before impl:
- Total: 36s164ms
- RowsVectorPredFiltered: 0
- RealRuntimeFilterType: bloomfilter
- HasPushDownToEngine: true
test after impl:
- Total: 2s345ms
- RowsVectorPredFiltered: 595.247102M (595247102)
- RealRuntimeFilterType: bloomfilter
- HasPushDownToEngine: true
```
Fix a bug when query bitmap return wrong result, even the simplest query.
Such as
```
CREATE TABLE `pv_bitmap_fix2` (
`dt` int(11) NULL COMMENT "",
`page` varchar(10) NULL COMMENT "",
`user_id_bitmap` bitmap BITMAP_UNION NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(`dt`, `page`)
COMMENT "OLAP" DISTRIBUTED BY HASH(`dt`) BUCKETS 2
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
)
Insert any hundreds of rows of data
select count(distinct user_id_bitmap) from pv_bitmap_fix2
the result is wrong
```
This is a bug of vectorization of storage layer.
First step to support Java UDF in Doris. After this PR, we can create Java UDF in doris.
For example, we create Java UDF function by code below.
```
CREATE FUNCTION test_udf(int) RETURNS int
PROPERTIES (
"file"="file:///root/hive-udf-1.0-SNAPSHOT.jar",
"symbol"="udf.Main",
"type"="JAVA_UDF"
)
```
1. `file` indicate where user file is.
2. `symbol` for java udf means udf class in this jar.
3. `type` indicate this function is a java udf.
Due to unlimited queue in OlapScanNode and NodeChannel, memory usage can be
very large for reading and writing large table, e.g 'insert into tableB select * from tableA'.
