At present, the application of vlog in the code is quite confusing.
It is inherited from impala VLOG_XX format, and there is also VLOG(number) format.
VLOG(number) format does not have a unified specification, so this pr standardizes the use of VLOG
1. Schema hash is useless long time ago
Currently, schema hash can only be generated as a random integer, no need to calculated
from real schema.
2. The CRC32 algo is not enough to generate the table' signature.
Table's signature is used to determine whether the tables have the same schema.
And current CRC32 algo may return same signature even if table's schema are different.
So I change it to calculate the md5 of a signature string assembled by schema info of a table.
Currently, fe's SystemMetrics only support tcp. I add system memory metrics for fe.
Then we can get system memory metrics , which is used to troubleshoot memory problems.
A reword suggestion.
Reasons: Before my change, the statement is "If this change need a document change, I have updated the document",
and there is a grammar error in it evidently: "change" cannot be paired with "need".
Either "changes need" or "change need" will be ok at the grammar level.
According to the context, "changes need" will be better.
Now, the statement is "If these changes need document changes, I have updated the document".
When Doris is in debug mode, function `Coordinator#traceInstance` is used to print
the physical execute plan of a fragment instance for debug.
Function `Coordinator#traceInstance` uses param `scanRangeAssignment` to print
the detail of a fragment. But bucket shuffle join and colocate shuffle join do not fill the param.
That will cause debug not work well.
This path fill assignment param of bucket shuffle and colocate shuffle for debug.
Based on PR #4475, this patch add a new feature for single tablet migration between different disks by http.
Co-authored-by: weizuo <weizuo@xiaomi.com>
Currently when a scan node scans many tablets, Doris will assure it load balance when choosing which replica for scan task to be executed. But it does not take other scan nodes into consideration to implement a global load balance. This patch tries to make all tables of all scan nodes to be load balance.
Co-authored-by: wangxixu <wangxixu@xiaomi.com>
ThreadSanitizer, aka TSAN, is a useful tool to detect multi-thread
problems, such as data race, mutex problems, etc.
We should detect TSAN problems for Doris BE, both unit tests and
server should pass through TSAN mode, to make Doris more robustness.
This is the very beginning patch to fix TSAN problems, and some
difficult problems are suppressed in file 'tsan_suppressions', you
can suppress these problems by setting:
export TSAN_OPTIONS="suppressions=tsan_suppressions"
before running:
`BUILD_TYPE=tsan ./run-be-ut.sh --run`
For the task of rebalancing tablet among different disks on the same BE,
It might be an effective strategy to ensure all tablets under the same partition
evenly distribute on the different disks. Thus, it is necessary to obtain the
distribution of tablets under the same partition between different disks on a BE.
This patch add a new http interface for BE to acquire the distribution of tablets
under a partition between different disks on the same BE.
This PR is to reduce lock competition by supporting read and write lock in table level. When we modify or read table's meta, we don't need to get database lock, just get table write or read lock. And when we get database lock, that means meta directly in db cannot be modified by other thread. Database lock only protect meta in Database class, while table lock protect meta in Table class.
Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
Currently, fe thread metrics is very simple, only have thread count and peak_count.
I think we may need more comprehensive prometheus jvm thread metrics on fe.
This will be useful when we want to analysis fe's running status.
If adding the ignore_eovercrowded flag, the `PTabletWriterAddBatchRequest`
won't failed on `EOVERCROWDED` to avoid load jobs failed in this error.
It only effects the NodeChannel(the load job), other rpc requests will still check if overcrowded.
1. fix build thirdparty may be failed in some os, because of default lib path is `lib` or`lib64` or `arrow` bulld failed by `brotil` and `zstd`
2. fix canot extract `.tar.bz2` file
#4995
**Implementation of Separated Page Cache**
- Add config "index_page_cache_ratio" to set the ratio of capacity of index page cache
- Change the member of StoragePageCache to maintain two type of cache
- Change the interface of StoragePageCache for selecting type of cache
- Change the usage of page cache in read_and_decompress_page in page_io.cpp
- add page type as argument
- check if current page type is available in StoragePageCache (cover the situation of ratio == 0 or 1)
- Add type as argument in superior call of read_and_decompress_page
- Change Unit Test
Currently there is no abort transaction related log. When troubleshooting transaction failure
related problems, we need to use some debug tools to find the failed transaction,
so we'd better add a log when abort transaction
#5146
Add histogram metrics into util/metrics.h. The data structure of histogram is implemented in util/histogram.h,
which could also be used in other situations that in need of histogram. Unit tests added as well.
Fix#5138
1. fix bug when create colocate table with empty partition.
2. put code groupName2Id.put(fullGroupName, groupId) to the end to avoid state inconsistent when exception thrown.
3. do not check backendsPerBucketSeq empty in replayAddTableToGroup(),
cause backendsPerBucketSeq can be empty for colocate table with empty partition.
Scanner threads may be running and using the member vars of OlapScanNode,
when the OlapScanNode has already destroyed.
We can use `_running_thread` to be the last accessed member variable.
And `transfer_thread` need to wait for `_running_thread==0`.
After `transfer_thread` joined, `OlapScanNode::close()` can continue.
Bucket shuffle join is an algorithm of joining two tables. Left table is distributed by a column.
Right table sends the data to the left table for joining operation.
It reduces the network cost. But when two table is without any data. Bucket shuffle join will fail.
Related Issue: #5144
