When LRUCache insert and evict a large number of entries, there are
frequently calls of HandleTable::remove(e->key, e->hash), it will
lookup the entry in the hash table. Now that we know the entry to
remove 'e', we can remove it directly from hash table's collision list
if it's a double linked list.
This patch refactor the collision list to double linked list, the simple
benchmark CacheTest.SimpleBenchmark shows that time cost reduced about
18% in my test environment.
The name and another config name are close to each other and are indistinguishable.
So this pr modify the name.
The document description has also been changed
(1) The implementation logic of `tablets web page` is that:
Firstly, get all the tablets of the BE;
Secondly, return specific number tablets in front to web page according to the value of request parameter `limit`.
This patch optimize the implementation logic of `tablets web page` that getting specific number tablets in BE
according to the value of request parameter `limit` and then return them to web page.
(2) It will return default `1000` tablets through http interface `http://be_host:webserver_port/tablets_page`.
If the number of tablet in the BE is less than 1000, there will be `coredump` in the following code:
be/src/http/action/tablets_info_action.cpp
Support persistence of configuration items modified at runtime via HTTP API.
```
FE:
GET /api/_set_config?key=value&persist=true
BE
POST /api/update_config?key=value&persist=true
```
The modified config will be saved in `fe_custom.conf` or `be_custom.conf`.
And when process starts, it will load `fe.conf/be.conf` first, then `fe_custom.conf/be_custom.conf`.
When selecting candidate rowsets to do the cumulative compaction,
some rowsets may not be selected because the protection time has not expired.
Therefore, we need to find the current longest continuous version path in the candidate rowsets.
We create some objects which are only used for unit tests, it's not necessary,
and it may cause create duplicate instances for some classes.
This patch remove unnecessary instance of class BlockManager and StoragePageCache.
At present, when some rpc errors occur, the client cannot obtain the error information well.
And this CL change the RPC error returned to client like this:
```
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(A error
occurred: errorCode=2001 errorMessage:Channel inactive error!)]
ERROR 1064 (HY000): failed to send brpc batch, error=The server is overcrowded, error_text=[E1011]The server is
overcrowded @xx.xx.xx.xx:8060 [R1][E1011]The server is overcrowded @xx.xx.xx.xx:8060 [R2][E1011]The server is
overcrowded @xx.xx.xx.xx:8060 [R3][E1011]The server is overcrowded @xx.xx.xx.xx:8060, client: yy.yy.yy.yy
```
1. Support limit clause push down both odbc table and mysql table.
2. Code refactor of ODBC Scan Node, change `build_connect_string` and `query_string` from BE to FE to make it easily to modify
Currently, there are M threads to do base compaction and N threads to do cumulative compaction for each disk.
Too many compaction tasks may run out of memory, so the max concurrency of running compaction tasks
is limited by semaphore.
If the running threads cost too much memory, we can't defense it. In addition, reducing concurrency to avoid OOM
will lead to some compaction tasks can't be executed in time and we may encounter more heavy compaction.
Therefore, concurrency limitation is not enough.
The strategy proposed in #3624 may be effective to solve the OOM.
A CompactionPermitLimiter is used for compaction limitation, and use single-producer/multi-consumer model.
Producer will try to generate compaction tasks and acquire `permits` for each task.
The compaction task which can hold `permits` will be executed in thread pool and each finished task will
release its `permits`.
`permits` should be applied for before a compaction task can execute. When the sum of `permits`
held by executing compaction tasks reaches a threshold, subsequent compaction task will be no longer allowed,
until some `permits` are released. Tablet compaction score is used as `permits` of compaction task here.
To some extent, memory consumption can be limited by setting appropriate `permits` threshold.
#4619
Add time_round functions that provides `time_floor` & `time_ceil` at each time unit.
Fix two related bugs.
- #4618
- Fix `struct TimeInterval` to use `int64_t` instead of `int32_t`, in case when the second diff overflow
Use static local variable instead of create it every calls.
Time cost of the new added unit benchmark test could reduce
from about 60 seconds to 10 seconds.
* When the different partition of the table is updated frequently, the partition key list of the cache is discontinuous,
and the partition key in the request cannot hit the key list in the cache, resulting in the access overrun，the BE will crash.
* Add some unit test case，add test cases that fail to hit the boundary value of cache
Describe the bug
DATA_TYPE in information_schema.columns is not compatible to mysql meta
To Reproduce
Steps to reproduce the behavior:
select * from information_schema.columns
Expected behavior
the result of data_type is (int, decimal, char, varchar, ...)，but doris data_type is (int(11), varchar(20), ...)
Excess number will affect some BI systems or upper system can't get right type
Suppose that the current available byte in a disk is `_available_bytes`. After the data with size of `incoming_data_size` is written in the disk, the left space in the disk should be calculated as follow :
`int64_t left_bytes = _available_bytes - incoming_data_size;`
rather than:
`int64_t left_bytes = _disk_capacity_bytes - _available_bytes - incoming_data_size;`
The tablet and disk information reporting threads need to report to the FE periodically.
At the same time these two reporting threads will also be triggered by certain events.
The modification in PR #4440 caused these two threads to be triggered only by events,
and could not report regularly.
1. Find the cache node by SQL Key, then find the corresponding partition data by Partition Key, and then decide whether to hit Cache by LastVersion and LastVersionTime
2. Refers to the classic cache algorithm LRU, which is the least recently used algorithm, using a three-layer data structure to achieve
3. The Cache elimination algorithm is implemented by ensuring the range of the partition as much as possible, to avoid the situation of partition discontinuity, which will reduce the hit rate of the Cache partition,
4. Use the two thresholds of maximum memory and elastic memory to control to avoid frequent elimination of data
When the keystype of mv and base table is difference, Doris should execute
sorting schema change instead of linked schema change.
If doesn't, the data size of mv actually is same as base table.
This will cause mv to have no pre-aggregation effect at all.
The query will not choose mv.
This commit fixed this problem. Fixed#4586
After 0.12 version, doris remove the format convert functiion which can convert from hdr_ format
to tabletmeta_ format when loading metas, the commit link: 3bca253
When we update doris version and there are old format meta in storage,
BE will not read the old format tablet. It can lead to data loss.
So we add meta format detection function to prevent data loss.
When there are old format meta in olap_meta, BE can find and print log or exit.
For[Spark Load]
1 support decimal andl largeint
2 add validate logic for char/varchar/decimal
3 check data load from hive with strict mode
4 support decimal/date/datetime aggregator
