When forwarding show tablet stmt to Master FE. If the result contains null, FE will throw exception.
Fix it by using `\N` instead of "null" in result, to make thrift rpc happy.
The tablet and disk information reporting threads need to report to the FE periodically.
At the same time these two reporting threads will also be triggered by certain events.
The modification in PR #4440 caused these two threads to be triggered only by events,
and could not report regularly.
1. Find the cache node by SQL Key, then find the corresponding partition data by Partition Key, and then decide whether to hit Cache by LastVersion and LastVersionTime
2. Refers to the classic cache algorithm LRU, which is the least recently used algorithm, using a three-layer data structure to achieve
3. The Cache elimination algorithm is implemented by ensuring the range of the partition as much as possible, to avoid the situation of partition discontinuity, which will reduce the hit rate of the Cache partition,
4. Use the two thresholds of maximum memory and elastic memory to control to avoid frequent elimination of data
* [SQL] Support where, limit, order clause in show resourcestmt.
Grammar
SHOW RESOURCES
[
WHERE
[NAME [ = "your_resource_name" | LIKE "name_matcher"]]
[RESOURCETYPE = ["SPARK"]]
]
[ORDER BY ...]
[LIMIT limit][OFFSET offset];
issue #4501
A new FE UI interface implemented with React framework.
Need to work with the new Spring framework and RESTful API in #4596.
This is a simple UI for Doris. User can develope their own UI if they want
Proposal #4308
Use spring mvc rest to replace the original netty http rest
Created a new package `org/apache/doris/httpv2`,
and the origin implementations under `org/apache/doris/http` remain unchanged.
This part of the code will not be used at present, so it will not affect existing functions.
API document can be found in #4584
Proposal #4308
When the keystype of mv and base table is difference, Doris should execute
sorting schema change instead of linked schema change.
If doesn't, the data size of mv actually is same as base table.
This will cause mv to have no pre-aggregation effect at all.
The query will not choose mv.
This commit fixed this problem. Fixed#4586
* [Bug]Fix bug that tablet info with wrong capacity may cause fe oom
* remove unused code
* fix
* remove empty line
Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
After 0.12 version, doris remove the format convert functiion which can convert from hdr_ format
to tabletmeta_ format when loading metas, the commit link: 3bca253
When we update doris version and there are old format meta in storage,
BE will not read the old format tablet. It can lead to data loss.
So we add meta format detection function to prevent data loss.
When there are old format meta in olap_meta, BE can find and print log or exit.
For[Spark Load]
1 support decimal andl largeint
2 add validate logic for char/varchar/decimal
3 check data load from hive with strict mode
4 support decimal/date/datetime aggregator
When Doris create a database, the database of data quota and replica quota is initialized by configuration(Config.default_db_data_quota_bytes and Config.default_db_replica_quota_bytes).
In other words, I convert FeConstants.default_db_data_quota_bytes to Config.default_db_data_quota_bytes
Config.default_db_data_quota_bytes can be change in configuration file.
When fe replay rollup job(v1) with deleted table, it will throw null pointer exception and exit.
This commit ignore this error and print a warning log to avoid fe exit.
Fixed#4571
Change-Id: I302b554a94d42aee645db6b224cd989e00cd3ca6
Fix stale path delete checking logic.
When current main path is version missing, then delete checking logic is always core dumped. So we fix the checking logic to tolerate current main version missing.
After tablet level metrics is supported, the http metrics API may response
a very large body when a BE holds a large number of tablets, and cause heavy
network traffic.
This patch introduce http content compression to reduce network traffic.
Many tables are so large that need seperate partitions with "HOUR" time unit.
But now dynamic partition doesn't support "HOUR" time unit and it was marked as "TODO".
So I support the feature and it works.
Add tracing broker log. When fe get filestatus for distributing load task to broker,
the broker maybe get empty files and not give correct error code.
Add this log to easy track which broker process filestatus operation and we can get the error log.
BE can not graceful exit because some threads are running in endless
loop. This patch do the following optimization:
- Use the well encapsulated Thread and ThreadPool instead of std::thread
and std::vector<std::thread>
- Use CountDownLatch in thread's loop condition to avoid endless loop
- Introduce a new class Daemon for daemon works, like tcmalloc_gc,
memory_maintenance and calculate_metrics
- Decouple statistics type TaskWorkerPool and StorageEngine notification
by submit tasks to TaskWorkerPool's queue
- Reorder objects' stop and deconstruct in main(), i.e. stop network
services at first, then internal services
- Use libevent in pthreads mode, by calling evthread_use_pthreads(),
then EvHttpServer can exit gracefully in multi-threads
- Call brpc::Server's Stop() and ClearServices() explicitly
We store all table ids involved in the Load Job in TransactionState.
However, for Hadoop Load job, table ids are set incorrectly.
This caused the WAITING_TXN phase to not correctly wait for the completion
of the previous load transaction when doing the alter table,
which caused some data version loss problems.
Fix segment group add zone map bug when schema change.
(1) WrapperField null point check
(2) in DUP_KEYS, let _zone_maps index consistent with _schema column index
