After tablet level metrics is supported, the http metrics API may response
a very large body when a BE holds a large number of tablets, and cause heavy
network traffic.
This patch introduce http content compression to reduce network traffic.
Many tables are so large that need seperate partitions with "HOUR" time unit.
But now dynamic partition doesn't support "HOUR" time unit and it was marked as "TODO".
So I support the feature and it works.
Add tracing broker log. When fe get filestatus for distributing load task to broker,
the broker maybe get empty files and not give correct error code.
Add this log to easy track which broker process filestatus operation and we can get the error log.
BE can not graceful exit because some threads are running in endless
loop. This patch do the following optimization:
- Use the well encapsulated Thread and ThreadPool instead of std::thread
and std::vector<std::thread>
- Use CountDownLatch in thread's loop condition to avoid endless loop
- Introduce a new class Daemon for daemon works, like tcmalloc_gc,
memory_maintenance and calculate_metrics
- Decouple statistics type TaskWorkerPool and StorageEngine notification
by submit tasks to TaskWorkerPool's queue
- Reorder objects' stop and deconstruct in main(), i.e. stop network
services at first, then internal services
- Use libevent in pthreads mode, by calling evthread_use_pthreads(),
then EvHttpServer can exit gracefully in multi-threads
- Call brpc::Server's Stop() and ClearServices() explicitly
We store all table ids involved in the Load Job in TransactionState.
However, for Hadoop Load job, table ids are set incorrectly.
This caused the WAITING_TXN phase to not correctly wait for the completion
of the previous load transaction when doing the alter table,
which caused some data version loss problems.
Fix segment group add zone map bug when schema change.
(1) WrapperField null point check
(2) in DUP_KEYS, let _zone_maps index consistent with _schema column index
we can create odbc_table use SQL like
```
CREATE EXTERNAL TABLE `baseall_oracle` (
`k1` decimal(9, 3) NOT NULL COMMENT "",
`k2` char(10) NOT NULL COMMENT "",
`k3` datetime NOT NULL COMMENT "",
`k5` varchar(20) NOT NULL COMMENT "",
`k6` double NOT NULL COMMENT ""
) ENGINE=ODBC
PROPERTIES (
"host" = "192.168.0.1",
"port" = "8086",
"user" = "happenlee",
"password" = "doris",
"database" = "doris",
"table" = "baseall",
"driver" = "Oracle 19 ODBC driver",
"type" = "oracle"
);
```
Now we only support Oracle and MySQL Database and this feature default turned off by conf enable_odbc_table.
Main CL:
1. Copy the code from BE to implement the `str_to_date()` function in FE.
2. `str_to_date("2020-08-08", "%Y-%m-%d %H:%i:%s")` will return `2020-08-08 00:00:00` instead of `2020-08-08`.
It is possible to report "Illegal column/field reference'table2.DORIS_DELETE_SIGN' of semi-/anti-join"
when executing a semi/anti join statement on a table with hidden columns.
This is because the filter conditions of semi/anti join cannot added in the where statement.
Now we add delete flag related where predicate in OlapScanNode level.
The parameter 'part' of parse_url function does not support lower case, and parse protocol not right.
And This function does not support parse 'port'.
This PR tries to make parse_url function case insensitive and support parse 'port'.
The issue: #4451
(1) fix recover persistent stale rowsets bug from multi-single version rowset in stale rowsets
(2) delete_expired_inc_rowsets check consistent version convert to [0, max_version]
1. Analyze what mode of cache can be used by query
2. Query cache before executing query in StmtExecutor
3. Two cache mode, sqlcache and partitioncache, are implemented
The param of rand() function should be literal, but current compiler ignore to
validate the literal param of rand function, it is validated in execution step.
This PR make it validated in compile step, and make it more earlier to find the usage error of rand() function.
Sometimes we want to detect the hotspot of a cluster, for example, hot scanned tablet, hot wrote tablet,
but we have no insight about tablets in the cluster.
This patch introduce tablet level metrics to help to achieve this object, now support 4 metrics on tablets: `query_scan_bytes `, `query_scan_rows `, `flush_bytes `, `flush_count `.
However, one BE may holds hundreds of thousands of tablets, so I add a parameter for the metrics HTTP request,
and not return tablet level metrics by default.
In DistributedPlanner, do not add the unnecessary Exchanges.
For case 1, we only need to judge that the table's distribute hash keys is a subset of the aggregate keys.
For case 2, we should judge two conditions:
- partition keys are also hash keys.
- the table's distribute hash keys is a subset of the aggregate keys.
1. When WITH_MYSQL is off, load error hub does not suport MySQL load error hub,
we should check its return value.
2. misjudge the return value of `change_row_block` in schema_change.cpp
Segment index file content is not set as 0 when it is constructed in write procedure,
so when load index from this file, and meet a null VARCHAR cell,
the null field of this cell is 0, but the length field which is not initialized maybe a large random number,
then memory copy may cause overflow.
This patch fix this bug, and also skip useless memory copy to improve a bit of performance.
Persistence stale rowsets meta. When BE reboots, stale rowsets meta
can resume and the stale version can also be readable before stale gc time.
ISSUE: #4453
1. fix write dpp result when dpp throw exception
2. boolean value：true, false(IgnoreCase), 0, 1
3. wrong dest column for source data check
4. support * in source file path
5. if job state is cancelled or finished, submitPushTasks would throw all partitions have no load data exception,
because tableToLoadPartitions was already cleaned up
#3433
In the process of historical data transformation of materialized views, it may occur that the transformation fails due to data quality.
Add an error status code ：OLAP_ERR_DATE_QUALITY_ERR to determine if a data problem is causing the failure
#3344
Send fields after first row arrived so that error packet can be send to client
when exception thrown from coord.getNext().
Golang and Python can not identify error if fields packet arrived before error packet.
This cl will use yarn command as follows to kill or get status of application running on YARN.
```
yarn --config confdir application <-kill | -status> <Application ID>
```
1. Support convert(expr, target_type) function, which is same as CastExpr
2. Support cast (expr as signed/unsigned int)
This is just for compatibility, the signed/unsigned specification is meaningless.
1. Fix core bug wild pointer in PlanFragmentExecutor, fix issue #4447
2. Fix core bug wild pointer json load, fix issue #4452
3. Change the declare order of ODBC type in thrift for compatibility
