All Column create in inlineView will set `allowNull = false`, which will cause `NULL` data in CTE be process will be ignore.
So we should set column in inlineView allowNull to make sure correct of query.
This CL mainly changes:
1. Avoid repeated sending of common components in Fragments
In the previous implementation, a query may generate multiple Fragments,
these Fragments contain some common information, such as DescriptorTable.
Fragment will be sent to BE in a certain order, so these public information will be sent repeatedly
and generated repeatedly on the BE side.
In some complex SQL, these public information may be very large,
thereby increasing the execution time of Fragment.
So I improved this. For multiple Fragments sent to the same BE, only the first Fragment will carry
these public information, and it will be cached on the BE side, and subsequent Fragments
no longer need to carry this information.
In the local test, the execution time of some complex SQL can be reduced from 3 seconds to 1 second.
2. Add the time-consuming part of FE logic in Profile
Including SQL analysis, planning, Fragment scheduling and sending on the FE side, and the time to fetch data.
The function equalSets is not efficient enough currently, the time complexity is O(n^2).
To improve the performance of comparing two lists, this patch tries to use hash map structure
to make the time complexity to be O(n).
Many time, our users want to use UDFs they developed to ETL the data
when loading the data into Doris.
But currently, broker load does not support to use UDF.
As UDF belongs to a database, it needs to check whether it has the SELECT permission of the database.
This patch try to solve this problem.
In colocate join, the memory limit of each instance is usually less than the value of exec_mem_limit,
which could lead to query failure (Memory exceed limit).
Since the purpose of resetting colocate-join memory limit
(/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java) is unclear to me,
I just change the default value of query_colocate_join_memory_limit_penalty_factor from 8 to 1, as a hotfix.
I use containerized deployment of BE nodes, both using the same distributed disk.
When doing data migration, the current logic will lead to errors.
For example, my distributed disk has 10t and has been used by other services for 9T,
at this time, it is assumed that all the 9T data is used by BE nodes
This reverts commit c8df76a807b4856f71bcb6a3a023849f3bf294d7.
This commit has some problem when handling predicate like:
`k1 = "2020-10-10 10:00:00.000"`
This is a valid predicate, and FE Datetime can not support milli or micro seconds, so it will treat it as invalid date time value.
So we revert it, and may find some better solution later.
We use 'LastStartTime' in backends list to check whether there is an unexpected
restart of BE, but it will be changed as BE's first heartbeat time after FE
restarted, it would be better to set it to BE's actual start time.
* [Broker Load] Ignore empty file when file format is parquet or orc.
We can not ready empty parquet or orc format file. So we should skip them
when doing broker load.
* [Bug] Fix some bugs of load job scheduler
1. The fix load meta bug logic should be removed since 0.12.
2. The load task thread pool's waiting queue should be as long as desired pending jobs num.
3. Submit the load task outside database lock to prevent holding lock for long time.
* Update fe-idea-dev.md
use `brew install thrift@0.9` to install thrift 0.9.3.1
`brew edit thrift090 | head` shows thrift@0.9 uses thrift 0.9.3.1
* [Refactor] Remove the unnecessary if statement
Future<?> submit(Runnable task)
Submits a Runnable task for execution and returns a Future representing that task. The Future's get method will return null upon successful completion.
When `Load Job Task Queue` is filled, continue to submit more jobs to the queue will cause
`RejectedExecutionException`.
But `callback.onTaskFailed` function does not catch the exception, that will cause
re-submitting job failed, and status is not updated to failed.
issue: #4795
SQL like:
`select a join b union select c join d`;
if a b is colocate join, and c d is also colocate join, the query may failed
with error like:
`failed to get tablet. tablet_id=26846, with schema_hash=398972982, reason=tablet does not exist`
1. Add a search boxer in the left tree view of Playground.
2. Fix some visual bugs of UI.
3. Fix bugs that link failed in QueryProfile view.
4. Fix bugs that cookie is always invalid.
5. Set cookie to HTTP_ONLY to make it more safe.
`select day from test where day='2020-10-32'`
table 'test' is parititioned by day. In this case, '2020-10-32' will be taken as CastExpr not LiteralExpr,
and condition "day='2020-10-32'" will not be recognized as partitionfilter.
This case will scan all partitions. To avoid scall all partitions, it is better to filter invalid date value.
issue: #4755
1.
When we decommission some BEs with SSD disks,
if there are no SSD disks on the remaining BEs, it will be impossible to select a suitable destination path.
In this case, we need to ignore the storage medium property and try to select the destination path again.
Set `isSupplement` to true will ignore the storage medium property.
2.
When the BE nodes where all replicas of a tablet are located are decommission,
and this task is a VERSION_INCOMPLETE task.
This will lead to failure to select a suitable dest replica.
At this time, we should try to convert this task to a REPLICA_MISSING task, and schedule it again.
Support persistence of configuration items modified at runtime via HTTP API.
```
FE:
GET /api/_set_config?key=value&persist=true
BE
POST /api/update_config?key=value&persist=true
```
The modified config will be saved in `fe_custom.conf` or `be_custom.conf`.
And when process starts, it will load `fe.conf/be.conf` first, then `fe_custom.conf/be_custom.conf`.
Thread [Connect-Scheduler-Check-Timer] fail to iterate the connectionMap , because other threads will modify the connectionMap at the same time.
Fe's connection which is timed out can't be released because of thread [Connect-Scheduler-Check-Timer] not working.
