Commit Graph

18429 Commits

Author SHA1 Message Date
067a9a3a22 [opt](ES catalog)Add more description for limitations of docvalue_scan (#29420)
> [ignore_above](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html)
Do not index any string longer than this value. Defaults to 2147483647 so that all values would be accepted. Please however note that default dynamic mapping rules create a sub keyword field that overrides this default by setting ignore_above: 256.

When the field value is longer than the limitation, we will get an empty result.
The solution is turn off `enable_docvalue_scan`, to get data from `_source` field.
2024-01-03 11:17:48 +08:00
b3da9bad3e [fix](move-memtable) fix load stream stub fault injection test (#29431) 2024-01-03 11:15:18 +08:00
3a8ae590ec [fix](pipelineX) Fix unexpected OOM on pipelineX (#29436) 2024-01-03 10:40:44 +08:00
83d36927b3 [Choricals] (ci) Temporarily cancel the mandatory restrictions on the 2.0 branch shellCheck (#29442) 2024-01-03 10:39:52 +08:00
2578f61451 [Chore](ci)Temporarily cancel the mandatory restrictions of shellCheck (#29440)
https://github.com/apache/doris/pull/29289
I modified the shell-check attribute, once the PR was combined, I revert this
2024-01-03 10:20:02 +08:00
Pxl
358995e4ac [Chore](sort) fix block used after it was moved(#29416)
/mnt/disk2/pengyu/codebase/apache/doris/be/src/vec/common/sort/heap_sorter.cpp:91:23: error: 'tmp_block' used after it was moved [bugprone-use-after-move,-warnings-as-errors]
2024-01-03 10:02:35 +08:00
cdf2bb24a4 [fix](Nereids) topn runtime filter only support simplest case (#29312)
only support simple case: select ... from tbl [where ...] order by ... limit ...
2024-01-03 09:55:00 +08:00
97234a6485 [Enhancement](inverted index) strictly checkout inverted index properties (#29421) 2024-01-03 09:54:31 +08:00
298d0c6904 [Chore](Notice)Update Notice year (#29415) 2024-01-03 09:50:30 +08:00
f54f79515c [Bug](fix) str_to_date "" should be null (#29402) 2024-01-03 08:25:22 +08:00
1edf5b31b6 [Regression-test](wal) Add fault injection case for wal mem back pressure (#29298) 2024-01-03 00:06:52 +08:00
797238cbb7 [fix](merge-on-write) fix schema change may result in delete bitmap incorrect (#29386) 2024-01-02 23:45:04 +08:00
69a01e0cf5 [improve](move-memtable) skip load stream stub close wait when cancel (#29427) 2024-01-02 23:35:50 +08:00
69524ccf98 [improve](group commit) Group commit support commit by data size (#29428) 2024-01-02 23:20:23 +08:00
d54cde51ba [fix](group commit) fix http stream group commit begin a txn (#29399) 2024-01-02 21:58:00 +08:00
5db496d844 [Improve](Variant) make output stable (#29389) 2024-01-02 20:29:17 +08:00
9490d5e9a2 [Debug](Variant) sanitize variant in write_column_to_mysql (#29380) 2024-01-02 20:28:59 +08:00
243b075eef [improve](move-memtable) report failed tablets with status (#29351) 2024-01-02 20:24:53 +08:00
f2fa62f524 [minor](Nereids): cleanup code (#29413) 2024-01-02 20:13:59 +08:00
af39217d14 [improve](agg)support push down min/max on unique table (#29242) 2024-01-02 19:40:23 +08:00
3eca457edd [fix](statistics)Fix set partition loaded replayer try to write edit log bug (#29395)
While observer or checkpointer replay SetPartitionLoaded log, it shouldn't try to write the log. This pr is to fix the bug.
2024-01-02 19:38:34 +08:00
04abcc1245 [cases](index compaction)Add index compaction cases (#29393)
Add some cases for index compaction:

1. index compaction with duplicate key table
2. index compaction with unique key table 
3. optimize index compaction with multi segments in one  inverted index
2024-01-02 18:29:09 +08:00
Pxl
d474d4e701 [Bug](rollup) fallback and disable tulpe when rollup select failed on old planner (#29226)
* fallback and disable tulpe when rollup select failed on old planner

* add case
2024-01-02 17:54:56 +08:00
b3f69211b7 [refactor](wal) move group commit load content length to runtime state (#29188) 2024-01-02 15:53:38 +08:00
4692a62e47 [fix](planner)nullable info of agg node's intermediate and output slots should be same (#29315) 2024-01-02 15:53:28 +08:00
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
c07e3b7b47 [feature](Nereids): eliminate left outer join by unique (#28853) 2024-01-02 15:50:49 +08:00
2ed122b787 [improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx (#29388)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-02 15:28:27 +08:00
4581618b09 [improvement](disk) pick disk randomly when usage is less than 0.7 (#29368) 2024-01-02 14:08:09 +08:00
eac96005b3 [Fix](Job)Concurrency may result in event loss (#29385) 2024-01-02 14:06:16 +08:00
f6443efe1a [doc](fix)fix kill doc error and add example (#29313) 2024-01-02 14:05:19 +08:00
bf6488e518 [fix](doc) show catalog example output fix (#29150)
Co-authored-by: Rohit Satardekar <rohitrs1983@gmail.com>
2024-01-02 14:05:05 +08:00
90b2ee90b2 [nereids] consider numNulls in filter estimation (#29184)
consider numNulls in filter estimation
2024-01-02 13:51:11 +08:00
b7487430da Revert "[improve](move-memtable) cancel load rapidly when stream close wait (#29322)" (#29371)
This reverts commit bbf58c5aa42d40e66bc6ccc9ed91a4fcb4bdfff7.
2024-01-02 11:32:14 +08:00
7cb3239599 [fix](sink) add writting restriction for OlapTableSinkV2Operator (#29340) 2024-01-02 11:16:06 +08:00
cec69f4cbd [Enhancement](wal) Add timout for wal memory back pressure (#29178) 2024-01-02 11:02:17 +08:00
4cbbd25d8c [fix](ctx) manager the lifecycle of connection context (#29346)
In FrontendService, we may create some connection context and set it as a thread local varaible.
These context should be removed from thread local after call.
Otherwise, it may be reused by other thread incorrectly.
2024-01-01 23:32:28 +08:00
738abac9ed [minor](context) duplicate query context in fragment ctx (#29364)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-01 22:08:23 +08:00
216b53058d [doc](mtmv)mtmv doc (#27549) 2024-01-01 17:46:21 +08:00
3d59f24403 [fix](compatibility) add some keywords (#29251)
1. Add `BINARY` keywords
    `BINARY` is used for tell MySQL to treat a String(or String column) as case sensitive.
     eg: `"abc" = "ABC"` is true, but `BINARY "abc" = "ABC"` is false.

    But in Doris, `"abc" = "ABC"` is false by default.

     I add this `BINARY` keyword just for compatibility, it will take no effect.

2. Add `PARTITIONS` and `AUTO_INCREMENT` as reserved words.

    `PARTITIONS` is the table name in `information_schema` database.
    `AUTO_INCREMENT` is a column name of a table in `information_schema`
2024-01-01 17:20:45 +08:00
01a0f0915f [fix](mtmv)fix insert overwrite will generate garbage temporary partition when restarting FE (#29075) 2024-01-01 08:12:26 +08:00
5985d216f3 [feature](mtmv)support cancel mtmv task command (#29252)
- `CANCEL MATERIALIZED VIEW TASK taskId on mvName`
- CANCEL MATERIALIZED VIEW TASK, tasks("type"="mv") and jobs("type"="mv") support check auth use priv of mv
- tasks and jobs add column mvName and mvDbName,you can use `select * from tasks("type"="mv") where MvName="xxx"` get all tasks of mv
- fix `desc mv all` error
- fix p0 The task sequence is incorrect
2023-12-31 23:10:30 +08:00
b07ce175de Revert "[fix](move-memtable) only check missing tablets when commit i… (#29352)
* Revert "[fix](move-memtable) only check missing tablets when commit info is not empty (#29326)"

This reverts commit 4634c723d7fb86f434060c0e6fa6168d92630cb9.

* Revert "[fix](move-memtable) check missing tablets before commit (#29223)"

This reverts commit fb0ed8c253f5241c23c71fb88f031604b0bf5a1f.
2023-12-31 22:25:25 +08:00
3c6c652997 [Fix](schema change) disable convert light schema change (#28205) (#29300) 2023-12-31 17:02:15 +08:00
cc40cdf2d4 [improve](load) don't limit add_segment running count in load stream (#29332)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2023-12-31 16:47:46 +08:00
9a90c1cd57 [improve](move-memtable)add multi replica fault injection (#29348) 2023-12-31 16:30:27 +08:00
bbf58c5aa4 [improve](move-memtable) cancel load rapidly when stream close wait (#29322) 2023-12-31 16:26:41 +08:00
6c74313f60 [Fix](ut)(show) added UT cases for show backup stmt (#28604)
* Added UT for show backup
 * fixed some toSql() space and missing CLAUSE
 * Updated Doc with relevant syntax
2023-12-31 15:49:57 +08:00
4634c723d7 [fix](move-memtable) only check missing tablets when commit info is not empty (#29326)
Some sink may not get any success tablets, because load stream will only report them at the last CLOSE_LOAD.
For these sinks (no commit info), we should not check missing tablets since it's expected to be empty.
2023-12-31 13:50:13 +08:00
f92c0acba0 [Bug] When using the hive catalog the hive table name is uppercase, the Doris query is reported wrong #29058 (#29059) 2023-12-31 13:48:08 +08:00