Commit Graph

13158 Commits

Author SHA1 Message Date
09bcedb116 [feature](merge-cloud) Remove deprecated old cache (#23881)
* Remove deprecated old cache
2023-09-06 08:07:05 +08:00
44bb94d5e7 [fe](default parameters) change remote_fragment_exec_timeout_ms from 5s to 30s (#23909) 2023-09-06 00:16:23 +08:00
13c9c41c1f [opt](hudi) reduce the memory usage of avro reader (#23745)
1. Reduce the number of threads reading avro logs and keep the readers in a fixed thread pool.
2. Regularly cleaning the cached resolvers in the thread local map by reflection.
2023-09-05 23:59:23 +08:00
d7f1a4fb9a [Fix](inverted index) need_read_data only supports DUP_KEYS (#23818)
need_read_data optimization does not work properly when table is UNQIUE_KEY MOR
2023-09-05 19:57:22 +08:00
2c60ae46c0 [enhancement](Nereids): remove useless if-condition (#23922) 2023-09-05 19:32:52 +08:00
771221a3f3 [improvement](fe-meta) check the image's meta version (#23847)
Sometimes, user may use a low version FE to read high version image, which may cause some undefined behavior
and hard to debug.
This PR throw an explicit error to notify the user
2023-09-05 19:08:29 +08:00
53265cc4f5 [Improvement](test)Add property to support manually use auto analyzer to analyze db (#23751)
This pr is mainly for test, make it easier to add regression test case for auto analyzer.
2023-09-05 17:08:17 +08:00
b5c66cddab [feat](optimizer) Auto upgrade column_statistics from 1.2 #23853
In Doris ver1.2 all the stats fields is not null, which might cause insert failure in the ver2.0. Add logic to upgrade table schema automatically
2023-09-05 15:57:23 +08:00
1a8913f8f4 [fix](shared hash table) fix p0 test failure (#23907) 2023-09-05 14:48:46 +08:00
801ddc0313 [feature-wip](arrow-flight) BE not start Arrow Flight Service by default (#23901) 2023-09-05 14:48:29 +08:00
f3680af8c5 [fix](nereids)fix CheckPolicy not match uniqueTable in nereids (#23714)
fix CheckPolicy not match uniqueTable in nereids
2023-09-05 14:41:48 +08:00
ee98cea8a1 [improve](segment-cache) Change the segment cache granularity from rowset_id to rowset_id+segment_id (#23561) 2023-09-05 14:32:07 +08:00
672a6ad3dd [typo](docs) Add some explanations about AUTO_INCREMENT columns (#23899) 2023-09-05 14:31:50 +08:00
7709fa5ea5 fix feut failed by mockit.internal.expectations.invocation.MissingInvocation: Missing 1 invocation to: (#23893)
mockit.internal.expectations.invocation.MissingInvocation: Missing 1 invocation to:
org.apache.doris.catalog.Env#isCheckpointThread()
mockit.internal.expectations.invocation.MissingInvocation: 
Missing 1 invocation to:
org.apache.doris.catalog.Env#isCheckpointThread()
Caused by: mockit.internal.expectations.invocation.ExpectationError
resolve http://43.132.222.7:8111/viewLog.html?buildId=213066&buildTypeId=Doris_Doris_FeUt

Issue Number: close #xxx
2023-09-05 14:30:52 +08:00
2bce8bbc66 [fix](spark load) not setting the file format cause null pointer exception (#16202) 2023-09-05 12:14:07 +08:00
e090b83e33 [improvemnt](script) support custom lib dir to save custom libs (#23887)
Sometimes, user need to add some custom libs to the cluster, such lzo.jar, orai18n.jar, etc.
In previous, these lib files are places in fe/lib or be/lib.
But when upgrading cluster, the lib dir will be replaced by new lib dir, so that all custom libs are lost.

In this PR, I add new dir custom_lib for FE and BE, and user can place custom lib files in it.
2023-09-05 11:54:19 +08:00
3f2f2a7218 [typo](docs) remove redundant column data from the insert statement (#23880) 2023-09-05 11:52:18 +08:00
d9461d77ba [Fix](bitmap index) like predicate does not work in bitmap index (#23819) 2023-09-05 11:43:16 +08:00
4dac2d3b94 [Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions (#23874) 2023-09-05 11:26:59 +08:00
e525e021ee [Enhancement](Load) stream tvf support csv header (#23797)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-05 11:15:45 +08:00
a91f9fbccb [fix][UT]fix coverage compute fail if run string cmd directly (#23878) 2023-09-05 10:57:54 +08:00
32dd088d30 [fix](inverted)fix inverted index report be hostname error #23872
when inverted index report task status to fe,not put the correct host
delete unused _backend
2023-09-05 10:52:23 +08:00
a36c387a2b [Refactor](transformer) convert to file format writer to transformer (#23888) 2023-09-05 10:50:10 +08:00
a02ee8e1d0 [test](regression) Reduce dependency of other database in test_alter_table_column (#22547) 2023-09-05 10:15:01 +08:00
5853ed385e [pipelineX](join) Support shared hash table (#23876) 2023-09-05 10:14:40 +08:00
b51d8aa7b7 [pipelineX](feature) support empty set operator (#23877) 2023-09-05 10:11:08 +08:00
1d1a9e2bfc [improvement](graceful shutdown) waiting for all query finished when graceful shutdown (#23865)
In some cloud native deployment scenario, BE(especially the Compute Node BE) will be add to cluster and remove from cluster very frequently. User's query will fail if there is a fragment is running on the shutting down BE. Users could use stop_be.sh --grace, then BE will wait all running queries to stop to avoiding running query failure, but if the waiting time exceed the limit, then be will exit directly. During this period, FE will not send any queries to BE and waiting for all running queries to stop
2023-09-05 09:52:28 +08:00
774a771e0c [Improve](be)check swap (#18891)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2023-09-05 09:39:55 +08:00
c9b4749276 [fix](RemoteUDF) fix string type do not set PGenericType #23832 2023-09-05 09:23:20 +08:00
eea9a43d97 [fix](load) add lock for serialize profile (#23856) 2023-09-05 09:11:55 +08:00
2885de1d63 [chore](macOS) Fix invalid option errors in start_be.sh (#23861) 2023-09-05 09:07:53 +08:00
a1db5ef562 [bugfix](regression-test) Repeated create user during parallel testing #23869 2023-09-05 09:06:34 +08:00
babd0430c7 [fix](stacktrace) Fix StackTraceCache initialized before ExecEnv (#23828) 2023-09-05 09:06:16 +08:00
e639454753 [fix](regressionTest) fix test_external_catalog_es p2 regression test #23838 2023-09-05 09:02:16 +08:00
9d758a6c1a [fix](case) disable unstable case in dir nereids_tpcds_shape_sf100_p0 (#23884) 2023-09-04 22:51:21 +08:00
2a3fc92d13 [fix](auth)fix after setting the user password to expire, changing the password again will not take effect (#23426)
Create a jack user and set the password to expire after 10 days.
`CREATE USER 'jack' IDENTIFIED BY '12345' PASSWORD_EXPIRE INTERVAL 10 DAY FAILED_LOGIN_ATTEMPTS 3 PASSWORD_LOCK_TIME 1 DAY;`

After the password has expired, reset the password.
`SET PASSWORD FOR 'jack' = PASSWORD('123');`


Log in with a new password, ERROE ` Your password has expired. To log in you must change it using a client that supports expired passwords`
2023-09-04 21:19:54 +08:00
c1620f9e1a [chore](case) Update regression-conf.groovy (#23873) 2023-09-04 20:54:48 +08:00
7727535b91 [Enhancement](build) try to download commit specific source code when git submodule failed (#23846) 2023-09-04 20:12:46 +08:00
039c76cbc0 [feature-wip] (arrow-flight) (step1) BE support Arrow Flight server, read data only (#23765) 2023-09-04 19:19:55 +08:00
21aea76333 [pipelineX](feature) support assert rows num operator (#23857) 2023-09-04 18:17:26 +08:00
d694f4a946 [doc](bitmap) add docs from bitmap_to_base64 and bitmap_from_base64 (#23805) 2023-09-04 17:57:24 +08:00
Pxl
251e79d9f8 [Bug](compare) fix core dump on decimal128 when be is build by debug mode (#23840)
fix core dump on decimal128 when be is build by debug mode
2023-09-04 17:40:57 +08:00
db306a51b3 [fix](nereids) missing return value of resetLogicalProperties() (#23850)
physicalPlan.resetLogicalProperties(); will not change the origin plan but create a new plan with no logical property. So should update the plan using resetLogicalProperties()'s return value.
2023-09-04 17:37:26 +08:00
72b709d6a9 [opt](stats) split period collector from auto collector (#23622)
1. Split period analyze from auto collector
2. Analyze table incrementally by default
3. Rename StatisticsAutoAnalyzer to StatisticsAutoCollector
2023-09-04 17:04:16 +08:00
1c73fd2687 [pipelineX](ssb) add SSB test cases (#23844) 2023-09-04 16:38:37 +08:00
f37078db99 [fix](testcase) add analyze table in load of tpcds_sf1_index testcase (#23811) 2023-09-04 16:19:48 +08:00
9bc7010639 fix topn be inoperative because Field == Null always return true (#23830)
```if (!new_top.is_null() && new_top != old_top)``` is always false since old_top is Null when init and Field == Null always return true.

We add old_top.is_null() check first to avoid the problem and then issue more carefull discussion about Field == Null semantics.
2023-09-04 16:02:07 +08:00
5dc46efcdf [feature](nereids) eliminate cascading outer join (#23754)
after eliminate outer join, create is-not-null predicate, and then this is-not-null predicate can be used to eliminate descendant outer join. the newly created is-not-null predicate will be eliminated in EliminateNotNull rule.
2023-09-04 16:00:18 +08:00
422159bd94 [pipelineX](refactor) add repeat node in pipelineX (#23750) 2023-09-04 15:55:09 +08:00
301a1d97e1 [fix](row-policy) fix creating row policy with forward issue (#23801)
The `CreateRowPolicyCommand` is implemented with overriding `run()` method.

So when executing `create row policy` in non-master FE, and forward it to Master FE,
it will call `execute(TUniqueId queryId)` method and go through `executeByNereids()`.
And because without `run()` method, it will do nothing and return OK.
So after `show row policy`, user will get empty result.

This PR fix it by implmenting the `run()` method but throw an Exception, so that it will
fallback to old planner, to do the creating row policy command normally.

The full implement of `run()` method should be implemented later.
This is just a tmp fix.
2023-09-04 15:19:37 +08:00