Commit Graph

13721 Commits

Author SHA1 Message Date
32a7eef96a [schedule](pipeline) Remove wait schedule time in pipeline query engine (#23994)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-10 17:06:51 +08:00
648bf77c72 [Fix](MemtableMemoryLimiter) fix memtable memory limiter tigger flush log (#24137) 2023-09-10 16:33:35 +08:00
14f8f0cae0 [Improvement](errorcode) use error code when disk exceed capacity limit (#24136) 2023-09-10 16:32:17 +08:00
cae5a9d3cd [Fix](auth) fix revoke role operation cause fe down (#23852)
If there 3 above fe nodes,
the following opeartions will cause all FE nodes down.

DROP USER revoke_test_user
DROP ROLE revoke_test_role
DROP DATABASE IF EXISTS revoke_test_db
CREATE DATABASE revoke_test_db
CREATE ROLE revoke_test_role
CREATE USER revoke_test_user IDENTIFIED BY 'revoke_test_pwd'
GRANT SELECT_PRIV ON revoke_test_db.* TO ROLE 'revoke_test_role'
GRANT 'revoke_test_role' TO revoke_test_user
SHOW GRANTS FOR revoke_test_user
REVOKE 'revoke_test_role' from revoke_test_user
SHOW GRANTS FOR revoke_test_user
DROP USER revoke_test_user
DROP ROLE revoke_test_role
DROP DATABASE revoke_test_db
2023-09-10 16:16:07 +08:00
71645a391c [debug](FileCache) fail over to remote file reader if local cache failed (#24097)
Fail over to remote file reader even if local file cache failed. This operation can increase the robustness of file cache.
2023-09-10 12:26:17 +08:00
69f599bb53 [regression-test](fix)add test_ifnull. (#23956) 2023-09-10 12:11:43 +08:00
262c669918 [fix](jdbc catalog) fix jdbc catalog creating json columns when reading json data (#24122) 2023-09-10 12:00:53 +08:00
953958c486 [fix](create tablet) fix backend create tablet timeout (#23879) 2023-09-10 11:41:00 +08:00
93c1151f1a [fix](join) incorrect result of mark join (#24112) 2023-09-10 11:30:45 +08:00
232b58a27d [fix](broker-load) make sequence column name case insensitive (#24071) 2023-09-10 10:51:07 +08:00
a1090f20c7 [supplement](regression-test) pass ccr case when fe enable_feature_binlog=false (#24077) 2023-09-10 10:49:42 +08:00
5f2ca8c84c [log](load) print more message about load job on tablet error (#24096) 2023-09-10 10:30:43 +08:00
650af8f4df [fix](test) fix broker load with default value test case (#24123) 2023-09-10 10:28:22 +08:00
f9a75b5c4f [feature](csv_serde)1.append csv serde for serialize to csv and deserialize from csv. 2.let csvReader use csv serde not text_converter. (#23352)
1. append csv serde for serialize to csv and deserialize from csv.
2. let csvReader use csv serde not text_converter.
2023-09-10 00:16:21 +08:00
5eb9e10b51 [pipelineX](pick) pick 2 PRs to fix bugs (#24117) 2023-09-09 23:10:46 +08:00
7c7e44fcc8 [refactor](nereids) make forbid_unknown_col_stats check more accurate (#24061)
ignore unknown col stats check if:
colunm not used in query
column is Array/Json/Map/Struct type
2023-09-09 22:42:17 +08:00
e09e030652 [fix](Nereids) mv in select materialized_view should disable show table (#24104)
mv in select materialized_view should disable show table,
because Nereids planner can output the string such as
slot#[0] in toSql() of SlotRef. Note this is only a
temporary solution, will use an expression translator later
2023-09-09 21:57:52 +08:00
21e30d4374 [fix](planner)ctas's query part is not analyzed correctly (#24111)
* [fix](planner)ctas's query part is not analyzed correctly
2023-09-09 20:55:09 +08:00
8c2a721873 [opt](nereids)push down filter through window #23935
select rank() over (partition by A, B) as r, sum(x) over(A, C) as s from T;
A is a common partition key for all windowExpressions, that is A is intersection of {A,B} and {A, C}
we could push filter A=1 through this window, since A is a common Partition key:
select * from (select a, row_number() over (partition by a) from win) T where a=1;
origin plan:

----filter((T.a = 1))
----------PhysicalWindow
------------PhysicalQuickSort
--------------PhysicalProject
------------------PhysicalOlapScan[win]
transformed to

----PhysicalWindow
------PhysicalQuickSort
--------PhysicalProject
----------filter((T.a = 1))
------------PhysicalOlapScan[win]
But C=1 can not be pushed through window.
2023-09-09 20:53:31 +08:00
7b62013d21 [refactor](nereids) print "ifnull" instead of "nvl" in explain #23979
'ifnull' is used more general.
2023-09-09 20:33:23 +08:00
a8ed1d87d7 [enhancement](config): Change root log level to info in broker log (#24023) 2023-09-09 17:56:50 +08:00
6b9698a248 [bugfix](insert into) should not send profile during report process (#24127)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-09-09 17:12:35 +08:00
c3f3195721 [Fix](clucene) fix clucene build error in arm (#24130) 2023-09-09 15:31:40 +08:00
03757d0672 [bug](explode) fix table node not implement alloc_resource function (#24031)
fix table node not implement alloc_resource function
2023-09-09 08:25:28 +08:00
698fe55662 remove unused configs in be and broker (#24021) 2023-09-09 08:24:50 +08:00
153c7982f3 [Optimize](invert index) Optimize multiple terms conjunction query (#23871) 2023-09-09 01:52:58 +08:00
0f408d1192 [improvement](executor)Add name for task scheduler #23983 2023-09-09 00:56:39 +08:00
b5e1e36750 [fix](pipeline)add logs for unstable cases #24073
Issue Number: close #xxx

ShowTableStmtTest.testNoDb and DropDbStmtTest.testNoPriv are unstable cases,error msg is:

java.lang.Exception: Unexpected exception, expected<org.apache.doris.common.AnalysisException> but was<mockit.internal.expectations.invocation.MissingInvocation>
we can not know what is missing ,and this issue cannot be reproduced locally,so add some log
2023-09-09 00:49:40 +08:00
7abd23cad1 [fix](tablet clone) fix be load rebalancer choose candidate tablets #23915
When be load reblancer choose candidate tablets, it will try moving tablets from high load backends to low backend backends. If the higher HIGH BE has no available slot num, it should try next HIGH BE.
2023-09-09 00:48:27 +08:00
2fb4c818da [fix](tablet clone) delete tablet check other catchup #24038
Sometimes FE replica's version is unreliable. FE's replica may bigger than BE's real version. Need check if BE missing version (last failed version > 0).
2023-09-09 00:42:32 +08:00
aad3eb257f update gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b to 3.0.0 (#24056)
There are 1 security vulnerabilities found in gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b

CVE-2022-28948
What did I do?
Upgrade gopkg.in/yaml.v3 from v3.0.0-20210107192922-496545a6307b to 3.0.0 for vulnerability fix

What did you expect to happen?
Ideally, no insecure libs should be used.

How can we automate the detection of these types of issues?
By using the GitHub Actions configurations provided by murphysec, we can conduct automatic code security checks in our CI pipeline.

The specification of the pull request
PR Specification from OSCS
2023-09-09 00:37:39 +08:00
3e7f531d2b [fix](sec)upgrade org.yaml:snakeyaml to 2.0 #24057 2023-09-09 00:37:07 +08:00
0f0ffa3482 [Fix](Parquet Reader) fix parquet read issue (#24092) 2023-09-09 00:35:18 +08:00
0143ae8266 [fix]Add logging before _builtin_unreachable() (#24101)
Co-authored-by: 宋光璠 <songguangfan@sf.com>
2023-09-09 00:30:11 +08:00
8336bf0b06 [regression](pipelineX) disable runtime filter for pipelineX test cases (#24119) 2023-09-08 23:31:26 +08:00
69cc6fee97 [fix](explain) fix explain physical plan with external table issue (#23845)
The `SelectedPartitions` can not be null, or it will throw NPE and fallback to origin planner.
2023-09-08 21:11:48 +08:00
894aa48743 [fix](Nereids) remove PARTITIONS from non-reserved list (#24110)
according to PR #24053, it removed partitions from non-reserved list in
legacy planner's parser. For consistency, remove it from Nereids' parser
too.
2023-09-08 20:47:07 +08:00
929a9ad143 [Fix](RoutineLoad) Delete duplicate attribute in job property #24037 2023-09-08 20:42:28 +08:00
5c2f9eb92e [Improvement] (pipeline) Cancel related query if backend restarts or dead (#23863) 2023-09-08 20:30:52 +08:00
e140938d81 [Perfomance][export] Opt the export of CSV tranformer (#24003) 2023-09-08 20:26:54 +08:00
0b24bd6a42 [Bug](pipelineX) init runtime filter profile at first (#24106) 2023-09-08 20:01:02 +08:00
2638ad0550 [fix](compaction) rowid_conversion should ignore deleted row on normal compaction (#24005) 2023-09-08 19:44:24 +08:00
f8fd8a3d17 [fix](trash) fix clean trash not working (#23936)
When executing admin clean trash, if the backend daemon clean thread is cleaning trash, then SQL command will return immediately. But for the backend daemon thread, it doesn't clean all the trashes, it clean only the expired trashes.
Also if there's lots of trashes, the daemon clean thread will busy handling trashes for a long time.
2023-09-08 18:13:22 +08:00
76ca57cf21 [bug](join) fix outer join not add tuple is null column when build rows is 0 (#23974)
fix outer join not add tuple is null column when build rows is 0
2023-09-08 17:55:03 +08:00
Pxl
69868f18d6 [Bug](join) fix nested loop join some problems (#24034) 2023-09-08 17:40:41 +08:00
01ea024497 [fix](nereids) runtimefilter not generated after postprocessor (#23948)
fix bug: rf not generated
2023-09-08 17:37:04 +08:00
1abf5e779d [pipelineX](refactor) refactor debug string (#24083) 2023-09-08 16:58:53 +08:00
161520feb4 [feature](Nereids): enable convert CASE WHEN to IF (#24050)
enable rule to convert CASE WHEN to IF.
2023-09-08 16:58:33 +08:00
c0a41dc0f8 [fix](nereids) external scan use STORAGE_ANY instead of ANY as distibution (#24039) 2023-09-08 16:25:35 +08:00
d8bdd6c137 [fix](nereids) avoid throw analysis exception for unsupported type to make ext table goes nereids (#24089)
void throw analysis ex for unsupported type to make ext table goes nereids.
this will improve the nereids' availability for external table if unsupported type is in the basic table schema but not referenced in the real sql.

tested in external table env.

Consider the following case:
select pu.pk_ct_pu as id
  from fms_rd_nc65_zb.NC65P.CT_PU pu
  left join fms_rd_nc65_zb.NC65P.PUB_WF_INSTANCE pwi
    on pu.pk_ct_pu = pwi.billid 
    and pu.vtrantypecode=pwi.billtype
 left join fms_rd_nc65_zb.NC65P.SM_USER su
    on pu.creator = su.cuserid
 where pu.pk_ct_pu='1001A110000000K8XPVN'; 

PUB_WF_INSTANCE table has a BLOB type column and currently it will throw analysis exception and fallback to old optimizer, although this column is not referenced in the real sql. The old optimizer doesn't have the outer join -> inner join rule and the "pu.pk_ct_pu='1001A110000000K8XPVN'; " is not pushed down and the performance will drop down. After the pr, we add the unsupported type instead of throw exception directly, it will decide the unused case and can continue goes nereids and use all advanced optimization for the sql.
2023-09-08 15:56:29 +08:00