in hash join condition, some equals are trustable, some are not.
an equal is trustable if one side is almost unique, like primary key. for such equal condition we could estimate more accurate.
the problem is in rewriten q20, the are 2 equal condition, one is trustable, another is not. But we treat both of them as trustable.
Test result:
on tpch100, from 2.2 sec to 0.44 sec
no impact on tpch other queries
no performance impact on tpcds queries
1. Don't write to table_statistics when create sync analyze job anymore since it's meaningless
2. Capture exceptions when creating each system analyze job to avoid the failure of creation of all automatic collection jobs due to a single job creation failure.
3. Mark auto triggered period job's job type as system
Sometimes we meet a hive table with parameter: "transactional" = "true", but format is parquet, which is not supported.
So we need to check the input format for transactional table.
When converting query results into MySQL format, it involves transforming columnar data storage into row-based storage. This process raises the question of choosing between sequential reading and sequential writing. In reality, sequential writing is the better choice for performance optimization.
Test with 9M rows contains more than 20 columns, this patch can reduce the conversion time from 20s to 11s.
Fix bug of left and full outer join with other conjuncts. When equal matched row count of a probe row exceed batch_size, some times the _join_node->_is_any_probe_match_row_output flag is not set correcty, which result in outputing extra rows for the probe row.
1. get inverted index size before segment writer's column writer clear, then add size to total data size and total index size
2. also do this in vertical compaction
The metadata indexes in MaterializedIndexMeta is introduced in 2.0-beta version and it's relied by writing inverted index. When upgrade doris from old version to 2.0-beta, the metadata indexes in MaterializedIndexMeta is empty and no inverted index file will be written.
This PR add compatibility logic to use metadata indexes in table if indexes in MaterializedIndexMeta is empty and indexes in MaterializedIndexMeta indexId is equal to table baseIndexId.
This PR also fix metadata indexes missing for schema change.
stability and some fixes
1. fix s3 availability check
2. add independent minio properties
3. add job conf cache
4. remove extra s3 propertie when convert catalog properties
5. add some ut case to check conveted properties
fix 3 issues about mv selection:
1. in legacy planner, should not consider aux expr when do mv selection
2. in Nereids, should not hit mv when agg function on value column is distinct
3. select mv cannot rewrite agg function in on table with column name in uppercase
in order to solve agg of sum/count is not compatibility during the upgrade process.
in PR [refactor](agg_state) refactor agg_state type to support fixed length object type #20370 have changed the serialize type and serialize column of sum/count
before is ColumnVector, now sum/count change to use ColumnFixedLengthObject
so during the upgrade process, will be not compatible if exist Old BE and Newer BE
In JdbcMysqlClient, I've added methods to retrieve auto-increment and default value columns from MySQL. These columns are then mapped into Doris metadata to make them visible to users.
When handling the InsertStmt into an execution plan, Doris used to automatically fill in NULL or default values for columns not specified in the InsertStmt. However, in the JDBC catalog, we don't need Doris to handle these unspecified columns, so I've made changes to skip them directly.
For the insert prepared statement required for writing, our previous behavior was to obtain all columns for placeholders. So, the change I made is to pass in the columns processed by the execution plan during the sink task generation stage for dynamic generation.
we add penalty for broadcast join (bc for brief in the following).
the intuition of penalty is as follow:
1. if the build side is very small (< 1M), we prefer bc, and set `penalty=1`, which means no penalty
2. if build side is more than 1M, we consider the ratio of the probe row count to the build row count. the less the ratio is, the higher penalty is.
this pr has positive impact on tpch queries. Only q3 is changed. in out test (tpch 1T, 3BE) q3 improved from 5.1sec to 2.5 sec.
this pr has positive impact on tpcds queries. test on tpcds sf100 (3BE), cold run improve from 163 sec to 156 sec, hot run improves from 155 sec to 149 sec
