It is a painful work to read profile, especially there are multi-parallel instances.
This tool helps us to grasp the main information of profile in a graphical view.
The profile is represented by a tree.
Sql operation nodes contains operation type(join, scan...), its node id, its fragment id. The number on the arrow edge means how many rows output by child node. This tool will sum the output rows of the same node in multi-parallel instances, that is if there are 4 parallel instance, and each ScanNode on lineitem table output 10 rows, the label on the arrow beginning with ScanNode(lineitem) is 40.
Here is a demo for tpch Q2
tpch q2 profile viewer
Issue Number: close #xxx
Fix Thread pool token was shut down error.
This is because when there are more than 1 fragment of a query on one BE, the thread token maybe
reset incorrectly, causing thread token shutdown earlier.
cherry-pick from master
Introduced from #13021
* [enhancemeng](sql) Remove unused wide common factors to improve scan performance in ExtractCommonFactorsRule
* fix regression test
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
Originally, Order By Limit returned a maximum of 65535 rows of data by default during the query,
but now many businesses do not apply this limit.
It is necessary to add larger data after the query statement to complete the full data query,
which is extremely inconvenient, so adjustments have been made.
At the same time, I added the variable DEFAULT_ORDER_BY_LIMIT to the SessionVariable,
the default value is -1, if the user does not use the LIMIT keyword or the LIMIT value is a negative integer,
the default query return value is Long.MAX_VALUE. If the corresponding maximum query value is set,
the number of data items is returned according to the maximum query value or the value followed by the
LIMIT keyword.
For the table name check, the regular match error and the length exceeds the limit, both of which display the message "Incorrect table name 'xxx'. Table name regex is 'xxx'".
Obviously, the message cannot clearly point out what kind of error it is.
So it is a better way to separate the two error messages.
# Proposed changes
- Refactor AggregateFunction
1. AggregateFunction implement ComputeSignature
3. Add a CustomSignature to dynamic compute signature, we can check input type and compute implicit cast type in the `customSignature` method
2. Add PartialAggType to record some type information before disassemble aggregate
4. Refine and create a custom catalog function when translate AggregateFunction, without `finalizeForNereids`
- Support explain plan
1. explain parsed plan select ...
5. explain analyzed plan select ...
6. explain rewritten/logical plan select ...
7. explain optimized/physical plan select ...
8. explain all plan select ...
before:
MySQL [test]> select cast('135.759999999' as DECIMAL(10,3));
+----------------------------------------+
| CAST('135.759999999' AS DECIMAL(10,3)) |
+----------------------------------------+
| 135.759999999 |
+----------------------------------------+
1 row in set (0.00 sec)
now:
MySQL [stage]> select cast('135.759999999' as DECIMAL(10,3));
+----------------------------------------+
| CAST('135.759999999' AS DECIMAL(10,3)) |
+----------------------------------------+
| 135.759 |
+----------------------------------------+
1 row in set (0.01 sec)
When executing create table as select stmt,
the varchar/char/string type of column in created table will be unified to string type.
Because when select from external table (mysql/pg, etc), the length of varchar in external database
is calculated by "char" length, not "byte" length.
So if there is a column with varchar(10) in external table, then there will be a same varchar(10)
in created table. But the byte length of data in external table may be larger than 10, causing failure of CTAS.
Change to string will not impact performance of the capacity of disk storage.
And notice that if a string type column is the first column, it will be changed to varchar(65535),
because we do not allow string type column as sort key column.
#13195 left some unresolved issues. One of them is that some BE unit tests fail.
This PR fixes this issue. Now, we can run the command ./run-be-ut.sh --run successfully on macOS.
When you create the Uniq table, you can specify the mapping of sequence column to other columns.
You no longer need to specify mapping column when importing.
Invalidate catalog/db/table cache when doing
refresh catalog/db/table.
Tested table with 10000 partitions. The refresh operation will cost about 10-20 ms.
Collect HMS external table statistic information through external metadata.
Insert the result into __internal_schema.column_statistics using insert into SQL.
Add regression test for external hive orc table. This PR has generated all basic types support by hive orc, and create a hive external table to touch them in docker environment.
Functions to be tested:
1. Ensure that all types are parsed correctly
2. Ensure that the null map of all types are parsed correctly
3. Ensure that the `SearchArgument` of `OrcReader` works well
4. Only select partition columns
If a table already been analyzed, then we analyze it again, the new statistics would larger than expected since the incremental would contain the values from table level statistics since the SQL lack the predication for the nullability of part_id
