* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
* add ut for cooldown on be
This pr does three things:
1. Use Druid instead of HikariCP in JdbcClient
2. when download udf jar, add the name of the jar package after the local file name.
3. refactor some jdbcResource code
During concurrent import, the same row location may be marked delete multiple times by different versions of rowset.
Duplicate row location need to be removed.
be use EnginePublishVersionTask to publish all replica of all tablets of table of one transaction,
and EnginePublishVersionTask use TabletPublishTxnTask to truly publish tablet and make rowset visible.
but if TabletPublishTxnTask error, tablet id will add _error_tablet_ids but no return some errors,
and EnginePublishVersionTask will not report any error to fe, and fe make this transaction visible,
and partition's version add 1.
but if you query this table, will return error like "MySQL [test]> select * from test12;ERROR 1105 (HY000): errCode = 2, detailMessage = [INTERNAL_ERROR]failed to initialize storage reader. tablet=14023.730105214.d742d664692db946-386daa993d84d89d, res=[INTERNAL_ERROR][9.134.167.25]fail to find path in version_graph. spec_version: 0-3, backend=9.134.167.25".
after this pr, _error_tablet_ids will report to fe, this transaction will not be visible and add ErrMsg like "publish on tablet 14038 failed.".
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Fix three bugs:
1. `repeated_parent_def_level ` should be the definition of its repeated parent.
2. Failed to parse schema like `decimal(p, s)`
3. Fill wrong offsets for array type
remove duplicate type definition in function context
remove unused method in function context
not need stale state in vexpr context because vexpr is stateless and function context saves state and they are cloned.
remove useless slot_size in all tuple or slot descriptor.
remove doris_udf namespace, it is useless.
remove some unused macro definitions.
init v_conjuncts in vscanner, not need write the same code in every scanner.
using unique ptr to manage function context since it could only belong to a single expr context.
Issue Number: close #xxx
---------
Co-authored-by: yiguolei <yiguolei@gmail.com>
Modify the default value of mem_limit to auto. auto means process mem limit is equal to max(physical mem * 0.9, 6.4G).
6.4G is the maximum memory reserved for the system.
fix heap-use-after-free
The OrcReader has a internal FileInputStream, If the file is empty, the memory of FileInputStream will leak.
Besides, there is a Statistics instance in FileInputStream. FileInputStream maybe delete if the orc reader
is inited failed, but Statistics maybe used when orc reader is closed, causing heap-use-after-free error.
Potential memory leak
When init file scanner in file scan node, the file scanner prepare failed, the memory of file scanner will leak.
segcompaction is async and in parallel with load job. If the load job is
canncelling, memory structures will be destroyed and cause segcompaction
crash. This commit will wait segcompaction finished before destruction.
* [enhancement](stream load pipe) using queryid or load id to identify stream load pipe instead of fragment instance id
NewLoadStreamMgr already has pipe and other info. Do not need save the pipe into fragment state. and FragmentState should be more clear.
But this pr will change the behaviour of BE.
I will pick the pr to doris 1.2.3 and add the load id to FE support. The user could upgrade from 1.2.3 to 2.x
Co-authored-by: yiguolei <yiguolei@gmail.com>
Reason: column_name[k5], decimal value is not valid for definition, value=123.123, precision=9, scale=3, min=-999999.999, max=-999999.999; . src line [];
#17273
This CL mainly changes:
Support specifying csv schema manually in s3/hdfs table valued function
s3 (
'URI' = 'https://bucket1/inventory.dat',
'ACCESS_KEY'= 'ak',
'SECRET_KEY' = 'sk',
'FORMAT' = 'csv',
'column_separator' = '|',
'csv_schema' = 'k1:int;k2:int;k3:int;k4:decimal(38,10)',
'use_path_style'='true'
)
Add new session variable dry_run_query
If set to true, the real query result will not be returned, instead, it will only return the number of returned rows.
mysql> select * from bigtable;
+--------------+
| ReturnedRows |
+--------------+
| 10000000 |
+--------------+
This can avoid large result set transmission time and focus on real execution time of query engine.
For debug and analysis purpose.
background：
At the moment, match query must with inverted index,
problem description:
After drop inverted index which is the only index in table, there still can use match query for this index column.
fix it:
The index should be updated on BE regardless of whether the indexes_desc from FE is empty.
We set LIBHDFS3_CONF env in start_be.sh, so libhdfs3 will try to read this hdfs-site.xml,
if file does not exist, it will throw error. But Doris does not handle this error, cause BE crash.
This CL mainly changes:
Modify start_be.sh to only set LIBHDFS3_CONF if hdfs-site.xml exist.
Refactor the HDFSCommonBuilder so that it can return error correctly.
Add BE IP info in status, so that we can get ip from error msg like:
ERROR 1105 (HY000): errCode = 2, detailMessage = [INTERNAL_ERROR]failed to init reader for file 000.snappy.orc, err:
[INTERNAL_ERROR][172.21.0.101]failed to init HDFSCommonBuilder, please check check be/conf/hdfs-site.xml
The logic of prefer compute node is wrong, which causing the external table query can only assign up to 3 backends.
This CL refactor this logic and also change some FE config:
prefer_compute_node_for_external_table
If set to true, query on external table will prefer to assign to compute node.
And the max number of compute node is controlled by min_backend_num_for_external_table.
If set to false, query on external table will assign to any node.
min_backend_num_for_external_table
Only take effect when prefer_compute_node_for_external_table is true.
If the compute node number is less than this value, query on external table will try to get some mix node
to assign, to let the total number of node reach this value.
If the compute node number is larger than this value, query on external table will assign to compute node only.
Add a special counter for memtracker, faster, but relaxed ordering and not accurate in real time
Track thread create and destroy memory, which was previously removed due to performance loss and added back
