Issue Number: #31442
- Add hive-writer runtime profiles.
- Change output file names to `${query_id}${uuid}-${index}.${compression}.${format}`. e.g. `"d8735c6fa444a6d-acd392981e510c2b_34fbdcbb-b2e1-4f2c-b68c-a384238954a9-0.snappy.parquet"`. For the same partition writer, when the file size exceeds `hive_sink_max_file_size`, the currently written file will be closed and a new file will be generated, in which ${index} in the new file name will be incremented, while the rest will be the same .
Issue #31442#32726
1. add LocalDfsFileSystem to manipulate local files.
2. add HMSCachedClientTest to analog HMS services.
3. add test for rollback commit.
Issue #31442
Add `TransactionManager` and `Transaction`.
```
public interface Transaction {
void commit() throws UserException;
void rollback();
}
public interface TransactionManager {
long begin();
void commit(long id) throws UserException;
void rollback(long id);
Transaction getTransaction(long id);
}
```
`TransactionManager` is used to manage all external transactions:
The application layer should manage the entire transaction through this `TransactionManager`, like:
```
transactionManager.commit();
transactionManager.rollback();
```
`Transaction` is an interface. You can implement this interface according to the specific content, such as `HMSTransaction` currently implemented, iceberg that may be implemented in the future, etc.
Issue Number: #31442
dependent on #32824
add ddl(create and drop) test
add ctas test
add complex type test
TODO:
bucketed table test
truncate test
add/drop partition test
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
fix:
F20240411 10:26:06.693233 1368925 thread_context.h:293] __builtin_unreachable, If you crash here, it means that SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. starting position of each thread is expected to use SCOPED_ATTACH_TASK to bind a MemTrackerLimiter belonging to Query/Load/Compaction/Other Tasks, otherwise memory alloc using Doris Allocator in the thread will crash. If you want to switch MemTrackerLimiter during thread execution, please use SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER, do not repeat Attach. Of course, you can modify enable_memory_orphan_check=false in be.conf to avoid this crash.
*** Check failure stack trace: ***
@ 0x562d9b5aa6a6 google::LogMessage::SendToLog()
@ 0x562d9b5a70f0 google::LogMessage::Flush()
@ 0x562d9b5aaee9 google::LogMessageFatal::~LogMessageFatal()
@ 0x562d7ebd1b7e doris::thread_context()
@ 0x562d7ec203b8 Allocator<>::sys_memory_check()
@ 0x562d7ec255a3 Allocator<>::memory_check()
@ 0x562d7ec274a1 Allocator<>::alloc_impl()
@ 0x562d7ec27227 Allocator<>::alloc()
@ 0x562d67a12207 doris::vectorized::PODArrayBase<>::alloc<>()
@ 0x562d67a11fde doris::vectorized::PODArrayBase<>::realloc<>()
@ 0x562d67a11e26 doris::vectorized::PODArrayBase<>::reserve<>()
@ 0x562d77331ee3 doris::vectorized::ColumnVector<>::reserve()
@ 0x562d7e64328e doris::vectorized::ColumnNullable::reserve()
@ 0x562d7ec79a84 doris::vectorized::Block::Block()
@ 0x562d6b86b81b doris::PInternalServiceImpl::_multi_get()
@ 0x562d6b8a4a07 doris::PInternalServiceImpl::multiget_data()::$_0::operator()()
Problem: An error is encountered when executing a schema change on a unique table to add a column with a complex type, such as bitmap, as documented in https://github.com/apache/doris/issues/31365
Reason: The error arises because the schema change logic erroneously applies an aggregation check for new columns across all table types, demanding an explicit aggregation type declaration. However, unique and duplicate tables inherently assume default aggregation types for newly added columns, leading to this misstep.
Solution: The schema change process for introducing new columns needs to distinguish between table types accurately. For unique and duplicate tables, it should automatically assign the appropriate aggregation type, which, for the purpose of smooth integration with subsequent processes, should be set to NONE.
