During the renaming of `JdbcCatalog`, I noticed that the `jdbcClient` was being closed,
resulting in exceptions during subsequent queries. This happens because the `removeCatalog`
method is invoked when changing the name, which in turn calls the `onClose` method of the catalog.
Ideally, the client should not be closed when renaming the catalog.
However, to avoid extra checks in the `removeCatalog` method, we can simply execute `onRefresh`
in the `addCatalog` method to address this issue.
Following #25138, unified schema change interface for parquet and orc reader, and can be applied to other format readers as well.
Unified schema change interface for all format readers:
- First, read the data according to the column type of the file into source column;
- Second, convert source column to the destination column with type planned by FE.
The meta of the restore may come from different clusters, so the
original ID in the meta may conflict with the ID of the new cluster. For
example, if a newly allocated ID happens to be the same as an original ID,
the original one may be overwritten when executing `put`.
Issue Number: #31442
- Add hive-writer runtime profiles.
- Change output file names to `${query_id}${uuid}-${index}.${compression}.${format}`. e.g. `"d8735c6fa444a6d-acd392981e510c2b_34fbdcbb-b2e1-4f2c-b68c-a384238954a9-0.snappy.parquet"`. For the same partition writer, when the file size exceeds `hive_sink_max_file_size`, the currently written file will be closed and a new file will be generated, in which ${index} in the new file name will be incremented, while the rest will be the same .
Issue #31442#32726
1. add LocalDfsFileSystem to manipulate local files.
2. add HMSCachedClientTest to analog HMS services.
3. add test for rollback commit.
Issue #31442
Add `TransactionManager` and `Transaction`.
```
public interface Transaction {
void commit() throws UserException;
void rollback();
}
public interface TransactionManager {
long begin();
void commit(long id) throws UserException;
void rollback(long id);
Transaction getTransaction(long id);
}
```
`TransactionManager` is used to manage all external transactions:
The application layer should manage the entire transaction through this `TransactionManager`, like:
```
transactionManager.commit();
transactionManager.rollback();
```
`Transaction` is an interface. You can implement this interface according to the specific content, such as `HMSTransaction` currently implemented, iceberg that may be implemented in the future, etc.
Issue Number: #31442
dependent on #32824
add ddl(create and drop) test
add ctas test
add complex type test
TODO:
bucketed table test
truncate test
add/drop partition test
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
* [Fix](nereids) fix qualifier problem that affects delete stmt in another catalog
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
fix:
F20240411 10:26:06.693233 1368925 thread_context.h:293] __builtin_unreachable, If you crash here, it means that SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. starting position of each thread is expected to use SCOPED_ATTACH_TASK to bind a MemTrackerLimiter belonging to Query/Load/Compaction/Other Tasks, otherwise memory alloc using Doris Allocator in the thread will crash. If you want to switch MemTrackerLimiter during thread execution, please use SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER, do not repeat Attach. Of course, you can modify enable_memory_orphan_check=false in be.conf to avoid this crash.
*** Check failure stack trace: ***
@ 0x562d9b5aa6a6 google::LogMessage::SendToLog()
@ 0x562d9b5a70f0 google::LogMessage::Flush()
@ 0x562d9b5aaee9 google::LogMessageFatal::~LogMessageFatal()
@ 0x562d7ebd1b7e doris::thread_context()
@ 0x562d7ec203b8 Allocator<>::sys_memory_check()
@ 0x562d7ec255a3 Allocator<>::memory_check()
@ 0x562d7ec274a1 Allocator<>::alloc_impl()
@ 0x562d7ec27227 Allocator<>::alloc()
@ 0x562d67a12207 doris::vectorized::PODArrayBase<>::alloc<>()
@ 0x562d67a11fde doris::vectorized::PODArrayBase<>::realloc<>()
@ 0x562d67a11e26 doris::vectorized::PODArrayBase<>::reserve<>()
@ 0x562d77331ee3 doris::vectorized::ColumnVector<>::reserve()
@ 0x562d7e64328e doris::vectorized::ColumnNullable::reserve()
@ 0x562d7ec79a84 doris::vectorized::Block::Block()
@ 0x562d6b86b81b doris::PInternalServiceImpl::_multi_get()
@ 0x562d6b8a4a07 doris::PInternalServiceImpl::multiget_data()::$_0::operator()()
