Too much AlterJobsV2 may consume too much memory, which may cause FullGC. Clear some data for finished or cancelled alterJobs and remove them when expired.
During the use of the `block`, some methods in the block manager will be referenced.
So `file_block_mgr` should be a resident and globally unique object.
I put it in `StorageEngine`.
TODO: the `BlockManager`, `Env` need to be reorganized.
Sometimes a replica is broken on BE, but FE does not notice that.
In this case, we have to manually delete that replica on BE.
If there are hundreds of replicas need to be handled, this is a disaster.
So I add a new stmt:
ADMIN SET REPLICA STATUS
which support setting tablet on specified BE as BAD or OK.
LLVM related codes have already be removed in master branch,
so there is no need to build llvm tool(which need a long
time to compile it).
Currently, some old release of Doris may still need it, so
for now, we just comment it, instead of remove it.
This CL solve the issue #3105
I add a new temporary table state WAITING_STABLE.
When an alter job is ready to start, it checks whether the table is stable. If it is not stable,
the table state is set to WAITING_STABLE. In this state, the tablet repair logic will continue to
repair the tablet until the table becomes stable.
After that, the table state will be reset to SCHEMA_CHANGE/ROLLUP and alter operations will begin.
This is just a temporary state, it does not need to be persistent, and only the master FE can see this state.
This CL try to fix a potential bug describe in ISSUE: #3097. But I'm not sure this is the root cause.
Also remove lots of verbose log, and fix a memory leak.
Backup job in BE only backup index which is visible, but the backup meta in FE contains the shadow index, after restore from this snapshot, the shadow index is visible to load process, and the tablets is not exist in BE, so load process would be cancelled. we could fix this bug by remove the useless shadow index at backup process.
2 Changes in this CL:
## Support multiple statements in one request like:
```
select 10; select 20; select 30;
```
ISSUE: #3049
For simple testing this CL, you can using mysql-client shell command tools:
```
mysql> delimiter //
mysql> select 1; select 2; //
+------+
| 1 |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
+------+
| 2 |
+------+
| 2 |
+------+
1 row in set (0.02 sec)
Query OK, 0 rows affected (0.02 sec)
```
I add a new class called `OriginStatement.java`, to save the origin statement in string format with an index. This class is mainly for the following cases:
1. User send a multi-statement to the non-master FE:
`DDL1; DDL2; DDL3`
2. Currently we cannot separate the original string of a single statement from multiple statements. So we have to forward the entire statement to the Master FE. So I add an index in the forward request. `DDL1`'s index is 0, `DDL2`'s index is 1,...
3. When the Master FE handle the forwarded request, it will parse the entire statement, got 3 DDL statements, and using the `index` to get the specified the statement.
## Optimized the display of syntax errors
I have also optimized the display of syntax errors so that longer syntax errors can be fully displayed.
In a large scale cluster, we may rolling upgrade BEs, this patch add a
column named 'Version' for command 'show backends;', as well as website
'/system?path=//backends', to provide a method to check whether there
is any BE missing upgraded.
```
be/src/olap/rowset/segment_v2/ordinal_page_index.cpp:103:22: warning: ‘ordinal’ may be used
uninitialized in this function [-Wmaybe-uninitialized]
_ordinals[i] = ordinal;
```
`DROP MATERIALIZE VIEW [ IF EXISTS ] <mv_name> ON [db_name].<table_name>`
Parameters:
IF EXISTS: Do not throw an error if the materialized view does not exist. A notice is issued in this case.
mv_name: The name of the materialized view to remove.
db_name: The name of db to which materialized view belongs.
table_name: The name of table to which materialized view belongs.
This bug occurred when BE make snapshot, the version required by fe had been merged into the cumulative version, so the snapshot task could not complete the task even if it retried. In order to solve this problem, the BackupJob could be set to CANCELLED, and the user could continue to retry the job.
Fix#3057
The index name in MaterializedViewMeta is still with `__doris_shadow` prefix
after schema change finished.
In this CL, I just remove the index name field in MaterializedViewMeta,
so that it would makes managing change of names less error-prone.
If delete predicate exists in meta in Doris-0.10, all of this predicates should
be remained. There is an confused place in Doris-0.10. The delete predicate
only exists in OLAPHeaderMessage and PPendingDelta, not in PDelta.
This trick results this bug.
The timestamp value load from orc file is error, the value has an offset with hive and spark.
Becuase the time zone of orc's timestamp is stored inside orc's stripe information, so the timestamp obtained here is an offset timestamp, so parse timestamp with UTC is actual datetime literal.
Firstly, add materialized index meta in olap table
The materialized index meta include index name, schema, schemahash, keystype etc.
The information itself scattered in each map is encapsulated into MaterializedIndexMeta.
Also the keys type of index meta maybe not same as keys type of base index after materialized view enabled.
Secondly, support the deduplicate mv.
If there is group by or aggregation function in create mv stmt, the keys type of mv is agg.
At the same time, the keys type of base table is duplicate.
For example
Duplicate table (k1, k2, v1)
MV (k1, k2) group by k1, k2
It should be aggregated during executing mv.
The default replication number of an olap table may not be set.
Every time we call `getReplicationNum()`, we have to check if it returns null,
which is inconvenience and may cause problem
So in this PR, I set a default value to table's replication number.
This bug is introduced by #2958
eg:
select str_to_date('2014-12-21 12%3A34%3A56', '%Y-%m-%d %H%%3A%i%%3A%s');
select unix_timestamp('2007-11-30 10:30%3A19', '%Y-%m-%d %H:%i%%3A%s');
This also enable us to extract column fields from HDFS file path with contains '%'.
Normalize the setting of mem limit to avoid some unexpected exception.
For example, use may not setting query mem limit in query plan, which
may cause BE crash.
