#6206
At present, our image file does not have file header/footer. When we need to change the image format (such as adding different journal versions to the image), there is no way to distinguish different image formats.
Therefore, we suggest adding file header and footer to the image. By the new image format, we can freely distinguish and define different image reading ways.
The format of the image is as follows:
```
/**
* Image Format:
* |- Image --------------------------------------|
* | - Magic String (4 bytes) |
* | - Header Length (4 bytes) |
* | |- Header -----------------------------| |
* | | |- Json Header ---------------| | |
* | | | - version | | |
* | | | - other key/value(undecided)| | |
* | | |-----------------------------| | |
* | |--------------------------------------| |
* | |
* | |- Image Body -------------------------| |
* | | Object a | |
* | | Object b | |
* | | ... | |
* | |--------------------------------------| |
* | |
* | |- Footer -----------------------------| |
* | | | - Checksum (8 bytes) | |
* | | |- object index --------------| | |
* | | | - index a | | |
* | | | - index b | | |
* | | | ... | | |
* | | |-----------------------------| | |
* | | - other value(undecided) | |
* | |--------------------------------------| |
* | - Footer Length (8 bytes) |
* | - Magic String (4 bytes) |
* |----------------------------------------------|
*/
```
1. Magic Number
One image format is identified by one magic string and one version field. The magic string is save in the first 4 bytes and last 4 bytes in the images.
2. Image Header:
The version is save in the header with json format now.
3. Image Body:
Equal to the original image.
4.Image Footer:
Image footer stores the file offset(index) of many image objects. If necessary, we can read some objects in the image by the footer.
Provides basic property related classes supports create, query, read, write, etc.
Currently, Doris FE mostly uses `if` statement to check properties in SQL. There is a lot of redundancy in the code.
The `PropertySet` class can be used in the analysis phase of `Statement`. The validation and correctness of the input properties are automatic verified. It can simplify the code and improve the readability of the code.
Usage:
1. Create a custom class that implements `SchemaGroup` interface.
2. Define the properties to be used. If it's a required parameter, there is no need to set the default value.
3. According the the requirements, in the logic called `readFromStrMap` and other functions to check and obtain parameters.
Demo:
Class definition
```
public class FileFormat implements PropertySchema.SchemaGroup {
public static final PropertySchema<FileFormat.Type> FILE_FORMAT_TYPE =
new PropertySchema.EnumProperty<>("type", FileFormat.Type.class).setDefauleValue(FileFormat.Type.CSV);
public static final PropertySchema<String> RECORD_DELIMITER =
new PropertySchema.StringProperty("record_delimiter").setDefauleValue("\n");
public static final PropertySchema<String> FIELD_DELIMITER =
new PropertySchema.StringProperty("field_delimiter").setDefauleValue("|");
public static final PropertySchema<Integer> SKIP_HEADER =
new PropertySchema.IntProperty("skip_header", true).setMin(0).setDefauleValue(0);
private static final FileFormat INSTANCE = new FileFormat();
private ImmutableMap<String, PropertySchema> schemas = PropertySchema.createSchemas(
FILE_FORMAT_TYPE,
RECORD_DELIMITER,
FIELD_DELIMITER,
SKIP_HEADER);
public ImmutableMap<String, PropertySchema> getSchemas() {
return schemas;
}
public static FileFormat get() {
return INSTANCE;
}
}
```
Usage
```
public class CreateXXXStmt extends DdlStmt {
private PropertiesSet<FileFormat> analyzedFileFormat = PropertiesSet.empty(FileFormat.get());
private final Map<String, String> fileFormatOptions;
...
public void analyze(Analyzer analyzer) throws UserException {
...
if (fileFormatOptions != null) {
try {
analyzedFileFormat = PropertiesSet.readFromStrMap(FileFormat.get(), fileFormatOptions);
} catch (IllegalArgumentException e) {
...
}
}
// 1. Get property value
String recordDelimiter = analyzedFileFormat.get(FileFormat.RECORD_DELIMITER)
// 2. Check the validity of parameters
PropertiesSet.verifyKey(FileFormat.get(), fileFormatOptions);
...
}
}
```
Currently, FeMetaVersion.java is in fe-common, users may forget to copy fe-common.jar when upgrading the service.
It's really dangerous because the data may be corrupted and can not be recovered.
The io related codes may be used by new modules, so It's better to move them to fe-common.
The modification to fe-core is frequent, but there are many generated java files by thrift
will slow down the compilation, so It's better to move thrift generation process to fe-common.
Currently both log4j1 and log4j2 are used, which leads to logs are written to wrong files.
Our modification will remove log4j1 from dependency, use slf4j + slf4j -> log4j2 instead.
This CL mainly changes:
1. Add 2 new FE modules
1. fe-common
save all common classes for other modules, currently only `jmockit`
2. spark-dpp
The Spark DPP application for Spark Load. And I removed all dpp related classes to this module, including unit tests.
2. Change the `build.sh`
Add a new param `--spark-dpp` to compile the `spark-dpp` alone. And `--fe` will compile all FE modules.
the output of `spark-dpp` module is `spark-dpp-1.0.0-jar-with-dependencies.jar`, and it will be installed to `output/fe/spark-dpp/`.
3. Modify some bugs of spark load
