40bc3fed53
Provides basic property related classes supports create, query, read, write, etc.
Currently, Doris FE mostly uses `if` statement to check properties in SQL. There is a lot of redundancy in the code.
The `PropertySet` class can be used in the analysis phase of `Statement`. The validation and correctness of the input properties are automatic verified. It can simplify the code and improve the readability of the code.
Usage:
1. Create a custom class that implements `SchemaGroup` interface.
2. Define the properties to be used. If it's a required parameter, there is no need to set the default value.
3. According the the requirements, in the logic called `readFromStrMap` and other functions to check and obtain parameters.
Demo:
Class definition
```
public class FileFormat implements PropertySchema.SchemaGroup {
public static final PropertySchema<FileFormat.Type> FILE_FORMAT_TYPE =
new PropertySchema.EnumProperty<>("type", FileFormat.Type.class).setDefauleValue(FileFormat.Type.CSV);
public static final PropertySchema<String> RECORD_DELIMITER =
new PropertySchema.StringProperty("record_delimiter").setDefauleValue("\n");
public static final PropertySchema<String> FIELD_DELIMITER =
new PropertySchema.StringProperty("field_delimiter").setDefauleValue("|");
public static final PropertySchema<Integer> SKIP_HEADER =
new PropertySchema.IntProperty("skip_header", true).setMin(0).setDefauleValue(0);
private static final FileFormat INSTANCE = new FileFormat();
private ImmutableMap<String, PropertySchema> schemas = PropertySchema.createSchemas(
FILE_FORMAT_TYPE,
RECORD_DELIMITER,
FIELD_DELIMITER,
SKIP_HEADER);
public ImmutableMap<String, PropertySchema> getSchemas() {
return schemas;
}
public static FileFormat get() {
return INSTANCE;
}
}
```
Usage
```
public class CreateXXXStmt extends DdlStmt {
private PropertiesSet<FileFormat> analyzedFileFormat = PropertiesSet.empty(FileFormat.get());
private final Map<String, String> fileFormatOptions;
...
public void analyze(Analyzer analyzer) throws UserException {
...
if (fileFormatOptions != null) {
try {
analyzedFileFormat = PropertiesSet.readFromStrMap(FileFormat.get(), fileFormatOptions);
} catch (IllegalArgumentException e) {
...
}
}
// 1. Get property value
String recordDelimiter = analyzedFileFormat.get(FileFormat.RECORD_DELIMITER)
// 2. Check the validity of parameters
PropertiesSet.verifyKey(FileFormat.get(), fileFormatOptions);
...
}
}
```
Apache Doris (incubating)
Doris is an MPP-based interactive SQL data warehousing for reporting and analysis. Its original name was Palo, developed in Baidu. After donated to Apache Software Foundation, it was renamed Doris.
1. License
2. Technology
Doris mainly integrates the technology of Google Mesa and Apache Impala, and it is based on a column-oriented storage engine and can communicate by MySQL client.
3. User cases
Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis.
Doris not only provides batch data loading, but also provides near real-time mini-batch data loading.
Doris also provides high availability, reliability, fault tolerance, and scalability.
The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris (refer to Overview).
4. Compile and install
Currently only supports Docker environment and Linux OS, such as Ubuntu and CentOS.
4.1 Compile in Docker environment (Recommended)
We offer a docker image as a Doris compilation environment. You can compile Doris from source in it and run the output binaries in other Linux environments.
Firstly, you need to install and start docker service.
And then you could build Doris as following steps:
Step1: Pull the docker image with Doris building environment
$ docker pull apachedoris/doris-dev:build-env-1.3
You can check it by listing images, for example:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
apachedoris/doris-dev build-env-1.3 c9665fbee395 5 days ago 3.55GB
NOTE: You may have to use different images to compile from source.
image version commit id release version apachedoris/doris-dev:build-env before ff0dd0d 0.8.x, 0.9.x apachedoris/doris-dev:build-env-1.1 ff0dd0d or later 0.10.x or 0.11.x apache/incubator-doris:build-env-1.2 4ef5a8c 0.12.x - 0.14.0 apache/incubator-doris:build-env-1.3 ad67dd3 later version
Step2: Run the Docker image
You can run the image directly:
$ docker run -it apachedoris/doris-dev:build-env-1.3
Or if you want to compile the source located in your local host, you can map the local directory to the image by running:
$ docker run -it -v /your/local/path/incubator-doris-DORIS-x.x.x-release/:/root/incubator-doris-DORIS-x.x.x-release/ apachedoris/doris-dev:build-env-1.3
Step3: Download Doris source
Now you should be attached in docker environment.
You can download Doris source by release package or by git clone in image.
(If you already downloaded the source in your local host and map it to the image in Step2, you can skip this step.)
$ wget https://dist.apache.org/repos/dist/dev/incubator/doris/xxx.tar.gz
or
$ git clone https://github.com/apache/incubator-doris.git
Step4: Build Doris
Enter Doris source path and build Doris.
$ sh build.sh
After successfully building, it will install binary files in the directory output/.
4.2 For Linux OS
Prerequisites
You should install the following softwares:
GCC 10.2.1+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.19.2+, Flex 2.6.0+
Then set them to environment variable PATH and set JAVA_HOME.
If your GCC version is lower than 10.2.1, you can run:
sudo yum install -y devtoolset-10-gcc*
If devtoolset-10 is not found in current repo. Oracle has already rebuilt the devtoolset-10 packages. You can create
repo file CentOS-SCLo-scl.ol.repo in path /etc/yum.repos.d/:
[ol7_software_collections]
name=Software Collection packages for Oracle Linux 7 ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
and then
wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-*
sudo yum install -y devtoolset-10-gcc*
Don't forget to set the path of GCC (e.g /opt/rh/devtoolset-10/root/usr/bin) to the environment variable PATH.
Compile and install
Run the following script, it will compile thirdparty libraries and build whole Doris.
sh build.sh
After successfully building, it will install binary files in the directory output/.
5. License Notice
Some of the third-party dependencies' license are not compatible with Apache 2.0 License. So you may have to disable
some features of Doris to be complied with Apache 2.0 License. Details can be found in thirdparty/LICENSE.txt
6. Reporting Issues
If you find any bugs, please file a GitHub issue.
7. Contact Us
Mailing lists
| Name | Scope | |||
|---|---|---|---|---|
| dev@doris.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |
8. Links
- Doris official site - http://doris.incubator.apache.org
- User Manual (GitHub Wiki) - https://github.com/apache/incubator-doris/wiki
- Developer Mailing list - dev@doris.apache.org. Mail to dev-subscribe@doris.apache.org, follow the reply to subscribe the mail list.
- Gitter channel - https://gitter.im/apache-doris/Lobby - Online chat room with Doris developers.
- Overview - https://github.com/apache/incubator-doris/wiki/Doris-Overview
- Compile and install - https://github.com/apache/incubator-doris/wiki/Doris-Install
- Getting start - https://github.com/apache/incubator-doris/wiki/Getting-start
- Deploy and Upgrade - https://github.com/apache/incubator-doris/wiki/Doris-Deploy-%26-Upgrade
- User Manual - https://github.com/apache/incubator-doris/wiki/Doris-Create%2C-Load-and-Delete
- FAQs - https://github.com/apache/incubator-doris/wiki/Doris-FAQ