Hu Yanjun
@ -29,7 +29,7 @@ under the License.
|
||||
|
||||
Doris can easily expand and shrink FE, BE, Broker instances.
|
||||
|
||||
## FE Expansion and Compression
|
||||
## FE Scaling
|
||||
|
||||
High availability of FE can be achieved by expanding FE to three top-one nodes.
|
||||
|
||||
@ -94,7 +94,7 @@ Delete the corresponding FE node using the following command:
|
||||
>
|
||||
> 1. When deleting Follower FE, make sure that the remaining Follower (including Leader) nodes are odd.
|
||||
|
||||
## BE Expansion and Compression
|
||||
## BE Scaling
|
||||
|
||||
Users can login to Leader FE through mysql-client. By:
|
||||
|
||||
@ -142,7 +142,7 @@ DECOMMISSION clause:
|
||||
|
||||
**For expansion and scaling of BE nodes in multi-tenant deployment environments, please refer to the [Multi-tenant Design Document](../multi-tenant).**
|
||||
|
||||
## Broker Expansion and Shrinkage
|
||||
## Broker Scaling
|
||||
|
||||
There is no rigid requirement for the number of Broker instances. Usually one physical machine is deployed. Broker addition and deletion can be accomplished by following commands:
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
{
|
||||
"title": "Construct Docker Compose",
|
||||
"title": "Build Docker Compose",
|
||||
"language": "en"
|
||||
}
|
||||
---
|
||||
@ -24,7 +24,7 @@ specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Construct Docker Compose
|
||||
# Build Docker Compose
|
||||
|
||||
*Todo*
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
{
|
||||
"title": "Construct Docker Image",
|
||||
"title": "Build Docker Image",
|
||||
"language": "en"
|
||||
}
|
||||
---
|
||||
@ -24,17 +24,17 @@ specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Construct Docker Image
|
||||
# Build Docker Image
|
||||
|
||||
This document mainly introduces how to make a running image of Apache Doris through Dockerfile, so that an Apache Doris Image can be quickly pulled in a containerized orchestration tool or during a quick test to complete the cluster creation.
|
||||
This topic is about how to build a running image of Apache Doris through Dockerfile, so that an Apache Doris image can be quickly pulled in a container orchestration tool or during a quick test to complete the cluster creation.
|
||||
|
||||
## Software and hardware requirements
|
||||
## Software and Hardware Requirements
|
||||
|
||||
### Overview
|
||||
|
||||
Before making a Docker image, the production machine must be prepared in advance. The platform architecture of the machine determines the applicable platform architecture of the Docker Image after production. For example, the X86_64 machine needs to download the Doris binary program of X86_64. The image after production can only be used on the X86_64 platform run on. The ARM platform (M1 is regarded as ARM) is the same.
|
||||
Prepare the production machine before building a Docker image. The platform architecture of the Docker image will be the same as that of the machine. For example, if you use an X86_64 machine to build a Docker image, you need to download the Doris binary program of X86_64, and the Docker image built can only run on the X86_64 platform. The ARM platform (or M1), likewise.
|
||||
|
||||
### Hardware requirements
|
||||
### Hardware Requirements
|
||||
|
||||
Minimum configuration: 2C 4G
|
||||
|
||||
@ -42,17 +42,17 @@ Recommended configuration: 4C 16G
|
||||
|
||||
### Software Requirements
|
||||
|
||||
Docker Version: 20.10 and later
|
||||
Docker Version: 20.10 or newer
|
||||
|
||||
## Docker Image build
|
||||
## Build Docker Image
|
||||
|
||||
Dockerfile scripting needs to pay attention to the following points:
|
||||
> 1. The base parent image uses the official OpenJDK image certified by Docker-Hub, and the version uses JDK 1.8
|
||||
> 2. The application uses the official binary package for download by default, do not use binary packages from unknown sources
|
||||
> 3. Embedded scripts are required to complete tasks such as FE startup, multi-FE registration, status check and BE startup, registration of BE to FE, status check, etc.
|
||||
> 4. The application should not be started using `--daemon` when starting in Docker, otherwise there will be exceptions during the deployment of orchestration tools such as K8S
|
||||
During Dockerfile scripting, please note that:
|
||||
> 1. Use the official OpenJDK image certified by Docker-Hub as the base parent image (Version: JDK 1.8).
|
||||
> 2. Use the official binary package for download; do not use binary packages from unknown sources.
|
||||
> 3. Use embedded scripts for tasks such as FE startup, multi-FE registration, FE status check, BE startup, registration of BE to FE, and BE status check.
|
||||
> 4. Do not use `--daemon` to start applications in Docker. Otherwise there will be exceptions during the deployment of orchestration tools such as K8S.
|
||||
|
||||
Since Apache Doris 1.2 began to support JavaUDF capability, BE also needs a JDK environment. The recommended mirror is as follows:
|
||||
Apache Doris 1.2 and the subsequent versions support JavaUDF, so you also need a JDK environment for BE. The recommended images are as follows:
|
||||
|
||||
| Doris Program | Recommended Base Parent Image |
|
||||
| ---------- | ----------------- |
|
||||
@ -60,23 +60,23 @@ Since Apache Doris 1.2 began to support JavaUDF capability, BE also needs a JDK
|
||||
| Backend | openjdk:8u342-jdk |
|
||||
| Broker | openjdk:8u342-jdk |
|
||||
|
||||
### Script preparation
|
||||
### Script Preparation
|
||||
|
||||
In the Dockerfile script for compiling the Docker Image, there are two ways to load the binary package of the Apache Doris program:
|
||||
In the Dockerfile script for compiling the Docker Image, there are two methods to load the binary package of the Apache Doris program:
|
||||
|
||||
1. Execute the download command at compile time via wget / curl, and then complete the docker build process
|
||||
2. Download the binary package to the compilation directory in advance, and then load it into the docker build process through the ADD or COPY command
|
||||
1. Execute the download command via wget / curl when compiling, and then start the docker build process.
|
||||
2. Download the binary package to the compilation directory in advance, and then load it into the docker build process through the ADD or COPY command.
|
||||
|
||||
Using the former will make the Docker Image Size smaller, but if the build fails, the download operation may be repeated, resulting in a long build time, while the latter is more suitable for a build environment where the network environment is not very good. The image built by the latter is slightly larger than the former, but not much larger.
|
||||
Method 1 can produce a smaller Docker image, but if the docker build process fails, the download operation might be repeated and result in longer build time; Method 2 is more suitable for less-than-ideal network environments. Method 2 will produce a Docker image that is slightly larger than that from Method 1.
|
||||
|
||||
**To sum up, the examples in this document are subject to the second method. If you have the first request, you can customize and modify it according to your own needs. **
|
||||
**The examples below are based on Method 2. If you prefer to go for Method 1, you may modify the steps accordingly.**
|
||||
|
||||
### Prepare binary package
|
||||
### Prepare Binary Package
|
||||
|
||||
It should be noted that if there is a need for customized development, you need to modify the source code and then [compile](../source-install/compilation) to package it, and then place it in the build directory.
|
||||
Please noted that if you have a need for custom development, you need to modify the source code, [compile](../source-install/compilation) and package it, and then place it in the build directory.
|
||||
|
||||
If there is no special requirement, just [download](https://doris.apache.org/download) the binary package provided by the official website.
|
||||
### Build Steps
|
||||
If you have no such needs, you can just [download](https://doris.apache.org/download) the binary package from the official website.
|
||||
### Steps
|
||||
|
||||
#### Build FE
|
||||
|
||||
@ -100,17 +100,17 @@ The build environment directory is as follows:
|
||||
|
||||
Copy the binary package to the `./docker-build/fe/resource` directory
|
||||
|
||||
3. Write FE's Dockerfile script
|
||||
3. Write the Dockerfile script for FE
|
||||
|
||||
```powershell
|
||||
# select the base image
|
||||
# Select the base image
|
||||
FROM openjdk:8u342-jdk
|
||||
|
||||
# Set environment variables
|
||||
ENV JAVA_HOME="/usr/local/openjdk-8/" \
|
||||
PATH="/opt/apache-doris/fe/bin:$PATH"
|
||||
|
||||
# Download the software to the mirror and replace it as needed
|
||||
# Download the software into the image (you can modify based on your own needs)
|
||||
ADD ./resource/apache-doris-fe-${x.x.x}-bin.tar.gz /opt/
|
||||
|
||||
RUN apt-get update && \
|
||||
@ -126,19 +126,19 @@ The build environment directory is as follows:
|
||||
ENTRYPOINT ["/opt/apache-doris/fe/bin/init_fe.sh"]
|
||||
```
|
||||
|
||||
After writing, name it `Dockerfile` and save it to the `./docker-build/fe` directory
|
||||
After writing, name it `Dockerfile` and save it to the `./docker-build/fe` directory.
|
||||
|
||||
4. Write the execution script of FE
|
||||
|
||||
You can refer to the content of copying [init_fe.sh](https://github.com/apache/doris/tree/master/docker/runtime/fe/resource/init_fe.sh)
|
||||
You can refer to [init_fe.sh](https://github.com/apache/doris/tree/master/docker/runtime/fe/resource/init_fe.sh).
|
||||
|
||||
After writing, name it `init_fe.sh` and save it to the `./docker-build/fe/resouce` directory
|
||||
After writing, name it `init_fe.sh` and save it to the `./docker-build/fe/resouce` directory.
|
||||
|
||||
5. Execute the build
|
||||
|
||||
It should be noted that `${tagName}` needs to be replaced with the tag name you want to package and name, such as: `apache-doris:1.1.3-fe`
|
||||
Please note that `${tagName}` needs to be replaced with the tag name you want to package and name, such as: `apache-doris:1.1.3-fe`
|
||||
|
||||
Build FEs:
|
||||
Build FE:
|
||||
|
||||
```shell
|
||||
cd ./docker-build/fe
|
||||
@ -162,17 +162,17 @@ mkdir -p ./docker-build/be/resource
|
||||
└── apache-doris-x.x.x-bin-x86_64/arm-be.tar.gz // binary package
|
||||
```
|
||||
|
||||
3. Write BE's Dockerfile script
|
||||
3. Write the Dockerfile script for BE
|
||||
|
||||
```powershell
|
||||
# select the base image
|
||||
# Select the base image
|
||||
FROM openjdk:8u342-jdk
|
||||
|
||||
# Set environment variables
|
||||
ENV JAVA_HOME="/usr/local/openjdk-8/" \
|
||||
PATH="/opt/apache-doris/be/bin:$PATH"
|
||||
|
||||
# Download the software to the mirror and replace it as needed
|
||||
# Download the software into the image (you can modify based on your own needs)
|
||||
ADD ./resource/apache-doris-be-${x.x.x}-bin-x86_64.tar.gz /opt/
|
||||
|
||||
RUN apt-get update && \
|
||||
@ -192,22 +192,22 @@ mkdir -p ./docker-build/be/resource
|
||||
|
||||
4. Write the execution script of BE
|
||||
|
||||
You can refer to the content of copying [init_be.sh](https://github.com/apache/doris/tree/master/docker/runtime/be/resource/init_be.sh)
|
||||
You can refer to [init_be.sh](https://github.com/apache/doris/tree/master/docker/runtime/be/resource/init_be.sh).
|
||||
|
||||
After writing, name it `init_be.sh` and save it to the `./docker-build/be/resouce` directory
|
||||
After writing, name it `init_be.sh` and save it to the `./docker-build/be/resouce` directory.
|
||||
|
||||
5. Execute the build
|
||||
|
||||
It should be noted that `${tagName}` needs to be replaced with the tag name you want to package and name, such as: `apache-doris:1.1.3-be`
|
||||
Please note that `${tagName}` needs to be replaced with the tag name you want to package and name, such as: `apache-doris:1.1.3-be`
|
||||
|
||||
Build BEs:
|
||||
Build BE:
|
||||
|
||||
```shell
|
||||
cd ./docker-build/be
|
||||
docker build . -t ${be-tagName}
|
||||
```
|
||||
|
||||
After the construction is complete, there will be a prompt of `Success`. At this time, the following command can be used to view the Image mirror that has just been built
|
||||
After the build process is completed, you will see the prompt `Success`. Then, you can check the built image using the following command.
|
||||
|
||||
```shell
|
||||
docker images
|
||||
@ -231,17 +231,17 @@ mkdir -p ./docker-build/broker/resource
|
||||
└── apache-doris-x.x.x-bin-broker.tar.gz // binary package
|
||||
```
|
||||
|
||||
3. Write Broker's Dockerfile script
|
||||
3. Write the Dockerfile script for Broker
|
||||
|
||||
```powershell
|
||||
# select the base image
|
||||
# Select the base image
|
||||
FROM openjdk:8u342-jdk
|
||||
|
||||
# Set environment variables
|
||||
ENV JAVA_HOME="/usr/local/openjdk-8/" \
|
||||
PATH="/opt/apache-doris/broker/bin:$PATH"
|
||||
|
||||
# Download the software to the mirror, where the broker directory is synchronously compressed to the binary package of FE, which needs to be decompressed and repackaged by itself, and can be replaced as needed
|
||||
# Download the software into the image, where the broker directory is synchronously compressed to the binary package of FE, which needs to be decompressed and repackaged (you can modify based on your own needs)
|
||||
ADD ./resource/apache_hdfs_broker.tar.gz /opt/
|
||||
|
||||
RUN apt-get update && \
|
||||
@ -261,13 +261,13 @@ mkdir -p ./docker-build/broker/resource
|
||||
|
||||
4. Write the execution script of BE
|
||||
|
||||
You can refer to the content of copying [init_broker.sh](https://github.com/apache/doris/tree/master/docker/runtime/broker/resource/init_broker.sh)
|
||||
You can refer to [init_broker.sh](https://github.com/apache/doris/tree/master/docker/runtime/broker/resource/init_broker.sh).
|
||||
|
||||
After writing, name it `init_broker.sh` and save it to the `./docker-build/broker/resouce` directory
|
||||
After writing, name it `init_broker.sh` and save it to the `./docker-build/broker/resouce` directory.
|
||||
|
||||
5. Execute the build
|
||||
|
||||
It should be noted that `${tagName}` needs to be replaced with the tag name you want to package and name, such as: `apache-doris:1.1.3-broker`
|
||||
Please note that `${tagName}` needs to be replaced with the tag name you want to package and name, such as: `apache-doris:1.1.3-broker`
|
||||
|
||||
Build Broker:
|
||||
|
||||
@ -276,21 +276,21 @@ mkdir -p ./docker-build/broker/resource
|
||||
docker build . -t ${broker-tagName}
|
||||
```
|
||||
|
||||
After the construction is complete, there will be a prompt of `Success`. At this time, the following command can be used to view the Image mirror that has just been built
|
||||
After the build process is completed, you will see the prompt `Success`. Then, you can check the built image using the following command.
|
||||
|
||||
```shell
|
||||
docker images
|
||||
```
|
||||
|
||||
## Push the image to DockerHub or private warehouse
|
||||
## Push Image to DockerHub or Private Warehouse
|
||||
|
||||
Log in to your DockerHub account
|
||||
Log into your DockerHub account
|
||||
|
||||
```
|
||||
docker login
|
||||
```
|
||||
|
||||
A successful login will prompt `Success` related prompts, and then push them to the warehouse
|
||||
If the login succeeds, you will see the prompt `Success` , and then you can push the Docker image to the warehouse.
|
||||
|
||||
```shell
|
||||
docker push ${tagName}
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
{
|
||||
"title": "Installation and deployment",
|
||||
"title": "Installation and Deployment",
|
||||
"language": "en"
|
||||
}
|
||||
---
|
||||
@ -25,16 +25,17 @@ under the License.
|
||||
-->
|
||||
|
||||
|
||||
# Installation and deployment
|
||||
# Installation and Deployment
|
||||
|
||||
This document mainly introduces the hardware and software environment needed to deploy Doris, the proposed deployment mode, cluster expansion and scaling, and common problems in the process of cluster building and running.
|
||||
Before reading this document, compile Doris according to the compiled document.
|
||||
This topic is about the hardware and software environment needed to deploy Doris, the recommended deployment mode, cluster scaling, and common problems occur in creating and running clusters.
|
||||
|
||||
## Software and hardware requirements
|
||||
Before continue reading, you might want to compile Doris following the instructions in the [Compile](https://doris.apache.org/docs/dev/install/source-install/compilation/) topic.
|
||||
|
||||
## Software and Hardware Requirements
|
||||
|
||||
### Overview
|
||||
|
||||
Doris, as an open source MPP architecture OLAP database, can run on most mainstream commercial servers. In order to make full use of the concurrency advantages of MPP architecture and the high availability features of Doris, we recommend that the deployment of Doris follow the following requirements:
|
||||
Doris, as an open source OLAP database with an MPP architecture, can run on most mainstream commercial servers. For you to take full advantage of the high concurrency and high availability of Doris, we recommend that your computer meet the following requirements:
|
||||
|
||||
#### Linux Operating System Version Requirements
|
||||
|
||||
@ -43,7 +44,7 @@ Doris, as an open source MPP architecture OLAP database, can run on most mainstr
|
||||
| Centos | 7.1 and above |
|
||||
| Ubuntu | 16.04 and above |
|
||||
|
||||
#### Software requirements
|
||||
#### Software Requirements
|
||||
|
||||
| Soft | Version |
|
||||
|---|---|
|
||||
@ -52,7 +53,7 @@ Doris, as an open source MPP architecture OLAP database, can run on most mainstr
|
||||
|
||||
#### OS Installation Requirements
|
||||
|
||||
##### Set the maximum number of open file handles in the system
|
||||
**Set the maximum number of open file descriptors in the system**
|
||||
|
||||
````
|
||||
vi /etc/security/limits.conf
|
||||
@ -62,24 +63,24 @@ vi /etc/security/limits.conf
|
||||
|
||||
##### Clock synchronization
|
||||
|
||||
The metadata of Doris requires the time precision to be less than 5000ms, so all machines in the cluster need to synchronize the clocks to avoid service exceptions caused by inconsistencies in metadata caused by clock problems.
|
||||
The metadata in Doris requires a time precision of less than 5000ms, so all machines in all clusters need to synchronize their clocks to avoid service exceptions caused by inconsistencies in metadata caused by clock problems.
|
||||
|
||||
##### Close the swap partition (swap)
|
||||
##### Close the swap partition
|
||||
|
||||
The Linux swap partition will cause serious performance problems for Doris, you need to disable the swap partition before installation
|
||||
The Linux swap partition can cause serious performance problems for Doris, so you need to disable the swap partition before installation.
|
||||
|
||||
##### Linux file system
|
||||
|
||||
Here we recommend using the ext4 file system. When installing the operating system, please select the ext4 file system.
|
||||
When installing the operating system, we recommend that you select the ext4 file system.
|
||||
|
||||
#### Development Test Environment
|
||||
|
||||
| Module | CPU | Memory | Disk | Network | Instance Number|
|
||||
| Module | CPU | Memory | Disk | Network | Number of Instances |
|
||||
|---|---|---|---|---|---|
|
||||
| Frontend | 8 core + | 8GB + | SSD or SATA, 10GB + * | Gigabit Network Card | 1|
|
||||
| Backend | 8 core + | 16GB + | SSD or SATA, 50GB + * | Gigabit Network Card | 1-3*|
|
||||
|
||||
#### Production environment
|
||||
#### Production Environment
|
||||
|
||||
| Module | CPU | Memory | Disk | Network | Number of Instances (Minimum Requirements) |
|
||||
|---|---|---|---|---|--------------------------------------------|
|
||||
@ -88,183 +89,188 @@ Here we recommend using the ext4 file system. When installing the operating syst
|
||||
|
||||
> Note 1:
|
||||
>
|
||||
> 1. The disk space of FE is mainly used to store metadata, including logs and images. Usually it ranges from several hundred MB to several GB.
|
||||
> 2. BE's disk space is mainly used to store user data. The total disk space is calculated according to the user's total data * 3 (3 copies). Then an additional 40% of the space is reserved for background compaction and some intermediate data storage.
|
||||
> 3. Multiple BE instances can be deployed on a single machine, but **can only deploy one FE**. If you need three copies of data, you need at least one BE instance per machine (instead of three BE instances per machine). **Clocks of multiple FE servers must be consistent (allowing a maximum of 5 seconds clock deviation)**
|
||||
> 4. The test environment can also be tested with only one BE. In the actual production environment, the number of BE instances directly determines the overall query latency.
|
||||
> 5. All deployment nodes close Swap.
|
||||
> 1. The disk space of FE is mainly used to store metadata, including logs and images. It usually ranges from several hundred MB to several GB.
|
||||
> 2. The disk space of BE is mainly used to store user data. The total disk space taken up is 3 times the total user data (3 copies). Then an additional 40% of the space is reserved for background compaction and intermediate data storage.
|
||||
> 3. On one single machine, you can deploy multiple BE instances but **only one FE instance**. If you need 3 copies of the data, you need to deploy 3 BE instances on 3 machines (1 instance per machine) instead of 3 BE instances on one machine). **Clocks of the FE servers must be consistent (allowing a maximum clock skew of 5 seconds).**
|
||||
> 4. The test environment can also be tested with only 1 BE instance. In the actual production environment, the number of BE instances directly determines the overall query latency.
|
||||
> 5. Disable swap for all deployment nodes.
|
||||
|
||||
> Note 2: Number of FE nodes
|
||||
>
|
||||
> 1. FE roles are divided into Follower and Observer. (Leader is an elected role in the Follower group, hereinafter referred to as Follower, for the specific meaning)
|
||||
> 2. FE node data is at least 1 (1 Follower). When one Follower and one Observer are deployed, high read availability can be achieved. When three Followers are deployed, read-write high availability (HA) can be achieved.
|
||||
> 3. The number of Followers **must be** odd, and the number of Observers is arbitrary.
|
||||
> 4. According to past experience, when cluster availability requirements are high (e.g. providing online services), three Followers and one to three Observers can be deployed. For offline business, it is recommended to deploy 1 Follower and 1-3 Observers.
|
||||
> 1. FE nodes are divided into Followers and Observers based on their roles. (Leader is an elected role in the Follower group, hereinafter referred to as Follower, too.)
|
||||
> 2. The number of FE nodes should be at least 1 (1 Follower). If you deploy 1 Follower and 1 Observer, you can achieve high read availability; if you deploy 3 Followers, you can achieve high read-write availability (HA).
|
||||
> 3. The number of Followers **must be** odd, and there is no limit on the number of Observers.
|
||||
> 4. According to past experience, for business that requires high cluster availability (e.g. online service providers), we recommend that you deploy 3 Followers and 1-3 Observers; for offline business, we recommend that you deploy 1 Follower and 1-3 Observers.
|
||||
|
||||
* **Usually we recommend about 10 to 100 machines to give full play to Doris's performance (3 of them deploy FE (HA) and the rest deploy BE)**
|
||||
* **Of course, Doris performance is positively correlated with the number and configuration of nodes. With a minimum of four machines (one FE, three BEs, one BE mixed with one Observer FE to provide metadata backup) and a lower configuration, Doris can still run smoothly.**
|
||||
* **If FE and BE are mixed, we should pay attention to resource competition and ensure that metadata catalogue and data catalogue belong to different disks.**
|
||||
* **Usually we recommend 10 to 100 machines to give full play to Doris' performance (deploy FE on 3 of them (HA) and BE on the rest).**
|
||||
* **The performance of Doris is positively correlated with the number of nodes and their configuration. With a minimum of four machines (one FE, three BEs; hybrid deployment of one BE and one Observer FE to provide metadata backup) and relatively low configuration, Doris can still run smoothly.**
|
||||
* **In hybrid deployment of FE and BE, you might need to be watchful for resource competition and ensure that the metadata catalogue and data catalogue belong to different disks.**
|
||||
|
||||
#### Broker deployment
|
||||
#### Broker Deployment
|
||||
|
||||
Broker is a process for accessing external data sources, such as hdfs. Usually, a broker instance is deployed on each machine.
|
||||
Broker is a process for accessing external data sources, such as hdfs. Usually, deploying one broker instance on each machine should be enough.
|
||||
|
||||
#### Network Requirements
|
||||
|
||||
Doris instances communicate directly over the network. The following table shows all required ports
|
||||
Doris instances communicate directly over the network. The following table shows all required ports.
|
||||
|
||||
| Instance Name | Port Name | Default Port | Communication Direction | Description|
|
||||
| ---|---|---|---|---|
|
||||
| BE | be_port | 9060 | FE --> BE | BE for receiving requests from FE|
|
||||
| BE | webserver\_port | 8040 | BE <--> BE | BE|
|
||||
| BE | heartbeat\_service_port | 9050 | FE --> BE | the heart beat service port (thrift) on BE, used to receive heartbeat from FE|
|
||||
| BE | brpc\_port | 8060 | FE <--> BE, BE <--> BE | BE for communication between BEs|
|
||||
| BE | be_port | 9060 | FE --> BE | Thrift server port on BE for receiving requests from FE |
|
||||
| BE | webserver\_port | 8040 | BE <--> BE | HTTP server port on BE |
|
||||
| BE | heartbeat\_service_port | 9050 | FE --> BE | Heart beat service port (thrift) on BE, used to receive heartbeat from FE |
|
||||
| BE | brpc\_port | 8060 | FE <--> BE, BE <--> BE | BRPC port on BE for communication between BEs |
|
||||
| FE | http_port | 8030 | FE <--> FE, user <--> FE | HTTP server port on FE |
|
||||
| FE | rpc_port | 9020 | BE --> FE, FE <--> FE | thrift server port on FE, the configuration of each fe needs to be consistent|
|
||||
| FE | query_port | 9030 | user <--> FE | FE|
|
||||
| FE | edit\_log_port | 9010 | FE <--> FE | FE|
|
||||
| Broker | broker ipc_port | 8000 | FE --> Broker, BE --> Broker | Broker for receiving requests|
|
||||
| FE | rpc_port | 9020 | BE --> FE, FE <--> FE | Thrift server port on FE; The configurations of each FE should be consistent. |
|
||||
| FE | query_port | 9030 | user <--> FE | MySQL server port on FE |
|
||||
| FE | edit\_log_port | 9010 | FE <--> FE | Port on FE for BDBJE communication |
|
||||
| Broker | broker ipc_port | 8000 | FE --> Broker, BE --> Broker | Thrift server port on Broker for receiving requests |
|
||||
|
||||
> Note:
|
||||
>
|
||||
> 1. When deploying multiple FE instances, make sure that the http port configuration of FE is the same.
|
||||
> 1. When deploying multiple FE instances, make sure that the http_port configuration of each FE is consistent.
|
||||
> 2. Make sure that each port has access in its proper direction before deployment.
|
||||
|
||||
#### IP binding
|
||||
#### IP Binding
|
||||
|
||||
Because of the existence of multiple network cards, or the existence of virtual network cards caused by the installation of docker and other environments, the same host may have multiple different ips. Currently Doris does not automatically identify available IP. So when you encounter multiple IP on the deployment host, you must force the correct IP to be specified through the priority\_networks configuration item.
|
||||
Because of the existence of multiple network cards, or the existence of virtual network cards caused by the installation of docker and other environments, the same host may have multiple different IPs. Currently Doris does not automatically identify available IPs. So when you encounter multiple IPs on the deployment host, you must specify the correct IP via the `priority_networks` configuration item.
|
||||
|
||||
Priority\_networks is a configuration that both FE and BE have, and the configuration items need to be written in fe.conf and be.conf. This configuration item is used to tell the process which IP should be bound when FE or BE starts. Examples are as follows:
|
||||
`priority_networks` is a configuration item that both FE and BE have. It needs to be written in fe.conf and be.conf. It is used to tell the process which IP should be bound when FE or BE starts. Examples are as follows:
|
||||
|
||||
`priority_networks=10.1.3.0/24`
|
||||
|
||||
This is a representation of [CIDR](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing). FE or BE will find the matching IP based on this configuration item as their own local IP.
|
||||
|
||||
**Note**: When priority networks is configured and FE or BE is started, only the correct IP binding of FE or BE is ensured. In ADD BACKEND or ADD FRONTEND statements, you also need to specify IP matching priority networks configuration, otherwise the cluster cannot be established. Give an example:
|
||||
**Note**: Configuring `priority_networks` and starting FE or BE only ensure the correct IP binding of FE or BE. You also need to specify the same IP in ADD BACKEND or ADD FRONTEND statements, otherwise the cluster cannot be created. For example:
|
||||
|
||||
BE is configured as `priority_networks = 10.1.3.0/24'.`.
|
||||
|
||||
When you want to ADD BACKEND use: `ALTER SYSTEM ADD BACKEND "192.168.0.1:9050";`
|
||||
If you use the following IP in the ADD BACKEND statement: `ALTER SYSTEM ADD BACKEND "192.168.0.1:9050";`
|
||||
|
||||
Then FE and BE will not be able to communicate properly.
|
||||
|
||||
At this point, DROP must remove the BE that added errors and re-use the correct IP to perform ADD BACKEND.
|
||||
At this point, you must DROP the wrong BE configuration and use the correct IP to perform ADD BACKEND.
|
||||
|
||||
FE is the same.
|
||||
The same works for FE.
|
||||
|
||||
BROKER does not currently have, nor does it need, priority\_networks. Broker's services are bound to 0.0.0 by default. Simply execute the correct accessible BROKER IP when ADD BROKER is used.
|
||||
Broker currently does not have the `priority_networks` configuration item, nor does it need. Broker's services are bound to 0.0.0.0 by default. You can simply execute the correct accessible BROKER IP when using ADD BROKER.
|
||||
|
||||
#### Table Name Case Sensitivity Setting
|
||||
#### Table Name Case Sensitivity
|
||||
|
||||
By default, doris is case-sensitive. If there is a need for case-insensitive table names, you need to set it before cluster initialization. The table name case sensitivity cannot be changed after cluster initialization is completed.
|
||||
By default, table names in Doris are case-sensitive. If you need to change that, you may do it before cluster initialization. The table name case sensitivity cannot be changed after cluster initialization is completed.
|
||||
|
||||
See the section on `lower_case_table_names` variables in [Variables](../../advanced/variables) for details.
|
||||
See the `lower_case_table_names` section in [Variables](../../advanced/variables) for details.
|
||||
|
||||
## Cluster deployment
|
||||
## Cluster Deployment
|
||||
|
||||
### Manual deployment
|
||||
### Manual Deployment
|
||||
|
||||
#### Deploy FE
|
||||
|
||||
* Copy the FE deployment file to the specified node
|
||||
* Copy the FE deployment file into the specified node
|
||||
|
||||
Copy the Fe folder under output generated by source code compilation to the node specified deployment path of FE and enter this directory.
|
||||
Find the Fe folder under the output generated by source code compilation, copy it into the specified deployment path of FE nodes and put it the corresponding directory.
|
||||
|
||||
* Configure FE
|
||||
|
||||
1. The configuration file is conf/fe.conf. Note: `meta_dir` indicates the Metadata storage location. The default value is `${DORIS_HOME}/doris-meta`. The directory needs to be **created manually**.
|
||||
2. JAVA_OPTS in fe.conf defaults to a maximum heap memory of 4GB for java, and it is recommended that the production environment be adjusted to more than 8G.
|
||||
1. The configuration file is conf/fe.conf. Note: `meta_dir` indicates the metadata storage location. The default value is `${DORIS_HOME}/doris-meta`. The directory needs to be **created manually**.
|
||||
|
||||
**Note: For production environments, it is better not to put the directory under the Doris installation directory but in a separate disk (SSD would be the best); for test and development environments, you may use the default configuration.**
|
||||
|
||||
2. The default maximum Java heap memory of JAVA_OPTS in fe.conf is 4GB. For production environments, we recommend that it be adjusted to more than 8G.
|
||||
|
||||
* Start FE
|
||||
|
||||
`bin/start_fe.sh --daemon`
|
||||
|
||||
The FE process starts and enters the background execution. Logs are stored in the log/ directory by default. If startup fails, you can view error messages by looking at log/fe.log or log/fe.out.
|
||||
The FE process starts and enters the background for execution. Logs are stored in the log/ directory by default. If startup fails, you can view error messages by checking out log/fe.log or log/fe.out.
|
||||
|
||||
* For deployment of multiple FEs, see the section "FE scaling and downsizing"
|
||||
* For details about deployment of multiple FEs, see the [FE scaling](https://doris.apache.org/docs/dev/admin-manual/cluster-management/elastic-expansion/) section.
|
||||
|
||||
#### Deploy BE
|
||||
|
||||
* Copy BE deployment files to all nodes to deploy BE
|
||||
* Copy the BE deployment file into all nodes that are to deploy BE on
|
||||
|
||||
Copy the be folder under output generated by source code compilation to the specified deployment path of the BE node.
|
||||
Find the BE folder under the output generated by source code compilation, copy it into to the specified deployment paths of the BE nodes.
|
||||
|
||||
* Modify all BE configurations
|
||||
|
||||
Modify be/conf/be.conf. Mainly configure `storage_root_path`: data storage directory. The default is be/storage, this directory needs to be **created manually** by. In multi directories case, using `;` separation (do not add `;` after the last directory).
|
||||
If the user does not use a mix of SSD and HDD disks, they do not need to configure the configuration methods in Example 1 and Example 2 below, but only need to specify the storage directory; they also do not need to modify the default storage media configuration of FE.
|
||||
Modify be/conf/be.conf, which mainly involves configuring `storage_root_path`: data storage directory. By default, under be/storage, the directory needs to be **created manually**. Use `;` to separate multiple paths (do not add `;` after the last directory).
|
||||
|
||||
eg.1:
|
||||
|
||||
Note: For SSD disks, '.SSD 'is followed by the directory, and for HDD disks,'.HDD 'is followed by the directory
|
||||
You may specify the directory storage medium in the path: HDD or SSD. You may also add capacility limit to the end of every path and use `,` for separation. Unless you use a mix of SSD and HDD disks, you do not need to follow the configuration methods in Example 1 and Example 2 below, but only need to specify the storage directory; you do not need to modify the default storage medium configuration of FE, either.
|
||||
|
||||
Example 1:
|
||||
|
||||
Note: For SSD disks, add `.SSD` to the end of the directory; for HDD disks, add `.HDD`.
|
||||
|
||||
`storage_root_path=/home/disk1/doris.HDD;/home/disk2/doris.SSD;/home/disk2/doris`
|
||||
|
||||
**instructions**
|
||||
**Description**
|
||||
|
||||
* 1./home/disk1/doris.HDD : The storage medium is HDD;
|
||||
* 2./home/disk2/doris.SSD : The storage medium is HDDSSD;
|
||||
* 3./home/disk2/doris : The storage medium is HDD(default)
|
||||
* 1./home/disk1/doris.HDD: The storage medium is HDD;
|
||||
* 2./home/disk2/doris.SSD: The storage medium is SSD;
|
||||
* 3./home/disk2/doris: The storage medium is HDD (default).
|
||||
|
||||
eg.2:
|
||||
Example 2:
|
||||
|
||||
Note: you do not need to add the suffix to either HDD or SSD disk directories. You only need to set the medium parameter
|
||||
Note: You do not need to add the `.SSD` or `.HDD` suffix, but to specify the medium in the `storage_root_path` parameter
|
||||
|
||||
`storage_root_path=/home/disk1/doris,medium:hdd;/home/disk2/doris,medium:ssd`
|
||||
|
||||
**instructions**
|
||||
**Description**
|
||||
|
||||
|
||||
* 1./home/disk1/doris,medium:hdd : The storage medium is HDD;
|
||||
* 2./home/disk2/doris,medium:ssd : The storage medium is SSD;
|
||||
* 2./home/disk2/doris,medium:ssd : The storage medium is SSD.
|
||||
|
||||
* BE webserver_port configuration
|
||||
|
||||
If the Be componet is installed in hadoop cluster , need to change configuration `webserver_port=8040` to avoid port used.
|
||||
If the BE componet is installed in hadoop cluster, you need to change the configuration `webserver_port=8040` in be.conf to avoid port conflicts.
|
||||
|
||||
* Set JAVA_HOME environment variable
|
||||
|
||||
<version since="1.2.0"></version>
|
||||
Java UDF are supported since version 1.2, so BE are dependent on the Java environment. It is necessary to set the `JAVA_HOME` environment variable before starting. You can also add `export JAVA_HOME=your_java_home_path` to the first line of the `start_be.sh` startup script to set the variable.
|
||||
Java UDF is supported since version 1.2, so BEs are dependent on the Java environment. It is necessary to set the `JAVA_HOME` environment variable before starting. You can also do this by adding `export JAVA_HOME=your_java_home_path` to the first line of the `start_be.sh` startup script.
|
||||
|
||||
* Install Java UDF functions
|
||||
* Install Java UDF
|
||||
|
||||
<version since="1.2.0">Install Java UDF functions</version>
|
||||
Because Java UDF functions are supported from version 1.2, you need to download the JAR package of Java UDF functions from the official website and put them in the lib directory of BE, otherwise it may fail to start.
|
||||
<version since="1.2.0"></version>
|
||||
Because Java UDF is supported since version 1.2, you need to download the JAR package of Java UDF from the official website and put them under the lib directory of BE, otherwise it may fail to start.
|
||||
|
||||
* Add all BE nodes to FE
|
||||
|
||||
BE nodes need to be added in FE before they can join the cluster. You can use mysql-client([Download MySQL 5.7](https://dev.mysql.com/downloads/mysql/5.7.html)) to connect to FE:
|
||||
BE nodes need to be added in FE before they can join the cluster. You can use mysql-client ([Download MySQL 5.7](https://dev.mysql.com/downloads/mysql/5.7.html)) to connect to FE:
|
||||
|
||||
`./mysql-client -h fe_host -P query_port -uroot`
|
||||
|
||||
The fe_host is the node IP where FE is located; the query_port in fe/conf/fe.conf; the root account is used by default and no password is used to login.
|
||||
`fe_host` is the node IP where FE is located; `query_port` is in fe/conf/fe.conf; the root account is used by default and no password is required in login.
|
||||
|
||||
After login, execute the following commands to add each BE:
|
||||
After login, execute the following command to add all the Best:
|
||||
|
||||
`ALTER SYSTEM ADD BACKEND "be_host:heartbeat_service_port";`
|
||||
|
||||
The be_host is the node IP where BE is located; the heartbeat_service_port in be/conf/be.conf.
|
||||
`be_host` is the node IP where BE is located; `heartbeat_service_port` is in be/conf/be.conf.
|
||||
|
||||
* Start BE
|
||||
|
||||
`bin/start_be.sh --daemon`
|
||||
|
||||
The BE process will start and go into the background for execution. Logs are stored in be/log/directory by default. If startup fails, you can view error messages by looking at be/log/be.log or be/log/be.out.
|
||||
The BE process will start and go into the background for execution. Logs are stored in be/log/directory by default. If startup fails, you can view error messages by checking out be/log/be.log or be/log/be.out.
|
||||
|
||||
* View BE status
|
||||
|
||||
Connect to FE using mysql-client and execute `SHOW PROC '/backends'; `View BE operation. If everything is normal, the `isAlive`column should be `true`.
|
||||
Connect to FE using mysql-client and execute `SHOW PROC '/backends'; ` to view BE operation status. If everything goes well, the `isAlive`column should be `true`.
|
||||
|
||||
#### (Optional) FS_Broker deployment
|
||||
#### (Optional) FS_Broker Deployment
|
||||
|
||||
Broker is deployed as a plug-in, independent of Doris. If you need to import data from a third-party storage system, you need to deploy the corresponding Broker. By default, it provides fs_broker to read HDFS ,Object storage (supporting S3 protocol). Fs_broker is stateless and it is recommended that each FE and BE node deploy a Broker.
|
||||
Broker is deployed as a plug-in, which is independent of Doris. If you need to import data from a third-party storage system, you need to deploy the corresponding Broker. By default, Doris provides fs_broker for HDFS reading and object storage (supporting S3 protocol). fs_broker is stateless and we recommend that you deploy a Broker for each FE and BE node.
|
||||
|
||||
* Copy the corresponding Broker directory in the output directory of the source fs_broker to all the nodes that need to be deployed. It is recommended to maintain the same level as the BE or FE directories.
|
||||
* Copy the corresponding Broker directory in the output directory of the source fs_broker to all the nodes that need to be deployed. It is recommended to keep the Broker directory on the same level as the BE or FE directories.
|
||||
|
||||
* Modify the corresponding Broker configuration
|
||||
|
||||
In the corresponding broker/conf/directory configuration file, you can modify the corresponding configuration.
|
||||
You can modify the configuration in the corresponding broker/conf/directory configuration file.
|
||||
|
||||
* Start Broker
|
||||
|
||||
@ -278,105 +284,103 @@ Broker is deployed as a plug-in, independent of Doris. If you need to import dat
|
||||
|
||||
`ALTER SYSTEM ADD BROKER broker_name "broker_host1:broker_ipc_port1","broker_host2:broker_ipc_port2",...;`
|
||||
|
||||
The broker\_host is Broker's node ip; the broker_ipc_port is in the Broker configuration file.
|
||||
`broker\_host` is the Broker node ip; `broker_ipc_port` is in conf/apache_hdfs_broker.conf in the Broker configuration file.
|
||||
|
||||
* View Broker status
|
||||
|
||||
Connect any booted FE using mysql-client and execute the following command to view Broker status: `SHOW PROC '/brokers';`
|
||||
Connect any started FE using mysql-client and execute the following command to view Broker status: `SHOW PROC '/brokers';`
|
||||
|
||||
**Note: In production environments, daemons should be used to start all instances to ensure that processes are automatically pulled up after they exit, such as [Supervisor](http://supervisord.org/). For daemon startup, in 0.9.0 and previous versions, you need to modify the start_xx.sh scripts to remove the last & symbol**. Starting with version 0.10.0, call `sh start_xx.sh` directly to start. Also refer to [here](https://www.cnblogs.com/lenmom/p/9973401.html)
|
||||
**Note: In production environments, daemons should be used to start all instances to ensure that processes are automatically pulled up after they exit, such as [Supervisor](http://supervisord.org/). For daemon startup, in Doris 0.9.0 and previous versions, you need to remove the last `&` symbol in the start_xx.sh scripts**. In Doris 0.10.0 and the subsequent versions, you may just call `sh start_xx.sh` directly to start.
|
||||
|
||||
## Common Questions
|
||||
## FAQ
|
||||
|
||||
### Process correlation
|
||||
### Process-Related Questions
|
||||
|
||||
1. How to determine the success of FE process startup
|
||||
1. How can we know whether the FE process startup succeeds?
|
||||
|
||||
After the FE process starts, metadata is loaded first. According to the different roles of FE, you can see ```transfer from UNKNOWN to MASTER/FOLLOWER/OBSERVER```in the log. Eventually, you will see the ``thrift server started`` log and connect to FE through MySQL client, which indicates that FE started successfully.
|
||||
After the FE process starts, metadata is loaded first. Based on the role of FE, you can see ```transfer from UNKNOWN to MASTER/FOLLOWER/OBSERVER``` in the log. Eventually, you will see the ``thrift server started`` log and can connect to FE through MySQL client, which indicates that FE started successfully.
|
||||
|
||||
You can also check whether the startup was successful by connecting as follows:
|
||||
|
||||
`http://fe_host:fe_http_port/api/bootstrap`
|
||||
|
||||
If returned:
|
||||
If it returns:
|
||||
|
||||
`{"status":"OK","msg":"Success"}`
|
||||
|
||||
The startup is successful, there may be problems in other cases.
|
||||
The startup is successful; otherwise, there may be problems.
|
||||
|
||||
> Note: If you can't see the information of boot failure in fe. log, you may see it in fe. out.
|
||||
> Note: If you can't see the information of boot failure in fe. log, you may check in fe. out.
|
||||
|
||||
2. How to determine the success of BE process startup
|
||||
2. How can we know whether the BE process startup succeeds?
|
||||
|
||||
After the BE process starts, if there is data before, there may be several minutes of data index loading time.
|
||||
After the BE process starts, if there have been data there before, it might need several minutes for data index loading.
|
||||
|
||||
If BE is started for the first time or the BE has not joined any cluster, the BE log will periodically scroll the words `waiting to receive first heartbeat from frontend`. BE has not received Master's address through FE's heartbeat and is waiting passively. This error log will disappear after ADD BACKEND in FE sends the heartbeat. If the word `````master client', get client from cache failed. host:, port: 0, code: 7````` master client appears again after receiving heartbeat, it indicates that FE has successfully connected BE, but BE cannot actively connect FE. It may be necessary to check the connectivity of rpc_port from BE to FE.
|
||||
If BE is started for the first time or the BE has not joined any cluster, the BE log will periodically scroll the words `waiting to receive first heartbeat from frontend`, meaning that BE has not received the Master's address through FE's heartbeat and is waiting passively. Such error log will disappear after sending the heartbeat by ADD BACKEND in FE. If the word `````master client', get client from cache failed. host:, port: 0, code: 7````` appears after receiving the heartbeat, it indicates that FE has successfully connected BE, but BE cannot actively connect FE. You may need to check the connectivity of rpc_port from BE to FE.
|
||||
|
||||
If BE has been added to the cluster, the heartbeat log from FE should be scrolled every five seconds: ```get heartbeat, host:xx. xx.xx.xx, port:9020, cluster id:xxxxxxx```, indicating that the heartbeat is normal.
|
||||
If BE has been added to the cluster, the heartbeat log from FE will be scrolled every five seconds: ```get heartbeat, host:xx. xx.xx.xx, port:9020, cluster id:xxxxxxx```, indicating that the heartbeat is normal.
|
||||
|
||||
Secondly, the word `finish report task success. return code: 0` should be scrolled every 10 seconds in the log to indicate that BE's communication to FE is normal.
|
||||
Secondly, if the word `finish report task success. return code: 0` is scrolled every 10 seconds in the log, that indicates that the BE-to-FE communication is normal.
|
||||
|
||||
At the same time, if there is a data query, you should see the rolling logs, and have `execute time is xxx` logs, indicating that BE started successfully, and the query is normal.
|
||||
Meanwhile, if there is a data query, you will see the rolling logs and the `execute time is xxx` logs, indicating that BE is started successfully, and the query is normal.
|
||||
|
||||
You can also check whether the startup was successful by connecting as follows:
|
||||
|
||||
`http://be_host:be_http_port/api/health`
|
||||
|
||||
If returned:
|
||||
If it returns:
|
||||
|
||||
`{"status": "OK","msg": "To Be Added"}`
|
||||
|
||||
If the startup is successful, there may be problems in other cases.
|
||||
That means the startup is successful; otherwise, there may be problems.
|
||||
|
||||
> Note: If you can't see the information of boot failure in be.INFO, you may see it in be.out.
|
||||
|
||||
3. How to determine the normal connectivity of FE and BE after building the system
|
||||
3. How can we confirm that the connectivity of FE and BE is normal after building the system?
|
||||
|
||||
Firstly, confirm that FE and BE processes have been started separately and normally, and confirm that all nodes have been added through `ADD BACKEND` or `ADD FOLLOWER/OBSERVER` statements.
|
||||
Firstly, you need to confirm that FE and BE processes have been started separately and worked normally. Then, you need to confirm that all nodes have been added through `ADD BACKEND` or `ADD FOLLOWER/OBSERVER` statements.
|
||||
|
||||
If the heartbeat is normal, BE logs will show ``get heartbeat, host:xx.xx.xx.xx, port:9020, cluster id:xxxxx`` If the heartbeat fails, the words ```backend [10001] get Exception: org.apache.thrift.transport.TTransportException``` will appear in FE's log, or other thrift communication abnormal log, indicating that the heartbeat fails from FE to 10001 BE. Here you need to check the connectivity of FE to BE host's heart-beating port.
|
||||
If the heartbeat is normal, BE logs will show ``get heartbeat, host:xx.xx.xx.xx, port:9020, cluster id:xxxxx``; if the heartbeat fails, you will see ```backend [10001] got Exception: org.apache.thrift.transport.TTransportException``` in FE's log, or other thrift communication abnormal log, indicating that the heartbeat from FE to 10001 BE fails. Here you need to check the connectivity of the FE to BE host heartbeat port.
|
||||
|
||||
If BE's communication to FE is normal, the BE log will display the words `finish report task success. return code: 0`. Otherwise, the words `master client`, get client from cache failed` will appear. In this case, the connectivity of BE to the rpc_port of FE needs to be checked.
|
||||
If the BE-to-FE communication is normal, the BE log will display the words `finish report task success. return code: 0`. Otherwise, the words `master client, get client from cache failed` will appear. In this case, you need to check the connectivity of BE to the rpc_port of FE.
|
||||
|
||||
4. Doris Node Authentication Mechanism
|
||||
4. What is the Doris node authentication mechanism?
|
||||
|
||||
In addition to Master FE, the other role nodes (Follower FE, Observer FE, Backend) need to register to the cluster through the `ALTER SYSTEM ADD` statement before joining the cluster.
|
||||
|
||||
When Master FE is first started, a cluster_id is generated in the doris-meta/image/VERSION file.
|
||||
When Master FE is started for the first time, a cluster_id is generated in the doris-meta/image/VERSION file.
|
||||
|
||||
When FE first joins the cluster, it first retrieves the file from Master FE. Each subsequent reconnection between FEs (FE reboot) checks whether its cluster ID is the same as that of other existing FEs. If different, the FE will exit automatically.
|
||||
When FE joins the cluster for the first time, it first retrieves the file from Master FE. Each subsequent reconnection between FEs (FE reboot) checks whether its cluster ID is the same as that of other existing FEs. If it is not the same, the FE will exit automatically.
|
||||
|
||||
When BE first receives the heartbeat of Master FE, it gets the cluster ID from the heartbeat and records it in the `cluster_id` file of the data directory. Each heartbeat after that compares to the cluster ID sent by FE. If cluster IDs are not equal, BE will refuse to respond to FE's heartbeat.
|
||||
When BE first receives the heartbeat of Master FE, it gets the cluster ID from the heartbeat and records it in the `cluster_id` file of the data directory. Each heartbeat after that compares that to the cluster ID sent by FE. If the cluster IDs are not matched, BE will refuse to respond to FE's heartbeat.
|
||||
|
||||
The heartbeat also contains Master FE's ip. When FE cuts the master, the new Master FE will carry its own IP to send the heartbeat to BE, BE will update its own saved Master FE ip.
|
||||
The heartbeat also contains Master FE's IP. If the Master FE changes, the new Master FE will send the heartbeat to BE together with its own IP, and BE will update the Master FE IP it saved.
|
||||
|
||||
> **priority\_network**
|
||||
>
|
||||
> priority network is that both FE and BE have a configuration. Its main purpose is to assist FE or BE to identify their own IP addresses in the case of multi-network cards. Priority network is represented by CIDR: [RFC 4632](https://tools.ietf.org/html/rfc4632)
|
||||
> priority_network is a configuration item that both FE and BE have. It is used to help FE or BE identify their own IP addresses in the cases of multi-network cards. priority_network uses CIDR notation: [RFC 4632](https://tools.ietf.org/html/rfc4632)
|
||||
>
|
||||
> When the connectivity of FE and BE is confirmed to be normal, if the table Timeout still occurs, and the FE log has an error message with the words `backend does not find. host:xxxx.xxx.XXXX`. This means that there is a problem with the IP address that Doris automatically identifies and that priority\_network parameters need to be set manually.
|
||||
> If the connectivity of FE and BE is confirmed to be normal, but timeout still occurs in creating tables, and the FE log shows the error message `backend does not find. host:xxxx.xxx.XXXX`, this means that there is a problem with the IP address automatically identified by Doris and that the priority\_network parameter needs to be set manually.
|
||||
>
|
||||
> The main reason for this problem is that when the user adds BE through the `ADD BACKEND` statement, FE recognizes whether the statement specifies hostname or IP. If it is hostname, FE automatically converts it to an IP address and stores it in metadata. When BE reports on the completion of the task, it carries its own IP address. If FE finds that BE reports inconsistent IP addresses and metadata, it will make the above error.
|
||||
> The explanation to this error is as follows. When the user adds BE through the `ADD BACKEND` statement, FE recognizes whether the statement specifies hostname or IP. If it is a hostname, FE automatically converts it to an IP address and stores it in the metadata. When BE reports on the completion of the task, it carries its own IP address. If FE finds that the IP address reported by BE is different from that in the metadata, the above error message will show up.
|
||||
>
|
||||
> Solutions to this error: 1) Set **priority\_network** parameters in FE and BE respectively. Usually FE and BE are in a network segment, so this parameter can be set to the same. 2) Fill in the `ADD BACKEND` statement directly with the correct IP address of BE instead of hostname to avoid FE getting the wrong IP address.
|
||||
> Solutions to this error: 1) Set **priority\_network** in FE and BE separately. Usually FE and BE are in one network segment, so their priority_network can be set to be the same. 2) Put the correct IP address instead of the hostname in the `ADD BACKEND` statement to prevent FE from getting the wrong IP address.
|
||||
|
||||
5. File descriptor number of BE process
|
||||
5. What is the number of file descriptors of a BE process?
|
||||
|
||||
The number of file descriptor of BE process is controlled by the two parameters min_file_descriptor_number/max_file_descriptor_number.
|
||||
The number of file descriptor of a BE process is determined by two parameters: `min_file_descriptor_number`/`max_file_descriptor_number`.
|
||||
|
||||
If it is not in the [min_file_descriptor_number, max_file_descriptor_number] interval, error will occurs when starting BE process.
|
||||
If it is not in the range [`min_file_descriptor_number`, `max_file_descriptor_number`], error will occurs when starting a BE process. You may use the ulimit command to reset the parameters.
|
||||
|
||||
Please using ulimit command to set file descriptor under this circumstance.
|
||||
The default value of `min_file_descriptor_number` is 65536.
|
||||
|
||||
The default value of min_file_descriptor_number is 65536.
|
||||
The default value of `max_file_descriptor_number` is 131072.
|
||||
|
||||
The default value of max_file_descriptor_number is 131072.
|
||||
For example, the command `ulimit -n 65536;` means to set the number of file descriptors to 65536.
|
||||
|
||||
For Example: ulimit -n 65536; this command set file descriptor to 65536.
|
||||
After starting a BE process, you can use **cat /proc/$pid/limits** to check the actual number of file descriptors of the process.
|
||||
|
||||
After starting BE process, you can use **cat /proc/$pid/limits** to see the actual limit of process.
|
||||
|
||||
if use `supervisord`, try to modify `minfds` in supervisord.conf
|
||||
If you have used `supervisord` and encountered a file descriptor error, you can fix it by modifying `minfds` in supervisord.conf.
|
||||
|
||||
```shell
|
||||
vim /etc/supervisord.conf
|
||||
|
||||
@ -24,15 +24,15 @@ specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Apache Doris ARM architecture compiled
|
||||
# Compile With ARM
|
||||
|
||||
This document describes how to compile Doris on the ARM64 platform.
|
||||
This topic is about how to compile Doris on the ARM64 platform.
|
||||
|
||||
Note that this document is for guidance only. Compiling in different environments may give other errors.
|
||||
Note that this is for reference only. Other errors may occur when compiling in different environments.
|
||||
|
||||
## KylinOS
|
||||
|
||||
### Software and hardware environment
|
||||
### Software and Hardware Environment
|
||||
|
||||
1. KylinOS Version：
|
||||
|
||||
@ -53,15 +53,15 @@ Note that this document is for guidance only. Compiling in different environment
|
||||
model name : Phytium,FT-2000+/64
|
||||
```
|
||||
|
||||
### Compile with ldb-toolchain
|
||||
### Compile With ldb-toolchain
|
||||
|
||||
This method works for Doris versions after [commit 7f3564](https://github.com/apache/doris/commit/7f3564cca62de49c9f2ea67fcf735921dbebb4d1).
|
||||
|
||||
Download [ldb\_toolchain\_gen.aarch64.sh](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.9.1/ldb_toolchain_gen.aarch64.sh)
|
||||
|
||||
For the subsequent compilation method, please refer to [Compile with LDB toolchain](./compilation-with-ldb-toolchain.md)
|
||||
For detailed instructions, please refer to [Compile with ldb-toolchain](./compilation-with-ldb-toolchain.md)
|
||||
|
||||
Note that both jdk and nodejs need to download the corresponding aarch64 version:
|
||||
Note that you need to download the corresponding aarch64 versions of jdk and nodejs:
|
||||
|
||||
1. [Java8-aarch64](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/jdk-8u291-linux-aarch64.tar.gz)
|
||||
2. [Node v12.13.0-aarch64](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v16.3.0-linux-arm64.tar.xz)
|
||||
@ -70,54 +70,54 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
|
||||
### Hardware Environment
|
||||
|
||||
1. System version: CentOS 8.4, Ubuntu 20.04
|
||||
1. System Version: CentOS 8.4, Ubuntu 20.04
|
||||
2. System Architecture: ARM X64
|
||||
3. CPU: 4C
|
||||
4. Memory: 16 GB
|
||||
5. Hard disk: 40GB (SSD), 100GB (SSD)
|
||||
5. Hard Disk: 40GB (SSD), 100GB (SSD)
|
||||
|
||||
### Software Environment
|
||||
|
||||
#### Software environment comparison table
|
||||
#### Software Environment List
|
||||
|
||||
| component name | component version |
|
||||
| ------------------------------------------------------------ | ----------------------------------------- |
|
||||
| Git | 2.0+ |
|
||||
| JDK | 1.8.0 |
|
||||
| Maven | 3.6.3 |
|
||||
| NodeJS | 16.3.0 |
|
||||
| LDB-Toolchain | 0.9.1 |
|
||||
| 常备环境：<br />byacc<br />patch<br />automake<br />libtool<br />make<br />which<br />file<br />ncurses-devel<br />gettext-devel<br />unzip<br />bzip2<br />zip<br />util-linux<br />wget<br />git<br />python2 | yum or apt can be installed automatically |
|
||||
| autoconf | 2.69 |
|
||||
| bison | 3.0.4 |
|
||||
| Component Name | Component Version |
|
||||
| ------------------------------------------------------------ | ------------------------------ |
|
||||
| Git | 2.0+ |
|
||||
| JDK | 1.8.0 |
|
||||
| Maven | 3.6.3 |
|
||||
| NodeJS | 16.3.0 |
|
||||
| LDB-Toolchain | 0.9.1 |
|
||||
| Commonly Used Components<br />byacc<br />patch<br />automake<br />libtool<br />make<br />which<br />file<br />ncurses-devel<br />gettext-devel<br />unzip<br />bzip2<br />zip<br />util-linux<br />wget<br />git<br />python2 | yum install or apt-get install |
|
||||
| autoconf | 2.69 |
|
||||
| bison | 3.0.4 |
|
||||
|
||||
#### Software environment installation command
|
||||
#### Software Environment Installation Command
|
||||
|
||||
##### CentOS 8.4
|
||||
|
||||
- Create software download and installation package root directory and software installation root directory
|
||||
- Create root directories
|
||||
|
||||
```shell
|
||||
# Create the root directory of the software download and installation package
|
||||
# Create root directory for software download and installation packages
|
||||
mkdir /opt/tools
|
||||
# Create software installation root directory
|
||||
# Create root directory for software installation
|
||||
mkdir /opt/software
|
||||
````
|
||||
|
||||
- Git
|
||||
|
||||
```shell
|
||||
# Save the trouble of compiling and install directly with yum
|
||||
# yum install (save the trouble of compilation)
|
||||
yum install -y git
|
||||
````
|
||||
|
||||
- JDK8
|
||||
- JDK8 (2 methods)
|
||||
|
||||
```shell
|
||||
# Two ways, the first is to save additional download and configuration, directly use yum to install, install the devel package to get some tools, such as the jps command
|
||||
# 1. yum install, which can avoid additional download and configuration. Installing the devel package is to get tools such as the jps command.
|
||||
yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
|
||||
|
||||
# The second is to download the installation package of the arm64 architecture, decompress and configure the environment variables and use
|
||||
# 2. Download the installation package of the arm64 architecture, decompress it, and configure the environment variables.
|
||||
cd /opt/tools
|
||||
wget https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/jdk-8u291-linux-aarch64.tar.gz && \
|
||||
tar -zxvf jdk-8u291-linux-aarch64.tar.gz && \
|
||||
@ -128,7 +128,7 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
|
||||
```shell
|
||||
cd /opt/tools
|
||||
# After the wget tool is downloaded, directly decompress the configuration environment variable to use
|
||||
# Download the wget tool, decompress it, and configure the environment variables.
|
||||
wget https://dlcdn.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz && \
|
||||
tar -zxvf apache-maven-3.6.3-bin.tar.gz && \
|
||||
mv apache-maven-3.6.3 /opt/software/maven
|
||||
@ -138,17 +138,17 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
|
||||
```shell
|
||||
cd /opt/tools
|
||||
# Download the installation package for arm64 architecture
|
||||
# Download the installation package of the arm64 architecture
|
||||
wget https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v16.3.0-linux-arm64.tar.xz && \
|
||||
tar -xvf node-v16.3.0-linux-arm64.tar.xz && \
|
||||
mv node-v16.3.0-linux-arm64 /opt/software/nodejs
|
||||
````
|
||||
|
||||
- LDB-Toolchain
|
||||
- ldb-toolchain
|
||||
|
||||
```shell
|
||||
cd /opt/tools
|
||||
# Download LDB-Toolchain ARM version
|
||||
# Download ldb-toolchain ARM version
|
||||
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.9.1/ldb_toolchain_gen.aarch64.sh && \
|
||||
sh ldb_toolchain_gen.aarch64.sh /opt/software/ldb_toolchain/
|
||||
````
|
||||
@ -164,10 +164,10 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
export LDB_HOME=/opt/software/ldb_toolchain
|
||||
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$NODE_JS_HOME/bin:$LDB_HOME/bin:$PATH
|
||||
|
||||
# save and exit and refresh environment variables
|
||||
# Save, exit, and refresh environment variables
|
||||
source /etc/profile.d/doris.sh
|
||||
|
||||
# test for success
|
||||
# Test
|
||||
java -version
|
||||
> java version "1.8.0_291"
|
||||
mvn -version
|
||||
@ -178,13 +178,13 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
> gcc-11
|
||||
````
|
||||
|
||||
- Install other extra environments and components
|
||||
- Install other environments and components
|
||||
|
||||
```shell
|
||||
# install required system packages
|
||||
# Install required system packages
|
||||
sudo yum install -y byacc patch automake libtool make which file ncurses-devel gettext-devel unzip bzip2 bison zip util-linux wget git python2
|
||||
|
||||
# install autoconf-2.69
|
||||
# Install autoconf-2.69
|
||||
cd /opt/tools
|
||||
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz && \
|
||||
tar zxf autoconf-2.69.tar.gz && \
|
||||
@ -205,7 +205,7 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
|
||||
- Check the shell command set
|
||||
|
||||
Ubuntu's shell installs dash instead of bash by default. It needs to be switched to bash to execute. Run the following command to view the details of sh and confirm which program corresponds to the shell:
|
||||
The Ubuntu shell installs dash instead of bash by default. It needs to be switched to bash for proper execution. Run the following command to view the details of sh and confirm which program corresponds to the shell:
|
||||
|
||||
```shell
|
||||
ls -al /bin/sh
|
||||
@ -217,30 +217,30 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
sudo dpkg-reconfigure dash
|
||||
````
|
||||
|
||||
Then select no or no and confirm
|
||||
Then select no to confirm.
|
||||
|
||||
Doing so will reconfigure dash from being the default shell tool
|
||||
After these steps, dash will no longer be the default shell tool.
|
||||
|
||||
- Create software download and installation package root directory and software installation root directory
|
||||
- Create root directories
|
||||
|
||||
```shell
|
||||
# Create the root directory of the software download and installation package
|
||||
# Create root directory for software download and installation packages
|
||||
mkdir /opt/tools
|
||||
# Create software installation root directory
|
||||
# Create root directory for software installation
|
||||
mkdir /opt/software
|
||||
````
|
||||
|
||||
- Git
|
||||
|
||||
```shell
|
||||
# Save the trouble of compiling and install directly with apt-get
|
||||
# apt-get install, which can save the trouble of compilation
|
||||
apt-get -y install git
|
||||
````
|
||||
|
||||
- JDK8
|
||||
|
||||
```shell
|
||||
# Download the installation package of arm64 architecture, decompress and configure environment variables and use
|
||||
# Download the installation package of the ARM64 architecture, decompress it, and configure environment variables.
|
||||
cd /opt/tools
|
||||
wget https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/jdk-8u291-linux-aarch64.tar.gz && \
|
||||
tar -zxvf jdk-8u291-linux-aarch64.tar.gz && \
|
||||
@ -251,7 +251,7 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
|
||||
```shell
|
||||
cd /opt/tools
|
||||
# After the wget tool is downloaded, directly decompress the configuration environment variable to use
|
||||
# Download the wget tool, decompress it, and configure the environment variables.
|
||||
wget https://dlcdn.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz && \
|
||||
tar -zxvf apache-maven-3.6.3-bin.tar.gz && \
|
||||
mv apache-maven-3.6.3 /opt/software/maven
|
||||
@ -261,17 +261,17 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
|
||||
```shell
|
||||
cd /opt/tools
|
||||
# Download the installation package for arm64 architecture
|
||||
# Download the installation package of ARM64 architecture.
|
||||
wget https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v16.3.0-linux-arm64.tar.xz && \
|
||||
tar -xvf node-v16.3.0-linux-arm64.tar.xz && \
|
||||
mv node-v16.3.0-linux-arm64 /opt/software/nodejs
|
||||
````
|
||||
|
||||
- LDB-Toolchain
|
||||
- ldb-toolchain
|
||||
|
||||
```shell
|
||||
cd /opt/tools
|
||||
# Download LDB-Toolchain ARM version
|
||||
# Download ldb-toolchain ARM version
|
||||
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.9.1/ldb_toolchain_gen.aarch64.sh && \
|
||||
sh ldb_toolchain_gen.aarch64.sh /opt/software/ldb_toolchain/
|
||||
````
|
||||
@ -287,10 +287,10 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
export LDB_HOME=/opt/software/ldb_toolchain
|
||||
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$NODE_JS_HOME/bin:$LDB_HOME/bin:$PATH
|
||||
|
||||
# save and exit and refresh environment variables
|
||||
# Save, exit, and refresh environment variables
|
||||
source /etc/profile.d/doris.sh
|
||||
|
||||
# test for success
|
||||
# Test
|
||||
java -version
|
||||
> java version "1.8.0_291"
|
||||
mvn -version
|
||||
@ -301,10 +301,10 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
> gcc-11
|
||||
````
|
||||
|
||||
- Install other extra environments and components
|
||||
- Install other environments and components
|
||||
|
||||
```shell
|
||||
# install required system packages
|
||||
# Install required system packages
|
||||
sudo apt install -y build-essential cmake flex automake bison binutils-dev libiberty-dev zip libncurses5-dev curl ninja-build
|
||||
sudo apt-get install -y make
|
||||
sudo apt-get install -y unzip
|
||||
@ -318,7 +318,7 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
sudo apt install gcc-11 g++-11
|
||||
sudo apt-get -y install autoconf autopoint
|
||||
|
||||
# install autoconf-2.69
|
||||
# Install autoconf-2.69
|
||||
cd /opt/tools
|
||||
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz && \
|
||||
tar zxf autoconf-2.69.tar.gz && \
|
||||
@ -329,18 +329,18 @@ Note that both jdk and nodejs need to download the corresponding aarch64 version
|
||||
make install
|
||||
````
|
||||
|
||||
#### Download the source code
|
||||
#### Download Source Code
|
||||
|
||||
```shell
|
||||
cd /opt
|
||||
git clone https://github.com/apache/doris.git
|
||||
```
|
||||
|
||||
#### Install and deploy
|
||||
#### Install and Deploy
|
||||
|
||||
##### Check if AVX2 instruction set is supported
|
||||
##### Check if the AVX2 instruction set is supported
|
||||
|
||||
If there is data returned, it means support, if no data is returned, it means not supported
|
||||
If there is data returned, that means AVX2 is supported; otherwise, AVX2 is not supported.
|
||||
|
||||
```shell
|
||||
cat /proc/cpuinfo | grep avx2
|
||||
@ -349,27 +349,27 @@ cat /proc/cpuinfo | grep avx2
|
||||
##### Execute compilation
|
||||
|
||||
```shell
|
||||
# For machines that support AVX2 instruction set, you can compile them directly
|
||||
# For machines that support the AVX2 instruction set, compile directly
|
||||
sh build.sh
|
||||
# For machines that do not support the AVX2 instruction set, use the following command to compile
|
||||
USE_AVX2=OFF sh build.sh
|
||||
````
|
||||
|
||||
### common problem
|
||||
### FAQ
|
||||
|
||||
1. Compile the third-party library libhdfs3.a , the folder cannot be found
|
||||
1. File not found when compiling the third-party library libhdfs3.a.
|
||||
|
||||
- Problem Description
|
||||
|
||||
During the compilation and installation process, the following error occurred
|
||||
During the compilation and installation process, the following error occurrs:
|
||||
|
||||
> not found lib/libhdfs3.a file or directory
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
There is a problem with the dependency download of the third-party library
|
||||
The third-party library dependency is improperly downloaded.
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
- Use a third-party download repository
|
||||
|
||||
@ -390,11 +390,11 @@ USE_AVX2=OFF sh build.sh
|
||||
>
|
||||
> Python 2.7.18
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
After searching, it is found that the system uses `python2.7`, `python3.6`, `python2`, `python3` by default to execute python commands. Doris installation requires python 2.7+ version, so only need Just add a command named `python` to connect, both version 2 and version 3 can be used
|
||||
The system uses `python2.7`, `python3.6`, `python2`, `python3` by default to execute python commands. Doris only requires python 2.7+ to install dependencies, so you just need to add a command `python` to connect.
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
Establish a soft link to the `python` command in `\usr\bin`
|
||||
|
||||
@ -409,19 +409,19 @@ USE_AVX2=OFF sh build.sh
|
||||
|
||||
- Problem Description
|
||||
|
||||
- After the execution of build.sh ends, the output folder is not found in the directory.
|
||||
- Cannot find the output folder in the directory after the execution of build.sh.
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
Failed to compile, need to recompile
|
||||
Compilation fails. Try again.
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
```shell
|
||||
sh build.sh --clean
|
||||
````
|
||||
|
||||
4. spark-dpp compilation failed
|
||||
4. spark-dpp compilation fails
|
||||
|
||||
- Problem Description
|
||||
|
||||
@ -429,54 +429,52 @@ USE_AVX2=OFF sh build.sh
|
||||
|
||||
> Failed to execute goal on project spark-dpp
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
The last error message is due to the download failure (and because it failed to connect to the repo.maven.apache.org central repository)
|
||||
This error message is caused by download failure (connection to the `repo.maven.apache.org` central repository fails).
|
||||
|
||||
> Could not transfer artifact org.apache.spark:spark-sql_2.12:jar:2.4.6 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo .maven.apache.org/maven2/org/apache/spark/spark-sql_2.12/2.4.6/spark-sql_2.12-2.4.6.jar: Unknown host repo.maven.apache.org
|
||||
|
||||
rebuild
|
||||
- Solution
|
||||
|
||||
- solution
|
||||
- Rebuild
|
||||
|
||||
- rebuild
|
||||
|
||||
5. The remaining space is insufficient, and the compilation fails
|
||||
5. No space left, compilation fails
|
||||
|
||||
- Problem Description
|
||||
|
||||
- Failed to build CXX object during compilation, indicating insufficient free space
|
||||
- Failed to build CXX object during compilation, error message showing no space left
|
||||
|
||||
> fatal error: error writing to /tmp/ccKn4nPK.s: No space left on device
|
||||
> 1112 | } // namespace doris::vectorized
|
||||
> compilation terminated.
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
Insufficient free space on the device
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
Expand the remaining space of the device, such as deleting unnecessary files, etc.
|
||||
Expand the free space on the device by deleting files you don't need, etc.
|
||||
|
||||
6. Failed to start FE, transaction -20 problem
|
||||
6. Failed to start FE, transaction error -20
|
||||
|
||||
- Problem Description
|
||||
|
||||
When starting FE, a transaction error 20 is reported, and the status is UNKNOWN
|
||||
When starting FE, a transaction error 20 is reported with UNKNOWN status.
|
||||
|
||||
> [BDBEnvironment.setup():198] error to open replicated environment. will exit.
|
||||
> com.sleepycat.je.rep.ReplicaWriteException: (JE 18.3.12) Problem closing transaction 20. The current state is:UNKNOWN. The node transitioned to this state at:Fri Apr 22 12:48:08 CST 2022
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
Insufficient hard disk space, need more space
|
||||
Insufficient hard disk space
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
Free up hard disk space or mount a new hard disk
|
||||
|
||||
7. BDB environment setting is abnormal, disk search error
|
||||
7. Abnormal BDB environment setting, disk search error
|
||||
|
||||
- Problem Description
|
||||
|
||||
@ -485,11 +483,11 @@ USE_AVX2=OFF sh build.sh
|
||||
> 2022-04-22 16:21:44,092 ERROR (MASTER 172.28.7.231_9010_1650606822109(-1)|1) [BDBJEJournal.open():306] catch an exception when setup bdb environment. will exit.
|
||||
> com.sleepycat.je.DiskLimitException: (JE 18.3.12) Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited: maxDiskLimit=0 freeDiskLimit=5,368,709,120 adjustedMaxDiskLimit=0 maxDiskOverage=0 freeDiskShortage=1,536,552,960 diskFreeSpace =3,832,156,160 availableLogSize=-1,536,552,960 totalLogSize=4,665 activeLogSize=4,665 reservedLogSize=0 protectedLogSize=0 protectedLogSizeMap={}
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
The location of the FE is migrated, the hard disk information stored in the metadata cannot be matched, or the hard disk is damaged or not mounted
|
||||
FE has been migrated to another location, which doesn't match the hard disk information stored in the metadata; or the hard disk is damaged or not mounted
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
- Check if the hard disk is normal, initialized and mounted correctly
|
||||
- Fix FE metadata
|
||||
@ -499,53 +497,51 @@ USE_AVX2=OFF sh build.sh
|
||||
|
||||
- Problem Description
|
||||
|
||||
- A file not found error occurred during compilation, and the error is as follows
|
||||
- A "file not found" error occurrs during compilation:
|
||||
|
||||
> Couldn't find pkg.m4 from pkg-config. Install the appropriate package for your distribution or set ACLOCAL_PATH to the directory containing pkg.m4.
|
||||
|
||||
- By looking up the above log, it is found that there is a problem with the compilation of the third-party library `libxml2`
|
||||
- Cause
|
||||
|
||||
- problem causes
|
||||
There is something wrong with the compilation of the third-party library `libxml2` .
|
||||
|
||||
`libxml2` tripartite library compilation error, pkg.m4 file not found
|
||||
***Possible Reasons:***
|
||||
|
||||
***guess:***
|
||||
1. An exception occurs when the Ubuntu system loads the environment variables so the index under the ldb directory is not successfully loaded.
|
||||
2. The retrieval of environment variables during libxml2 compilation fails, so the ldb/aclocal directory is not retrieved.
|
||||
|
||||
- Solution
|
||||
|
||||
1. An exception occurs when the Ubuntu system loads environment variables, resulting in the index under the ldb directory not being successfully loaded
|
||||
2. The retrieval of environment variables during libxml2 compilation fails, resulting in the compilation process not retrieving the ldb/aclocal directory
|
||||
|
||||
- solution
|
||||
|
||||
Copy the `pkg.m4` file in the ldb/aclocal directory to the libxml2/m4 directory, and recompile the third-party library
|
||||
Copy the `pkg.m4` file in the ldb/aclocal directory into the libxml2/m4 directory, and recompile the third-party library.
|
||||
|
||||
```shell
|
||||
cp /opt/software/ldb_toolchain/share/aclocal/pkg.m4 /opt/incubator-doris/thirdparty/src/libxml2-v2.9.10/m4
|
||||
sh /opt/incubator-doris/thirdparty/build-thirdparty.sh
|
||||
````
|
||||
|
||||
|
||||
9. Failed to execute test CURL_HAS_TLS_PROXY
|
||||
|
||||
- Problem Description
|
||||
|
||||
- An error is reported during the compilation process of the three-party package, the error is as follows
|
||||
- An error is reported during the compilation process of the third-party package:
|
||||
|
||||
> -- Performing Test CURL_HAS_TLS_PROXY - Failed
|
||||
> CMake Error at cmake/dependencies.cmake:15 (get_property):
|
||||
> INTERFACE_LIBRARY targets may only have whitelisted properties. The
|
||||
> property "LINK_LIBRARIES_ALL" is not allowed.
|
||||
|
||||
- After viewing the log, it is found that the internal is due to curl `No such file or directory`
|
||||
- The log shows: curl `No such file or directory`
|
||||
|
||||
> fatal error: curl/curl.h: No such file or directory
|
||||
> 2 | #include <curl/curl.h>
|
||||
> compilation terminated.
|
||||
> ninja: build stopped: subcommand failed.
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
There is an error in the compilation environment. After checking the gcc version, it is found that it is the 9.3.0 version that comes with the system, so it is not compiled with ldb, and the ldb environment variable needs to be set
|
||||
There is an error in the compilation environment. The gcc is of version 9.3.0 that comes with the system, so it is not compiled with ldb, so you need to configure the ldb environment variable.
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
Configure ldb environment variables
|
||||
|
||||
@ -554,21 +550,21 @@ USE_AVX2=OFF sh build.sh
|
||||
vim /etc/profile.d/ldb.sh
|
||||
export LDB_HOME=/opt/software/ldb_toolchain
|
||||
export PATH=$LDB_HOME/bin:$PATH
|
||||
# save and exit and refresh environment variables
|
||||
# Save, exit, and refresh environment variables
|
||||
source /etc/profile.d/ldb.sh
|
||||
# test
|
||||
# Test
|
||||
gcc --version
|
||||
> gcc-11
|
||||
````
|
||||
|
||||
10. Other abnormal problems
|
||||
10. Other problems
|
||||
|
||||
- Problem Description
|
||||
|
||||
If there is an error message of the following components, it will be solved with this solution
|
||||
The follow error prompts are all due to one root cause.
|
||||
|
||||
- bison related
|
||||
1. When installing bison-3.0.4, I get fseterr.c error
|
||||
1. fseterr.c error when installing bison-3.0.4
|
||||
- flex related
|
||||
1. flex command not found
|
||||
- cmake related
|
||||
@ -583,14 +579,13 @@ USE_AVX2=OFF sh build.sh
|
||||
- gcc related
|
||||
1. GCC version requires 11+
|
||||
|
||||
- problem causes
|
||||
- Cause
|
||||
|
||||
Not compiled with Ldb-Toolschain
|
||||
Not compiled with ldb-toolschain
|
||||
|
||||
- solution
|
||||
- Solution
|
||||
|
||||
- Check if the Ldb-Toolschain environment variable is configured
|
||||
- Check if the ldb-toolschain environment variable is configured
|
||||
- Check if gcc version is `gcc-11`
|
||||
- Delete the ldb directory after the `ldb-toolschain.sh` script is executed, re-execute and configure the environment variables, and verify the gcc version
|
||||
|
||||
|
||||
|
||||
@ -24,34 +24,34 @@ specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Compilation on macOS
|
||||
# Compile With macOS
|
||||
|
||||
This post introduces how to compile from source on macOS (both x86_64 and arm64).
|
||||
This topic is about how to compile Doris from source with macOS (both x86_64 and arm64).
|
||||
|
||||
## Prerequisites
|
||||
## Environment Requirements
|
||||
|
||||
1. macOS 12 (Monterey) or newer（_**both Intel chip and Apple Silicon chip are supported**_）
|
||||
1. macOS 12 (Monterey) or newer（_**both Intel chip and Apple Silicon chips are supported**_）
|
||||
2. Apple Clang 13 or newer（the latest version is recommended）
|
||||
3. [Homebrew](https://brew.sh/)
|
||||
|
||||
## Steps
|
||||
|
||||
1. Use [Homebrew](https://brew.sh/) to install tools
|
||||
1. Use [Homebrew](https://brew.sh/) to install dependencies.
|
||||
```shell
|
||||
brew install automake autoconf libtool pkg-config texinfo coreutils gnu-getopt \
|
||||
python cmake ninja ccache bison byacc gettext wget pcre maven openjdk@11 npm
|
||||
```
|
||||
|
||||
2. Compile from source
|
||||
2. Compile from source.
|
||||
```shell
|
||||
bash build.sh
|
||||
```
|
||||
|
||||
## The Third Party Libraries
|
||||
## Third-Party Libraries
|
||||
|
||||
1. [Apache Doris Third Party Prebuilt](https://github.com/apache/doris-thirdparty/releases/tag/automation) page contains all sources of the third party libraries, users can download [doris-thirdparty-source.tgz](https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-source.tgz) to achieve the sources.
|
||||
1. The [Apache Doris Third Party Prebuilt](https://github.com/apache/doris-thirdparty/releases/tag/automation) page contains the source code of all third-party libraries. You can download [doris-thirdparty-source.tgz](https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-source.tgz) to obtain them.
|
||||
|
||||
2. If the macOS is powered by _**Intel**_ chip，users can download the _**prebuilt**_ third party libraries [doris-thirdparty-prebuilt-darwin-x86_64.tar.xz](https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-prebuilt-darwin-x86_64.tar.xz) from [Apache Doris Third Party Prebuilt](https://github.com/apache/doris-thirdparty/releases/tag/automation) page and omit the building process for the third party libraries. Please refer to the following commands.
|
||||
2. If your macOS is powered by _**Intel**_ chips, you can download the _**precompiled**_ third party library [doris-thirdparty-prebuilt-darwin-x86_64.tar.xz](https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-prebuilt-darwin-x86_64.tar.xz) from the [Apache Doris Third Party Prebuilt](https://github.com/apache/doris-thirdparty/releases/tag/automation) page. You may refer to the following commands:
|
||||
```shell
|
||||
cd thirdparty
|
||||
curl -L https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-prebuilt-darwin-x86_64.tar.xz \
|
||||
@ -61,11 +61,11 @@ This post introduces how to compile from source on macOS (both x86_64 and arm64)
|
||||
|
||||
## Start-up
|
||||
|
||||
1. Set `file descriptors` up（_**NOTICE: This way is only available in current terminal session. Users who close the current session and open a new terminal session should set the variable again**_）。
|
||||
1. Set `file descriptors` （_**NOTICE: If you have closed the current session, you need to set this variable again**_）。
|
||||
```shell
|
||||
ulimit -n 65536
|
||||
```
|
||||
Users can also write the configuration to initialization files and don't need to set the variable again when opening a new terminal session.
|
||||
You can also write this configuration into the initialization files so you don't need to set the variables again when opening a new terminal session.
|
||||
```shell
|
||||
# bash
|
||||
echo 'ulimit -n 65536' >>~/.bashrc
|
||||
@ -73,7 +73,7 @@ This post introduces how to compile from source on macOS (both x86_64 and arm64)
|
||||
# zsh
|
||||
echo 'ulimit -n 65536' >>~/.zshrc
|
||||
```
|
||||
Check the configuration by executing the following command.
|
||||
Check if the configuration works by executing the following command.
|
||||
```shell
|
||||
$ ulimit -n
|
||||
65536
|
||||
@ -93,8 +93,9 @@ This post introduces how to compile from source on macOS (both x86_64 and arm64)
|
||||
|
||||
## FAQ
|
||||
|
||||
### Fail to start BE up，errors in log: `fail to open StorageEngine, res=file descriptors limit is too small`
|
||||
Please refer to the foregoing contents about `file descriptors`.
|
||||
Fail to start BE up. The log shows: `fail to open StorageEngine, res=file descriptors limit is too small`
|
||||
|
||||
### Java version
|
||||
To fix this, please refer to the "Start-up" section above and reset `file descriptors`.
|
||||
|
||||
### Java Version
|
||||
Java 11 is recommended.
|
||||
|
||||
@ -24,20 +24,20 @@ specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Compile on Windows platform
|
||||
# Compile With Windows
|
||||
|
||||
This article describes how to compile the source code on the Windows platform
|
||||
This topic is about how to compile Doris from source with Windows.
|
||||
|
||||
## Environmental requirements
|
||||
## Environment Requirements
|
||||
|
||||
1. Available in Windows 11 or Windows 10, Version 1903, Build 18362 or later
|
||||
2. WSL2 can be used normally, and the steps to start WSL2 will not be repeated here
|
||||
1. Windows 11 or Windows 10, Version 1903, Build 18362 or newer
|
||||
2. Normal-functioning WSL2
|
||||
|
||||
## Compilation steps
|
||||
## Steps
|
||||
|
||||
1. Install the Oracle Linux 7.9 distribution from the Microsoft Store
|
||||
|
||||
> You can also install other desired distros via Docker images or Github installs
|
||||
> You can also install other distros you want via Docker images or Github installs
|
||||
|
||||
2. Open CMD and specify the identity to run WSL2
|
||||
|
||||
@ -48,17 +48,17 @@ This article describes how to compile the source code on the Windows platform
|
||||
3. Install dependencies
|
||||
|
||||
```shell
|
||||
# install required system packages
|
||||
# Install required system packages
|
||||
sudo yum install -y byacc patch automake libtool make which file ncurses-devel gettext-devel unzip bzip2 zip util-linux wget git python2
|
||||
|
||||
# install autoconf-2.69
|
||||
|
||||
# Install autoconf-2.69
|
||||
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz && \
|
||||
tar zxf autoconf-2.69.tar.gz && \
|
||||
cd autoconf-2.69 && \
|
||||
./configure && \
|
||||
make && \
|
||||
make install
|
||||
|
||||
|
||||
# install bison-3.0.4
|
||||
wget http://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.gz && \
|
||||
tar xzf bison-3.0.4.tar.gz && \
|
||||
@ -89,6 +89,6 @@ This article describes how to compile the source code on the Windows platform
|
||||
cd doris
|
||||
sh build.sh
|
||||
```
|
||||
## Precautions
|
||||
## Note
|
||||
|
||||
The default WSL2 release version data storage drive letter is C drive, if necessary, switch the storage drive letter in advance to prevent the system drive letter from being full
|
||||
The default data storage drive letter of WSL2 distribution is the C drive. If neccessary, you can change that to prevent the system drive letter from getting full.
|
||||
|
||||
@ -24,15 +24,15 @@ specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Compiling with LDB toolchain
|
||||
# Compile With ldb-toolchain
|
||||
|
||||
This document describes how to compile Doris using the LDB toolchain. This method is currently used as a supplement to the Docker compilation method to facilitate developers and users without a Docker environment to compile Doris source code.
|
||||
This topic is about how to compile Doris using the LDB toolchain. This method is an alternative to the Docker method so developers and users without a Docker environment can compile Doris from source.
|
||||
|
||||
> You can still compile the latest code using the Docker development image: `apache/doris:build-env-ldb-toolchain-latest`
|
||||
|
||||
> Thanks to [Amos Bird](https://github.com/amosbird) for this contribution.
|
||||
> Special thanks to [Amos Bird](https://github.com/amosbird) for the contribution.
|
||||
|
||||
## Prepare the environment
|
||||
## Prepare the Environment
|
||||
|
||||
This works for most Linux distributions (CentOS, Ubuntu, etc.).
|
||||
|
||||
@ -42,15 +42,15 @@ This works for most Linux distributions (CentOS, Ubuntu, etc.).
|
||||
|
||||
> For more information, you can visit [https://github.com/amosbird/ldb_toolchain_gen](https://github.com/amosbird/ldb_toolchain_gen)
|
||||
|
||||
2. Execute the following command to generate the ldb toolchain
|
||||
2. Execute the following command to generate the ldb toolchain.
|
||||
|
||||
```
|
||||
sh ldb_toolchain_gen.sh /path/to/ldb_toolchain/
|
||||
```
|
||||
|
||||
where `/path/to/ldb_toolchain/` is the directory where the toolchain is installed.
|
||||
`/path/to/ldb_toolchain/` is the directory where the toolchain is installed.
|
||||
|
||||
After successful execution, the following directory structure will be created under `/path/to/ldb_toolchain/`.
|
||||
After execution, the following directory structure will be created under `/path/to/ldb_toolchain/`.
|
||||
|
||||
```
|
||||
├── bin
|
||||
@ -61,13 +61,13 @@ This works for most Linux distributions (CentOS, Ubuntu, etc.).
|
||||
└── usr
|
||||
```
|
||||
|
||||
3. Download and install other compiled components
|
||||
3. Download and install other compilation packages
|
||||
|
||||
1. [Java8](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/jdk-8u131-linux-x64.tar.gz)
|
||||
2. [Apache Maven 3.6.3](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/apache-maven-3.6.3-bin.tar.gz)
|
||||
3. [Node v12.13.0](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v12.13.0-linux-x64.tar.gz)
|
||||
|
||||
If your environment is somehow minimal, additional packages should be installed before compiling Doris. The following instructions describe how to setup a minimal CentOS 6 box to compile Doris. Other linux distros should be similar.
|
||||
Different Linux distributions might contain different packages, so you may need to install additional packages. The following instructions describe how to set up a minimal CentOS 6 box to compile Doris. It should work similarly for other Linux distros.
|
||||
|
||||
```
|
||||
# install required system packages
|
||||
@ -96,7 +96,7 @@ This works for most Linux distributions (CentOS, Ubuntu, etc.).
|
||||
git clone https://github.com/apache/doris.git
|
||||
```
|
||||
|
||||
After downloading, go to the Doris source directory, create the `custom_env.sh`, file, and set the PATH environment variable, e.g.
|
||||
After downloading, create the `custom_env.sh`, file under the Doris source directory, and set the PATH environment variable:
|
||||
|
||||
```
|
||||
export JAVA_HOME=/path/to/java/
|
||||
@ -106,7 +106,7 @@ This works for most Linux distributions (CentOS, Ubuntu, etc.).
|
||||
export PATH=/path/to/ldb_toolchain/bin:$PATH
|
||||
```
|
||||
|
||||
## Compiling Doris
|
||||
## Compile Doris
|
||||
|
||||
Enter the Doris source code directory and execute:
|
||||
|
||||
@ -114,14 +114,14 @@ Enter the Doris source code directory and execute:
|
||||
$ cat /proc/cpuinfo | grep avx2
|
||||
```
|
||||
|
||||
Check whether the compilation machine supports the avx2 instruction set
|
||||
Check whether the compilation machine supports the avx2 instruction set.
|
||||
|
||||
If it is not supported, use the following command to compile
|
||||
If it is not supported, use the following command to compile:
|
||||
|
||||
```
|
||||
$ USE_AVX2=0 sh build.sh
|
||||
```
|
||||
|
||||
If supported, execute `sh build.sh` directly
|
||||
If supported, execute `sh build.sh` directly.
|
||||
|
||||
This script will compile the third-party libraries first and then the Doris components (FE, BE) later. The compiled output is in the `output/` directory.
|
||||
This script will compile the third-party libraries first and then the Doris components (FE, BE) later. The compiled output will be in the `output/` directory.
|
||||
|
||||
@ -27,17 +27,17 @@ under the License.
|
||||
|
||||
# Compilation
|
||||
|
||||
This document focuses on how to code Doris through source code.
|
||||
This topic is about how to compile Doris from source.
|
||||
|
||||
## Developing mirror compilation using Docker (recommended)
|
||||
## Compile With Docker Development Image (Recommended)
|
||||
|
||||
### Use off-the-shelf mirrors
|
||||
### Use Off-the-Shelf Image
|
||||
|
||||
1. Download Docker Mirror
|
||||
1. Download Docker image
|
||||
|
||||
`$ docker pull apache/doris:build-env-ldb-toolchain-latest`
|
||||
|
||||
Check mirror download completed:
|
||||
Check if the download is completed
|
||||
|
||||
```
|
||||
$ docker images
|
||||
@ -45,11 +45,11 @@ This document focuses on how to code Doris through source code.
|
||||
apache/doris build-env-ldb-toolchain-latest 49f68cecbc1a 4 days ago 3.76GB
|
||||
```
|
||||
|
||||
> Note1: For different versions of Doris, you need to download the corresponding mirror version. From Apache Doris 0.15 version, the docker image will keep same version number with Doris. For example, you can use `apache/doris:build-env-for-0.15.0` to compile Apache Doris 0.15.0.
|
||||
> Note 1: For different versions of Doris, you need to download the corresponding image version. For Apache Doris 0.15 and above, the corresponding Docker image will have the same version number as Doris. For example, you can use `apache/doris:build-env-for-0.15.0` to compile Apache Doris 0.15.0.
|
||||
>
|
||||
> Node2: `apache/doris:build-env-ldb-toolchain-latest` is for compiling trunk code, and will be updated along with trunk code. View the update time in `docker/README.md`
|
||||
> Note 2: `apache/doris:build-env-ldb-toolchain-latest` is used to compile the latest trunk code. It will keep up with the update of the trunk code. You may view the update time in `docker/README.md`.
|
||||
|
||||
| image version | commit id | release version |
|
||||
| Image Version | commit id | Release Version |
|
||||
|---|---|---|
|
||||
| apache/incubator-doris:build-env | before [ff0dd0d](https://github.com/apache/doris/commit/ff0dd0d2daa588f18b6db56f947e813a56d8ec81) | 0.8.x, 0.9.x |
|
||||
| apache/incubator-doris:build-env-1.1 | [ff0dd0d](https://github.com/apache/doris/commit/ff0dd0d2daa588f18b6db56f947e813a56d8ec81) or later | 0.10.x or later |
|
||||
@ -64,39 +64,40 @@ This document focuses on how to code Doris through source code.
|
||||
| apache/doris:build-env-ldb-toolchain-latest | | trunk |
|
||||
| apache/doris:build-env-ldb-toolchain-no-avx2-latest | | trunk |
|
||||
|
||||
**note**:
|
||||
**Note**:
|
||||
|
||||
> 1. Third-party libraries in images with no-avx2 in their names that can run on CPUs that do not support avx2 instructions. Doris can be compiled with the USE_AVX2=0 option.
|
||||
> 1. Third-party libraries in images with "no-avx2" in their names can run on CPUs that do not support avx2 instructions. Doris can be compiled with the USE_AVX2=0 option.
|
||||
|
||||
> 2. Dev docker image [ChangeLog](https://github.com/apache/doris/blob/master/thirdparty/CHANGELOG.md)
|
||||
|
||||
> 3. Doris version 0.14.0 still uses apache/incubator-doris:build-env-1.2 to compile, and the 0.14.x code will use apache/incubator-doris:build-env-1.3.1.
|
||||
> 3. For Doris 0.14.0, use `apache/incubator-doris:build-env-1.2` to compile; for Doris 0.14.x, use `apache/incubator-doris:build-env-1.3.1` to compile.
|
||||
|
||||
> 4. Since the docker image of build-env-1.3.1 includes both OpenJDK 8 and OpenJDK 11, please confirm the default JDK version with `java -version`. You can also switch versions in the following ways (it is recommended to use JDK8 by default)
|
||||
>
|
||||
> Switch to JDK 8:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-1.8.0-openjdk.x86_64
|
||||
> alternatives --set javac java-1.8.0-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
|
||||
> ```
|
||||
>
|
||||
> Switch to JDK 11:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-11-openjdk.x86_64
|
||||
> alternatives --set javac java-11-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-11
|
||||
> ```
|
||||
> 4. The docker images of build-env-1.3.1 and above include both OpenJDK 8 and OpenJDK 11, please confirm the default JDK version with `java -version`. You can also switch versions as follows. (It is recommended to use JDK8.)
|
||||
>
|
||||
> Switch to JDK 8:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-1.8.0-openjdk.x86_64
|
||||
> alternatives --set javac java-1.8.0-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
|
||||
> ```
|
||||
>
|
||||
> Switch to JDK 11:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-11-openjdk.x86_64
|
||||
> alternatives --set javac java-11-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-11
|
||||
> ```
|
||||
>
|
||||
|
||||
2. Running Mirror
|
||||
2. Run the image
|
||||
|
||||
`$ docker run -it apache/doris:build-env-ldb-toolchain-latest`
|
||||
|
||||
It is recommended to run the container by mounting the local Doris source directory, so that the compiled binary file will be stored in the host machine and will not disappear because the container exits.
|
||||
It is recommended to run the image by mounting the local Doris source directory, so that the compiled binary file will be stored in the host machine and will not disappear because of the exit of the image.
|
||||
|
||||
At the same time, it is recommended to mount the maven `.m2` directory in the mirror to the host directory at the same time to prevent repeated downloading of maven's dependent libraries each time the compilation is started.
|
||||
Meanwhile, it is recommended to mount the maven `.m2` directory in the image to the host directory to prevent repeated downloading of maven's dependent libraries each time the compilation is started.
|
||||
|
||||
```
|
||||
$ docker run -it -v /your/local/.m2:/root/.m2 -v /your/local/doris-DORIS-x.x.x-release/:/root/doris-DORIS-x.x.x-release/ apache/doris:build-env-ldb-toolchain-latest
|
||||
@ -104,7 +105,7 @@ This document focuses on how to code Doris through source code.
|
||||
|
||||
3. Download source code
|
||||
|
||||
After starting the mirror, you should be in the container. The Doris source code can be downloaded from the following command (local source directory mounted is not required):
|
||||
After starting the image, you should be in the container. You can download the Doris source code using the following command (If you have mounted the local Doris source directory, you don't need do this):
|
||||
|
||||
```
|
||||
$ git clone https://github.com/apache/doris.git
|
||||
@ -112,51 +113,51 @@ This document focuses on how to code Doris through source code.
|
||||
|
||||
4. Compile Doris
|
||||
|
||||
First run the following command to check whether the compilation machine supports the avx2 instruction set
|
||||
Firstly, run the following command to check whether the compilation machine supports the avx2 instruction set.
|
||||
|
||||
```
|
||||
$ cat /proc/cpuinfo | grep avx2
|
||||
```
|
||||
|
||||
If it is not supported, use the following command to compile
|
||||
If it is not supported, use the following command to compile.
|
||||
|
||||
```
|
||||
$ USE_AVX2=0 sh build.sh
|
||||
```
|
||||
|
||||
If supported, compile directly without adding USE_AVX2=0
|
||||
If supported, use the following command to compile.
|
||||
|
||||
```
|
||||
$ sh build.sh
|
||||
```
|
||||
|
||||
After compilation, the output files are in the `output/` directory.
|
||||
After compilation, the output files will be in the `output/` directory.
|
||||
|
||||
> **Note:**
|
||||
>
|
||||
> If you are using `build-env-for-0.15.0` or later version for the first time, use the following command when compiling:
|
||||
> If you are using `build-env-for-0.15.0` or the subsequent versions for the first time, use the following command when compiling:
|
||||
>
|
||||
> `sh build.sh --clean --be --fe --ui`
|
||||
>
|
||||
> This is because from build-env-for-0.15.0, we upgraded thrift (0.9 -> 0.13), you need to use the --clean command to force the use of the new version of thrift to generate code files, otherwise incompatible code will appear.
|
||||
> This is we have upgraded the thrift (0.9 -> 0.13) for `build-env-for-0.15.0` and the subsequent versions. That means you need to use the --clean command to force the use of the new version of thrift to generate code files, otherwise it will result in code incompatibility.
|
||||
|
||||
After compilation, the output file is in the `output/` directory.
|
||||
After compilation, the output file will be in the `output/` directory.
|
||||
|
||||
### Self-compiling Development Environment Mirror
|
||||
### Self-Compile Development Environment Image
|
||||
|
||||
You can also create a Doris development environment mirror yourself, referring specifically to the `docker/README.md` file.
|
||||
You can create a Doris development environment image yourself. Check [docker/README.md](https://github.com/apache/doris/blob/master/docker/README.md) for details.
|
||||
|
||||
|
||||
## Direct Compilation (Ubuntu)
|
||||
|
||||
You can try to compile Doris directly in your own Linux environment.
|
||||
You can compile Doris directly in your own Linux environment.
|
||||
|
||||
1. System Dependence
|
||||
* Before commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) will use the dependencies as follows:
|
||||
1. System Dependencies
|
||||
* System dependencies before commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) are as follows:
|
||||
|
||||
`GCC 7.3+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.11+ Bison 3.0+`
|
||||
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies:
|
||||
|
||||
```
|
||||
sudo apt-get install build-essential openjdk-8-jdk maven byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python autopoint pkg-config
|
||||
@ -164,15 +165,15 @@ You can try to compile Doris directly in your own Linux environment.
|
||||
apt-get update && apt-get install cmake
|
||||
```
|
||||
|
||||
If you are using CentOS you can use the following command to install the dependencies
|
||||
If you are using CentOS, you can use the following command to install the dependencies:
|
||||
|
||||
`sudo yum groupinstall 'Development Tools' && sudo yum install maven cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk`
|
||||
|
||||
* After commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) will use the dependencies as follows:
|
||||
* System dependencies after commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) are as follows:
|
||||
|
||||
`GCC 10+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.19.2+ Bison 3.0+`
|
||||
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies:
|
||||
|
||||
```
|
||||
sudo apt install build-essential openjdk-8-jdk maven cmake byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python
|
||||
@ -185,39 +186,39 @@ You can try to compile Doris directly in your own Linux environment.
|
||||
|
||||
2. Compile Doris
|
||||
|
||||
Compiling with the Docker development image, check whether the avx2 instruction is supported before compiling
|
||||
This is the same as compiling with the Docker development image. Before compiling, you need to check whether the avx2 instruction is supported.
|
||||
|
||||
```
|
||||
$ cat /proc/cpuinfo | grep avx2
|
||||
```
|
||||
|
||||
If supported, use the following command to compile
|
||||
If it is supported, use the following command to compile:
|
||||
|
||||
```
|
||||
$ sh build.sh
|
||||
```
|
||||
|
||||
If it is not supported, you need to add USE_AVX2=0
|
||||
If not supported, use the following command to compile:
|
||||
|
||||
```
|
||||
$ USE_AVX2=0 sh build.sh
|
||||
```
|
||||
|
||||
After compilation, the output files are in the `output/` directory.
|
||||
After compilation, the output files will be in the `output/` directory.
|
||||
|
||||
## FAQ
|
||||
|
||||
1. `Could not transfer artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh from/to xxx`
|
||||
|
||||
If you encounter the above error, please refer to [PR #4769](https://github.com/apache/doris/pull/4769/files) to modify the cloudera-related repo configuration in `fe/pom.xml`.
|
||||
If you encounter the above error, please refer to [PR #4769](https://github.com/apache/doris/pull/4769/files) and modify the cloudera-related repo configuration in `fe/pom.xml`.
|
||||
|
||||
2. The third party relies on download connection errors, failures, etc.
|
||||
2. Invalid download links of third-party dependencies
|
||||
|
||||
The download links of the third-party libraries that Doris relies on are all in the `thirdparty/vars.sh` file. Over time, some download connections may fail. If you encounter this situation. It can be solved in the following two ways:
|
||||
The download links of the third-party libraries that Doris relies on are all in the `thirdparty/vars.sh` file. Over time, some download links may fail. If you encounter this situation. It can be solved in the following two ways:
|
||||
|
||||
1. Manually modify the `thirdparty/vars.sh` file
|
||||
|
||||
Manually modify the problematic download connection and the corresponding MD5 value.
|
||||
Manually modify the problematic download links and the corresponding MD5 value.
|
||||
|
||||
2. Use a third-party download warehouse:
|
||||
|
||||
@ -228,21 +229,27 @@ You can try to compile Doris directly in your own Linux environment.
|
||||
|
||||
REPOSITORY_URL contains all third-party library source code packages and their historical versions.
|
||||
|
||||
## Special statement
|
||||
3. `fatal error: Killed signal terminated program ...`
|
||||
|
||||
Starting from version 0.13, the dependency on the two third-party libraries [1] and [2] will be removed in the default compiled output. These two third-party libraries are under [GNU General Public License V3](https://www.gnu.org/licenses/gpl-3.0.en.html). This license is incompatible with [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), so it should not appear in the Apache release by default.
|
||||
If you encounter this error, the possible reason is not enough memory allocated to the image. (The default memory allocation for Docker is 2 GB, and the peak memory usage during the compilation might exceed that.)
|
||||
|
||||
You can fix this by increasing the memory allocation for the image, 4 GB ~ 8 GB, for example.
|
||||
|
||||
## Special Statement
|
||||
|
||||
Starting from version 0.13, the dependency on the two third-party libraries [1] and [2] will be removed in the default compiled output. These two third-party libraries are under [GNU General Public License V3](https://www.gnu.org/licenses/gpl-3.0.en.html). This license is incompatible with [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), so it will not appear in the Apache release by default.
|
||||
|
||||
Remove library [1] will result in the inability to access MySQL external tables. The feature of accessing MySQL external tables will be implemented through `UnixODBC` in future release version.
|
||||
|
||||
Remove library [2] will cause some data written in earlier versions (before version 0.8) to be unable to read. Because the data in the earlier version was compressed using the LZO algorithm, in later versions, it has been changed to the LZ4 compression algorithm. We will provide tools to detect and convert this part of the data in the future.
|
||||
|
||||
If required, users can continue to use these two dependent libraries. If you want to use it, you need to add the following options when compiling:
|
||||
If required, you can continue to use these two libraries by adding the following option when compiling:
|
||||
|
||||
```
|
||||
WITH_MYSQL=1 WITH_LZO=1 sh build.sh
|
||||
```
|
||||
|
||||
Note that when users rely on these two third-party libraries, Doris is not used under the Apache License 2.0 by default. Please pay attention to the GPL related agreements.
|
||||
Note that if you use these two third-party libraries, that means you choose not to use Doris under the Apache License 2.0, and you might need to pay attention to the GPL-related agreements.
|
||||
|
||||
* [1] mysql-5.7.18
|
||||
* [2] lzo-2.10
|
||||
|
||||
Reference in New Issue
Block a user