Hu Yanjun
@ -27,17 +27,17 @@ under the License.
|
||||
|
||||
# Compilation
|
||||
|
||||
This document focuses on how to code Doris through source code.
|
||||
This topic is about how to compile Doris from source.
|
||||
|
||||
## Developing mirror compilation using Docker (recommended)
|
||||
## Compile With Docker Development Image (Recommended)
|
||||
|
||||
### Use off-the-shelf mirrors
|
||||
### Use Off-the-Shelf Image
|
||||
|
||||
1. Download Docker Mirror
|
||||
1. Download Docker image
|
||||
|
||||
`$ docker pull apache/doris:build-env-ldb-toolchain-latest`
|
||||
|
||||
Check mirror download completed:
|
||||
Check if the download is completed
|
||||
|
||||
```
|
||||
$ docker images
|
||||
@ -45,11 +45,11 @@ This document focuses on how to code Doris through source code.
|
||||
apache/doris build-env-ldb-toolchain-latest 49f68cecbc1a 4 days ago 3.76GB
|
||||
```
|
||||
|
||||
> Note1: For different versions of Doris, you need to download the corresponding mirror version. From Apache Doris 0.15 version, the docker image will keep same version number with Doris. For example, you can use `apache/doris:build-env-for-0.15.0` to compile Apache Doris 0.15.0.
|
||||
> Note 1: For different versions of Doris, you need to download the corresponding image version. For Apache Doris 0.15 and above, the corresponding Docker image will have the same version number as Doris. For example, you can use `apache/doris:build-env-for-0.15.0` to compile Apache Doris 0.15.0.
|
||||
>
|
||||
> Node2: `apache/doris:build-env-ldb-toolchain-latest` is for compiling trunk code, and will be updated along with trunk code. View the update time in `docker/README.md`
|
||||
> Note 2: `apache/doris:build-env-ldb-toolchain-latest` is used to compile the latest trunk code. It will keep up with the update of the trunk code. You may view the update time in `docker/README.md`.
|
||||
|
||||
| image version | commit id | release version |
|
||||
| Image Version | commit id | Release Version |
|
||||
|---|---|---|
|
||||
| apache/incubator-doris:build-env | before [ff0dd0d](https://github.com/apache/doris/commit/ff0dd0d2daa588f18b6db56f947e813a56d8ec81) | 0.8.x, 0.9.x |
|
||||
| apache/incubator-doris:build-env-1.1 | [ff0dd0d](https://github.com/apache/doris/commit/ff0dd0d2daa588f18b6db56f947e813a56d8ec81) or later | 0.10.x or later |
|
||||
@ -64,39 +64,40 @@ This document focuses on how to code Doris through source code.
|
||||
| apache/doris:build-env-ldb-toolchain-latest | | trunk |
|
||||
| apache/doris:build-env-ldb-toolchain-no-avx2-latest | | trunk |
|
||||
|
||||
**note**:
|
||||
**Note**:
|
||||
|
||||
> 1. Third-party libraries in images with no-avx2 in their names that can run on CPUs that do not support avx2 instructions. Doris can be compiled with the USE_AVX2=0 option.
|
||||
> 1. Third-party libraries in images with "no-avx2" in their names can run on CPUs that do not support avx2 instructions. Doris can be compiled with the USE_AVX2=0 option.
|
||||
|
||||
> 2. Dev docker image [ChangeLog](https://github.com/apache/doris/blob/master/thirdparty/CHANGELOG.md)
|
||||
|
||||
> 3. Doris version 0.14.0 still uses apache/incubator-doris:build-env-1.2 to compile, and the 0.14.x code will use apache/incubator-doris:build-env-1.3.1.
|
||||
> 3. For Doris 0.14.0, use `apache/incubator-doris:build-env-1.2` to compile; for Doris 0.14.x, use `apache/incubator-doris:build-env-1.3.1` to compile.
|
||||
|
||||
> 4. Since the docker image of build-env-1.3.1 includes both OpenJDK 8 and OpenJDK 11, please confirm the default JDK version with `java -version`. You can also switch versions in the following ways (it is recommended to use JDK8 by default)
|
||||
>
|
||||
> Switch to JDK 8:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-1.8.0-openjdk.x86_64
|
||||
> alternatives --set javac java-1.8.0-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
|
||||
> ```
|
||||
>
|
||||
> Switch to JDK 11:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-11-openjdk.x86_64
|
||||
> alternatives --set javac java-11-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-11
|
||||
> ```
|
||||
> 4. The docker images of build-env-1.3.1 and above include both OpenJDK 8 and OpenJDK 11, please confirm the default JDK version with `java -version`. You can also switch versions as follows. (It is recommended to use JDK8.)
|
||||
>
|
||||
> Switch to JDK 8:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-1.8.0-openjdk.x86_64
|
||||
> alternatives --set javac java-1.8.0-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
|
||||
> ```
|
||||
>
|
||||
> Switch to JDK 11:
|
||||
>
|
||||
> ```
|
||||
> alternatives --set java java-11-openjdk.x86_64
|
||||
> alternatives --set javac java-11-openjdk.x86_64
|
||||
> export JAVA_HOME=/usr/lib/jvm/java-11
|
||||
> ```
|
||||
>
|
||||
|
||||
2. Running Mirror
|
||||
2. Run the image
|
||||
|
||||
`$ docker run -it apache/doris:build-env-ldb-toolchain-latest`
|
||||
|
||||
It is recommended to run the container by mounting the local Doris source directory, so that the compiled binary file will be stored in the host machine and will not disappear because the container exits.
|
||||
It is recommended to run the image by mounting the local Doris source directory, so that the compiled binary file will be stored in the host machine and will not disappear because of the exit of the image.
|
||||
|
||||
At the same time, it is recommended to mount the maven `.m2` directory in the mirror to the host directory at the same time to prevent repeated downloading of maven's dependent libraries each time the compilation is started.
|
||||
Meanwhile, it is recommended to mount the maven `.m2` directory in the image to the host directory to prevent repeated downloading of maven's dependent libraries each time the compilation is started.
|
||||
|
||||
```
|
||||
$ docker run -it -v /your/local/.m2:/root/.m2 -v /your/local/doris-DORIS-x.x.x-release/:/root/doris-DORIS-x.x.x-release/ apache/doris:build-env-ldb-toolchain-latest
|
||||
@ -104,7 +105,7 @@ This document focuses on how to code Doris through source code.
|
||||
|
||||
3. Download source code
|
||||
|
||||
After starting the mirror, you should be in the container. The Doris source code can be downloaded from the following command (local source directory mounted is not required):
|
||||
After starting the image, you should be in the container. You can download the Doris source code using the following command (If you have mounted the local Doris source directory, you don't need do this):
|
||||
|
||||
```
|
||||
$ git clone https://github.com/apache/doris.git
|
||||
@ -112,51 +113,51 @@ This document focuses on how to code Doris through source code.
|
||||
|
||||
4. Compile Doris
|
||||
|
||||
First run the following command to check whether the compilation machine supports the avx2 instruction set
|
||||
Firstly, run the following command to check whether the compilation machine supports the avx2 instruction set.
|
||||
|
||||
```
|
||||
$ cat /proc/cpuinfo | grep avx2
|
||||
```
|
||||
|
||||
If it is not supported, use the following command to compile
|
||||
If it is not supported, use the following command to compile.
|
||||
|
||||
```
|
||||
$ USE_AVX2=0 sh build.sh
|
||||
```
|
||||
|
||||
If supported, compile directly without adding USE_AVX2=0
|
||||
If supported, use the following command to compile.
|
||||
|
||||
```
|
||||
$ sh build.sh
|
||||
```
|
||||
|
||||
After compilation, the output files are in the `output/` directory.
|
||||
After compilation, the output files will be in the `output/` directory.
|
||||
|
||||
> **Note:**
|
||||
>
|
||||
> If you are using `build-env-for-0.15.0` or later version for the first time, use the following command when compiling:
|
||||
> If you are using `build-env-for-0.15.0` or the subsequent versions for the first time, use the following command when compiling:
|
||||
>
|
||||
> `sh build.sh --clean --be --fe --ui`
|
||||
>
|
||||
> This is because from build-env-for-0.15.0, we upgraded thrift (0.9 -> 0.13), you need to use the --clean command to force the use of the new version of thrift to generate code files, otherwise incompatible code will appear.
|
||||
> This is we have upgraded the thrift (0.9 -> 0.13) for `build-env-for-0.15.0` and the subsequent versions. That means you need to use the --clean command to force the use of the new version of thrift to generate code files, otherwise it will result in code incompatibility.
|
||||
|
||||
After compilation, the output file is in the `output/` directory.
|
||||
After compilation, the output file will be in the `output/` directory.
|
||||
|
||||
### Self-compiling Development Environment Mirror
|
||||
### Self-Compile Development Environment Image
|
||||
|
||||
You can also create a Doris development environment mirror yourself, referring specifically to the `docker/README.md` file.
|
||||
You can create a Doris development environment image yourself. Check [docker/README.md](https://github.com/apache/doris/blob/master/docker/README.md) for details.
|
||||
|
||||
|
||||
## Direct Compilation (Ubuntu)
|
||||
|
||||
You can try to compile Doris directly in your own Linux environment.
|
||||
You can compile Doris directly in your own Linux environment.
|
||||
|
||||
1. System Dependence
|
||||
* Before commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) will use the dependencies as follows:
|
||||
1. System Dependencies
|
||||
* System dependencies before commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) are as follows:
|
||||
|
||||
`GCC 7.3+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.11+ Bison 3.0+`
|
||||
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies:
|
||||
|
||||
```
|
||||
sudo apt-get install build-essential openjdk-8-jdk maven byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python autopoint pkg-config
|
||||
@ -164,15 +165,15 @@ You can try to compile Doris directly in your own Linux environment.
|
||||
apt-get update && apt-get install cmake
|
||||
```
|
||||
|
||||
If you are using CentOS you can use the following command to install the dependencies
|
||||
If you are using CentOS, you can use the following command to install the dependencies:
|
||||
|
||||
`sudo yum groupinstall 'Development Tools' && sudo yum install maven cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk`
|
||||
|
||||
* After commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) will use the dependencies as follows:
|
||||
* System dependencies after commit [ad67dd3](https://github.com/apache/doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) are as follows:
|
||||
|
||||
`GCC 10+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.19.2+ Bison 3.0+`
|
||||
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies
|
||||
If you are using Ubuntu 16.04 or newer, you can use the following command to install the dependencies:
|
||||
|
||||
```
|
||||
sudo apt install build-essential openjdk-8-jdk maven cmake byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python
|
||||
@ -185,39 +186,39 @@ You can try to compile Doris directly in your own Linux environment.
|
||||
|
||||
2. Compile Doris
|
||||
|
||||
Compiling with the Docker development image, check whether the avx2 instruction is supported before compiling
|
||||
This is the same as compiling with the Docker development image. Before compiling, you need to check whether the avx2 instruction is supported.
|
||||
|
||||
```
|
||||
$ cat /proc/cpuinfo | grep avx2
|
||||
```
|
||||
|
||||
If supported, use the following command to compile
|
||||
If it is supported, use the following command to compile:
|
||||
|
||||
```
|
||||
$ sh build.sh
|
||||
```
|
||||
|
||||
If it is not supported, you need to add USE_AVX2=0
|
||||
If not supported, use the following command to compile:
|
||||
|
||||
```
|
||||
$ USE_AVX2=0 sh build.sh
|
||||
```
|
||||
|
||||
After compilation, the output files are in the `output/` directory.
|
||||
After compilation, the output files will be in the `output/` directory.
|
||||
|
||||
## FAQ
|
||||
|
||||
1. `Could not transfer artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh from/to xxx`
|
||||
|
||||
If you encounter the above error, please refer to [PR #4769](https://github.com/apache/doris/pull/4769/files) to modify the cloudera-related repo configuration in `fe/pom.xml`.
|
||||
If you encounter the above error, please refer to [PR #4769](https://github.com/apache/doris/pull/4769/files) and modify the cloudera-related repo configuration in `fe/pom.xml`.
|
||||
|
||||
2. The third party relies on download connection errors, failures, etc.
|
||||
2. Invalid download links of third-party dependencies
|
||||
|
||||
The download links of the third-party libraries that Doris relies on are all in the `thirdparty/vars.sh` file. Over time, some download connections may fail. If you encounter this situation. It can be solved in the following two ways:
|
||||
The download links of the third-party libraries that Doris relies on are all in the `thirdparty/vars.sh` file. Over time, some download links may fail. If you encounter this situation. It can be solved in the following two ways:
|
||||
|
||||
1. Manually modify the `thirdparty/vars.sh` file
|
||||
|
||||
Manually modify the problematic download connection and the corresponding MD5 value.
|
||||
Manually modify the problematic download links and the corresponding MD5 value.
|
||||
|
||||
2. Use a third-party download warehouse:
|
||||
|
||||
@ -228,21 +229,27 @@ You can try to compile Doris directly in your own Linux environment.
|
||||
|
||||
REPOSITORY_URL contains all third-party library source code packages and their historical versions.
|
||||
|
||||
## Special statement
|
||||
3. `fatal error: Killed signal terminated program ...`
|
||||
|
||||
Starting from version 0.13, the dependency on the two third-party libraries [1] and [2] will be removed in the default compiled output. These two third-party libraries are under [GNU General Public License V3](https://www.gnu.org/licenses/gpl-3.0.en.html). This license is incompatible with [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), so it should not appear in the Apache release by default.
|
||||
If you encounter this error, the possible reason is not enough memory allocated to the image. (The default memory allocation for Docker is 2 GB, and the peak memory usage during the compilation might exceed that.)
|
||||
|
||||
You can fix this by increasing the memory allocation for the image, 4 GB ~ 8 GB, for example.
|
||||
|
||||
## Special Statement
|
||||
|
||||
Starting from version 0.13, the dependency on the two third-party libraries [1] and [2] will be removed in the default compiled output. These two third-party libraries are under [GNU General Public License V3](https://www.gnu.org/licenses/gpl-3.0.en.html). This license is incompatible with [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), so it will not appear in the Apache release by default.
|
||||
|
||||
Remove library [1] will result in the inability to access MySQL external tables. The feature of accessing MySQL external tables will be implemented through `UnixODBC` in future release version.
|
||||
|
||||
Remove library [2] will cause some data written in earlier versions (before version 0.8) to be unable to read. Because the data in the earlier version was compressed using the LZO algorithm, in later versions, it has been changed to the LZ4 compression algorithm. We will provide tools to detect and convert this part of the data in the future.
|
||||
|
||||
If required, users can continue to use these two dependent libraries. If you want to use it, you need to add the following options when compiling:
|
||||
If required, you can continue to use these two libraries by adding the following option when compiling:
|
||||
|
||||
```
|
||||
WITH_MYSQL=1 WITH_LZO=1 sh build.sh
|
||||
```
|
||||
|
||||
Note that when users rely on these two third-party libraries, Doris is not used under the Apache License 2.0 by default. Please pay attention to the GPL related agreements.
|
||||
Note that if you use these two third-party libraries, that means you choose not to use Doris under the Apache License 2.0, and you might need to pay attention to the GPL-related agreements.
|
||||
|
||||
* [1] mysql-5.7.18
|
||||
* [2] lzo-2.10
|
||||
|
||||
Reference in New Issue
Block a user