45e42bd003
Because the meta version is only be used in catalog saving and loading. So currently this version is a field of Catalog class. And we can get this version only by calling Catalog.getCurrentCatalogJournalVersion(). But in restore process, we need to read the meta data which is saved with a specified meta version. So we need a flexible way to read a meta data using a specified meta version, not only the version from Catalog. So we create a new class called MetaContext. Currently it only has one field, 'journalVersion', to save the current journal version. And it is a thread local variable, so that we can create a MetaContext anywhere we want, and setting the 'journalVersion' which we want to use for reading meta. Currently, there are 4 threads which is related to meta data saving and loading. The Frontend starting thread, which will call Catalog.initialize() to load the image. the Frontend state listener thread, which will listen the state changing, and call transferToMaster() or transferToNonMaster(). Edit log replayed thread, which is created when calling transferToNonMaster(). It will replay edit log Checkpoint thread, which is created when calling transferToMaster(). It will do the checkpoint periodically. Notice that we get the 'current meta version' only when 'READING' the meta (not WRITING). So we only need to take care of all 'READING' threads. We create MetaContext thread local variable for these 4 threads, and thread 2,3,4's meta context inherit from thread 1's meta context. Because thread 1 will load the origin image file and get the very first meta version. And we leave the Catalog.getCurrentCatalogJournalVersion()'s name unchanged, just change its content, because we don't want change a lot codes this time. On the other hand, we add the current meta version in backup job info file when doing backup job. So that when restoring from a backup snapshot, we can know which meta version we should use for read the meta. And also , we add a new property "meta_version" for Restore stmt, so that we can specify the meta version used for reading backup meta. It is for those old backup snapshots which do not has meta version saving in backup job info file.
Apache Doris (incubating)
Doris is an MPP-based interactive SQL data warehousing for reporting and analysis. It open-sourced by Baidu.
1. License
2. Technology
Doris mainly integrates the technology of Google Mesa and Apache Impala, and it based on a column-oriented storage engine and can communicate by MySQL client.
3. User cases
Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis.
Doris not only provides batch data loading, but also provides near real-time mini-batch data loading.
Doris also provides high availability, reliability, fault tolerance, and scalability.
The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris (refer to Overview).
4. Compile and install
Currently support Docker environment and Linux OS: Docker (Linux / Windows / Mac), Ubuntu and CentOS.
4.1 For Docker
Step 1: Install Docker
Take CentOS as an example:
yum -y install docker-io
service docker start
Step 2: Create Docker image for complilation
Given your work space is /my/workspace, and you can download Doris docker file as following:
wget https://github.com/apache/incubator-doris/blob/master/docker/Dockerfile -O /my/workspace/Dockerfile
Now build your image, and it may take a long time (40min to 1 hour)
cd /my/workspace && docker build -t doris-dev:v1.0 .
Step 3: Compile and install Doris
Clone Doris source:
git clone https://github.com/apache/incubator-doris.git /path/to/incubator-doris/
Start a container named doris-dev-test, and map /path/to/incubator-doris/ to /var/local/incubator-doris/ in container.
docker run -it --name doris-dev-test -v /path/to/incubator-doris/:/var/local/incubator-doris/ doris-dev:v1.0
Compile Doris source:
sh build.sh
After successfully building, it will install binary files in the directory output/.
4.2 For Linux
Prerequisites
GCC 5.3.1+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.4.3+
- For Ubuntu:
sudo apt-get install g++ cmake zip byacc flex automake libtool binutils-dev libiberty-dev bison python2.7 libncurses5-dev
sudo updatedb
- For CentOS:
sudo yum install gcc-c++ libstdc++-static cmake byacc flex automake libtool binutils-devel bison ncurses-devel
sudo updatedb
If your GCC version is less than 5.3.1, you can run:
sudo yum install devtoolset-4-toolchain -y
and then, set the path of GCC (e.g /opt/rh/devtoolset-4/root/usr/bin) to the environment variable PATH.
Compile and install
Run following script, it will compile thirdparty libraries and build whole Doris.
sh build.sh
After successfully building, it will install binary files in the directory output/.
5. Reporting Issues
If you find any bugs, please file a GitHub issue.
6. Links
- Doris official site - http://doris.incubator.apache.org
- User Manual (GitHub Wiki) - https://github.com/apache/incubator-doris/wiki
- Developer Mailing list - Subscribe to dev@doris.incubator.apache.org to discuss with us.
- Gitter channel - https://gitter.im/apache-doris/Lobby - Online chat room with Doris developers.
- Overview - https://github.com/apache/incubator-doris/wiki/Doris-Overview
- Compile and install - https://github.com/apache/incubator-doris/wiki/Doris-Install
- Getting start - https://github.com/apache/incubator-doris/wiki/Getting-start
- Deploy and Upgrade - https://github.com/apache/incubator-doris/wiki/Doris-Deploy-%26-Upgrade
- User Manual - https://github.com/apache/incubator-doris/wiki/Doris-Create%2C-Load-and-Delete
- FAQs - https://github.com/apache/incubator-doris/wiki/Doris-FAQ