bb7ae97845
Ref https://github.com/apache/incubator-doris/issues/3566 Introduce trace utility from Kudu to BE. This utility has been widely used in Kudu, Impala also import this trace utility. This trace util is used for tracing each phases in a thread, and can be dumped to string to see each phases' time cost and diagnose which phase cost more time. This util store a Trace object as a threadlocal variable, we can add trace entries which record the current file name, line number, user specified symbols and timestamp to this object, and it's able to add some counters to this Trace object. And then, it can be dumped to human readable string. There are some helpful macros defined in trace.h, here is a simple example for usage: ``` scoped_refptr<Trace> t1(new Trace); // New 2 traces scoped_refptr<Trace> t2(new Trace); t1->AddChildTrace("child_trace", t2.get()); // t1 add t2 as a child named "child_trace" TRACE_TO(t1, "step $0", 1); // Explicitly trace to t1 usleep(10); // ... do some work ADOPT_TRACE(t1.get()); // Explicitly adopt to trace to t1 TRACE("step $0", 2); // Implicitly trace to t1 { // The time spent in this scope is added to counter t1.scope_time_cost TRACE_COUNTER_SCOPE_LATENCY_US("scope_time_cost"); ADOPT_TRACE(t2.get()); // Adopt to trace to t2 for the duration of the current scope TRACE("sub start"); // Implicitly trace to t2 usleep(10); // ... do some work TRACE("sub before loop"); for (int i = 0; i < 10; ++i) { TRACE_COUNTER_INCREMENT("iterate_count", 1); // Increase counter t2.iterate_count MicrosecondsInt64 start_time = GetMonoTimeMicros(); usleep(10); // ... do some work MicrosecondsInt64 end_time = GetMonoTimeMicros(); int64_t dur = end_time - start_time; // t2's simple histogram metric with name prefixed with "lbm_writes" const char* counter = BUCKETED_COUNTER_NAME("lbm_writes", dur); TRACE_COUNTER_INCREMENT(counter, 1); } TRACE("sub after loop"); } TRACE("goodbye $0", "cruel world"); // Automatically restore to trace to t1 std::cout << t1->DumpToString(Trace::INCLUDE_ALL) << std::endl; ``` output looks like: ``` 0514 02:16:07.988054 (+ 0us) trace_test.cpp:76] step 1 0514 02:16:07.988112 (+ 58us) trace_test.cpp:80] step 2 0514 02:16:07.988863 (+ 751us) trace_test.cpp:103] goodbye cruel world Related trace 'child_trace': 0514 02:16:07.988120 (+ 0us) trace_test.cpp:85] sub start 0514 02:16:07.988188 (+ 68us) trace_test.cpp:88] sub before loop 0514 02:16:07.988850 (+ 662us) trace_test.cpp:101] sub after loop Metrics: {"scope_time_cost":744,"child_traces":[["child_trace",{"iterate_count":10,"lbm_writes_lt_1ms":10}]]} ``` Exclude the original source code, this patch do the following work to adapt to Doris: - Rename "kudu" namespace to "doris" - Update some names to the existing function names in Doris, i.g. strings::internal::SubstituteArg::kNoArg -> strings::internal::SubstituteArg::NoArg - Use doris::SpinLock instead of kudu::simple_spinlock which hasn't been imported - Use manual malloc() and free() instead of kudu::Arena which hasn't been imported - Use manual rapidjson::Writer instead of kudu::JsonWriter which hasn't been imported - Remove all TRACE_EVENT related unit tests since TRACE_EVENT is not imported this time - Update CMakeLists.txt
Apache Doris (incubating)
Doris is an MPP-based interactive SQL data warehousing for reporting and analysis. Its original name was Palo, developed in Baidu. After donating it to Apache Software Foundation, it was renamed Doris.
1. License
2. Technology
Doris mainly integrates the technology of Google Mesa and Apache Impala, and it is based on a column-oriented storage engine and can communicate by MySQL client.
3. User cases
Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis.
Doris not only provides batch data loading, but also provides near real-time mini-batch data loading.
Doris also provides high availability, reliability, fault tolerance, and scalability.
The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris (refer to Overview).
4. Compile and install
Currently only supports Docker environment and Linux OS, such as Ubuntu and CentOS.
4.1 Compile in Docker environment (Recommended)
We offer a docker image as a Doris compilation environment. You can compile Doris from source in it and run the output binaries in other Linux environment.
Firstly, you must be install and start docker service.
And then you could build Doris as following steps:
Step1: Pull the docker image with Doris building environment
$ docker pull apachedoris/doris-dev:build-env-1.2
You can check it by listing images, for example:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
apachedoris/doris-dev build-env-1.2 69cf7fff9d10 2 weeks ago 4.12GB
NOTE: You may have to use different images to compile from source.
image version commit id release version apachedoris/doris-dev:build-env before ff0dd0d 0.8.x, 0.9.x apachedoris/doris-dev:build-env-1.1 ff0dd0d or later 0.10.x or 0.11.x apachedoris/doris-dev:build-env-1.2 1648226 or later 0.12.x or later
Step2: Run the Docker image
You can run the image directly:
$ docker run -it apachedoris/doris-dev:build-env-1.2
Or if you want to compile the source located in your local host, you can map the local directory to the image by running:
$ docker run -it -v /your/local/path/incubator-doris-DORIS-x.x.x-release/:/root/incubator-doris-DORIS-x.x.x-release/ apachedoris/doris-dev:build-env-1.2
Step3: Download Doris source
Now you should attached in docker environment.
You can download Doris source by release package or by git clone in image.
(If you already downloaded the source in your local host and map it to the image in Step2, you can skip this step.)
$ wget https://dist.apache.org/repos/dist/dev/incubator/doris/xxx.tar.gz
or
$ git clone https://github.com/apache/incubator-doris.git
Step4: Build Doris
Enter Doris source path and build Doris.
$ sh build.sh
After successfully building, it will install binary files in the directory output/.
4.2 For Linux OS
Prerequisites
You must be install following softwares:
GCC 5.3.1+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.4.3+
After you installed above all, you also must be set them to environment variable PATH and set JAVA_HOME.
If your GCC version is lower than 5.3.1, you can run:
sudo yum install devtoolset-4-toolchain -y
and then, set the path of GCC (e.g /opt/rh/devtoolset-4/root/usr/bin) to the environment variable PATH.
Compile and install
Run the following script, it will compile thirdparty libraries and build whole Doris.
sh build.sh
After successfully building, it will install binary files in the directory output/.
5. License Notice
Some of the third-party dependencies' license are not compatible with Apache 2.0 License. So you may have to disable
some features of Doris to be complied with Apache 2.0 License. Details can be found in thirdparty/LICENSE.txt
6. Reporting Issues
If you find any bugs, please file a GitHub issue.
7. Links
- Doris official site - http://doris.incubator.apache.org
- User Manual (GitHub Wiki) - https://github.com/apache/incubator-doris/wiki
- Developer Mailing list - Subscribe to dev@doris.incubator.apache.org to discuss with us.
- Gitter channel - https://gitter.im/apache-doris/Lobby - Online chat room with Doris developers.
- Overview - https://github.com/apache/incubator-doris/wiki/Doris-Overview
- Compile and install - https://github.com/apache/incubator-doris/wiki/Doris-Install
- Getting start - https://github.com/apache/incubator-doris/wiki/Getting-start
- Deploy and Upgrade - https://github.com/apache/incubator-doris/wiki/Doris-Deploy-%26-Upgrade
- User Manual - https://github.com/apache/incubator-doris/wiki/Doris-Create%2C-Load-and-Delete
- FAQs - https://github.com/apache/incubator-doris/wiki/Doris-FAQ