Commit Graph

272 Commits

Author SHA1 Message Date
cfefa71daa Fix cast error in StreamLoadScanNodeush (#356) 2018-11-27 16:03:33 +08:00
b2d89dfee9 Add connection id to CurrentQueryStatisticsProcDir (#355) 2018-11-27 14:28:39 +08:00
cddd864d83 Avoid 'No more data to read' error when handling stream load RPC (#354)
* Avoid 'No more data to read' error when handling stream load rpc

1. Catch throwable of all stream load rpc.
2. Avoid setting null string as error msg of rpc result status.

* Change setError_msgs to addToError_msgs
2018-11-27 14:18:41 +08:00
dedfccfaf5 Optimize the publish logic of streaming load (#350)
1. Only collect all error replicas if publish task is timeout.
2. Add 2 metrics to monitor the success of failure of txn.
3. Change publish timeout to Config.load_straggler_wait_second
2018-11-26 19:01:50 +08:00
7ac011571f Improve the Backend's disk info report performance (#349)
1. Use the data size save in tablet's header to calculate the disk used capacity.
2. Decrease the default interval of disk and tablet report, from 10 min to 1 min.
2018-11-26 13:42:58 +08:00
33873f2446 Fix wrong query result when column value is Null (#344) 2018-11-26 13:36:23 +08:00
6413518b1e Update README.md (#347)
Move old content of READEME.md to the Overview of Wiki.
Add build guide and user cases to README.md.
2018-11-23 15:40:07 +08:00
bbdf4fba4a Add distributor which schedule task to be fairly, for routine load job (#333)
Step1: updateBeIdTaskMaps, remove unavailable BE and add new alive BE
Step2: process timeout tasks, if a task has already been allocated to BE but not finished before DEFAULT_TASK_TIMEOUT, it will be discarded.
       At the same time, the partitions belong to old tasks will be allocated to a new task. The new task with a signature will be added in the queue of needSchedulerRoutineLoadTask.
Step3: process all needSchedulerRoutineLoadTasks, allocate task to BE. The task will be executed by BE.
2018-11-23 10:35:10 +08:00
be6e9c393d Fix bug of using symbolic link dir as storage path (#340)
* Fix bug of #307

There is a bug to use symbolic link directory as storage root path.
It is a problem that whether the path is canonical

In DownloadAction, checking fails by comparing canonical path with non-canonical path.
So fix the bug by convert all path to canonical path before comparison
2018-11-23 10:12:26 +08:00
8fbc043f93 Change ByteBuffer to StorageByteBuffer in olap/byte_buffer.h (#341)
Class definition of ByteBuffer duplicates between olap/byte_buffer.h and util/byte_buffer.h.
All of the two classes has a function names as remaining().
Some place which want to call remaining() of util/byte_buffer.h is linked to the other remaining() function of olap/byte_buffer.h
2018-11-22 23:18:33 +08:00
cec1b971f6 Add special add_column_statistics method for linked_schema_change (#337) 2018-11-22 15:39:00 +08:00
f9494cbdf2 Fix a compile issue of DORIS_GCC_HOME (#339) 2018-11-22 12:32:14 +08:00
f2df5d523d Fix compile issue of thirdparty library (#338) 2018-11-22 11:15:42 +08:00
485db34f1e Modify partition's version name to what it means (#334)
* Modify partition's version name to what it means.

1. committedVersion(Hash) -> visibleVersion(Hash)
2. currentVersion(Hash) -> committedVersion(Hash)
3. add some comment to make the code more readable

* Check if editlog is null in CatalogIdGenerator
    To avoid unit test failure
2018-11-21 19:21:16 +08:00
fbde6a8636 Support for custom build toolchains (#330)
* Support for custom build toolchains

ISSUE #297

Locate the header file path of DORIS_GCC_HOME, add them to the CPLUS_INCLUDE_PATH environment variable.
So clang can find it.

* Compatible with custom toolchain for clang

ISSUE #297

1. Find default header file paths of DORIS_GCC_HOME and set as enviroment variable CLANG_COMPATIBLE_FLAGS.
2. Explicitly add these paths to CLANG_INCLUDE_FLAGS.
2018-11-21 15:58:12 +08:00
b57dfd4dd3 Fix a compile issue (#331)
* Fix a compile issue

* Fix a compile issue
2018-11-20 20:06:36 +08:00
791e89568e Change PaloMetrics' name and Catalog's Id generator (#329)
* Change PaloMetrics' name and Catalog's Id generator
1. Remove 'Palo' prefix of class Metric.
2. Add a new CatalogIdGenerator to replace the old AtomicLong, to avoid too many edit logs.
3. Add a new histogram to monitor write letency of edit log write.

* modify next id logic

* fix a bug that Metric is not init before using HISTO_EDIT_LOG_WRITE_LATENCY

* fix a problem
2018-11-20 18:59:18 +08:00
9a2ad18428 Add path info of replica in catalog (#327)
Add path info of replica in catalog

Also fix a bug that when calling check_none_row_oriented_table,
store is null, it cannot be used to create table.
Instead, OLAPHeader can be used to get storage type information.
2018-11-19 17:42:46 +08:00
dddaacc2d7 Fix fe.conf audit_log_moudles,sys_log_roll_mode,audit_log_roll_mode settings (#326) 2018-11-17 22:18:19 +08:00
fec3c58655 Change log verbose level to vlog(3) (#325)
* Transform row-oriented table to columnar-oriented table

* Transform row-oriented table to columnar-oriented table

* change log verbose level
2018-11-16 17:17:39 +08:00
b946f20e37 Add docker dev environment (#301)
Add a docker based dev environment.
User can build Doris from source in docker container.
2018-11-16 16:35:08 +08:00
1ba8a4ee4e Transform row-oriented table to columnar-oriented table (#311) 2018-11-16 16:03:56 +08:00
0aea149c0b Fix core local value UT failed (#324)
Issue: #323
2018-11-16 15:27:16 +08:00
e9bf332994 Fix fe.conf audit_log_moudles default value error (#321) 2018-11-16 08:54:57 +08:00
44029937e4 Add scheduler routine load job for stream load (#313)
1. fetch need_scheduler routine load job
2. caculate current concurrent task number of job
3. divide kafka partition into tasks
2018-11-15 21:04:22 +08:00
8ac9492b11 Fix SHOW BACKENDS return ERROR (#320)
In some case, errMsg in Backend maybe null. we change it when check it
nut null

Issue: #317
2018-11-15 20:14:39 +08:00
f1b673503e Fix lose of be's meta data bug (#318)
Modify rocksdb's WriteOptions.sync to true
to force sync WAL to disk every write operation
2018-11-15 19:23:42 +08:00
a2b299e3b9 Reduce UT binary size (#314)
* Reduce UT binary size

Almost every module depend on ExecEnv, and ExecEnv contains all
singleton, which make UT binary contains all object files.

This patch seperate ExecEnv's initial and destory to anthor file to
avoid other file's dependence. And status.cc include debug_util.h which
depend tuple.h tuple_row.h, and I move get_stack_trace() to
stack_util.cpp to reduce status.cc's dependence.

I add USE_RTTI=1 to build rocksdb to avoid linking librocksdb.a

Issue: #292

* Update
2018-11-15 16:17:23 +08:00
d7ee57e881 Optimize quota unit (#309)
Originally, we can only set quota in bytes unit. This commit add quota unit:K/KB/M/MB/G/GB/T/TB/P/PB
for convenience.
2018-11-15 14:03:52 +08:00
3a878fcf09 Improve build script (#305) 2018-11-14 14:10:46 +08:00
c6df9b1e2d Make BE UT success (#304) 2018-11-13 19:17:51 +08:00
063f7d7a9a Fix code LICENSE for file modified from LevelDB. (#300) 2018-11-12 16:09:40 +08:00
ae8d16c81e Fix failed cases in regression test (#299) 2018-11-12 11:15:39 +08:00
e8b4ebd4c5 Correct TxnKey comment (#295) 2018-11-10 15:49:10 +08:00
2081b7fea5 Be compatible with old RPC (#296)
Add palo.PInternalService which can server old version palo's client.

Issue: #293
2018-11-10 15:46:45 +08:00
6f206ae9c6 Fix some license (#290) 2018-11-09 14:30:09 +08:00
a1a795442a Fix be build error (#289)
#288
2018-11-09 10:39:03 +08:00
1d8fc4bb69 Improve cardinality, avgRowSize, numNodes stat info in OlapScanNode (#256)
Currently, the cardinality, avgRowSize, numNodes stat info in OlapScanNode is none, So the broadcastCost and partitionCost are both wrong and Doris couldn't auto choose a best join strategy.

So we should make the statistical information in OlapScanNode more precise.
2018-11-07 13:59:05 +08:00
fc8f78d81c Fix unit test failure (#286) 2018-11-07 12:53:11 +08:00
0c4edc2b3c Fix BE can't be grayscale upgraded (#285) 2018-11-07 09:34:39 +08:00
370e73ce5d Fix truncation error in CastExpr (#283) 2018-11-06 18:57:13 +08:00
8d7bd01a71 Simplify constant Expr (#255)
Simplify constant Expr could improve Partition Pruning. The examples for constant Expr Simplifing:

1 + 1 + 1 --> 3
date_add('2018-08-08', 1) --> 2018-08-09
year('2018-07-24')*12 + month('2018-07-24') -> 24223
2018-11-06 17:24:54 +08:00
acb332833a Fix view missed parenthesis bug (#253) 2018-11-06 15:25:40 +08:00
cb36e411e9 Support AnalyticExpr in View (#248) 2018-11-05 20:39:21 +08:00
8b665a41c8 Support NULLS LAST and NULLS FIRST syntax (#252)
Allow User specify the null ordering

NULLS FIRST: specifies that NULL values should be returned before
non-NULL values.
NULLS LAST: specifies that NULL values should be returned after
non-NULL values.
2018-11-05 20:35:10 +08:00
9ae631adb6 Fix InsertStmt reAnalyze bug (#251) 2018-11-05 15:36:40 +08:00
38e493df0b Fix typo in READEME.md (#278) 2018-11-05 14:12:11 +08:00
9c3620047a Fix BE can't be started by start_be.sh (#281) 2018-11-05 11:21:51 +08:00
c877b43013 Remove my aes and fix palo ns to doris (#277) 2018-11-02 17:05:48 +08:00
69f3b02485 Fix a bug that user can not kill it own connection (#276) 2018-11-02 16:36:59 +08:00