Commit Graph

133 Commits

Author SHA1 Message Date
f1db289934 Fix compile error (#461) 2018-12-24 10:26:03 +08:00
90d71508ff Add UserFunctionCache to cache UDF's library (#453)
* Add UserFunctionCache to cache UDF's library

This patch replace LibCache with UserFunctionCache. LibCache use HDFS
URL to identify a UDF's Library, and when BE process restart all of
downloaded library should be loaded another time. We use function id
corresponding to a library, and when process restart, all downloaded
libraries can be loaded without another downloading.

* update
2018-12-21 22:07:21 +08:00
0341ffde67 Revert commit 'Add log to detect empty load file' (#447)
Looks like we still need to send push task without file
to Backend, or load job will fail.
Fix it later.
2018-12-19 12:29:12 +08:00
5a6e5cfd07 Add log to detect empty load file (#445)
We find that a load file may not be generated for rollup tablet,
add a log to observe.
2018-12-18 12:44:36 +08:00
e99468c387 Add HttpClient class (#441)
Replace FileDownloader with HttpClient, this patch change clone_copy and
pusher's download.
2018-12-18 09:55:11 +08:00
e177c23787 Update the increased frequency of priority for remaining tasks in BlockingPriorityQueue (#434) 2018-12-17 21:05:57 +08:00
dc4cbab11e Report error when loading decimal value with scientific notation (#428)
Currently we do not support scientific notation of decimal value.
2018-12-17 21:04:18 +08:00
d291068b37 Fix memory_copy core dump in aggregate (#440) 2018-12-17 19:02:49 +08:00
0bedd336b5 Support add new key column for LinkedSchemaChange (#432)
* Support add key column for LinkedSchemaChange

* Move ColumnMapping to single file
2018-12-14 20:25:25 +08:00
74cc5c5404 Write summary line in load error file anyway (#425)
Summary line should be wrote in spite of the limit error number
2018-12-13 12:35:19 +08:00
850150896a Fix bug that load error log is empty sometimes (#424)
_num_print_error_rows is not initialized
2018-12-13 11:23:32 +08:00
371e3d18ca Fix base compaction bug. (#421)
Base compaction may choose tablet which has missed versions now.
After compaction, it will failed to check tablet integrity by versions and cored dump.
Ignore this tablet when find tablet to base_compaction.
2018-12-12 20:44:32 +08:00
e2bb86cf78 Add Md5Digest to util (#420) 2018-12-12 20:06:35 +08:00
842e943f56 Fix compaction and ingestion core (#417)
Error occurs when reading data by compaction and ingestion.
Under the circumstance, the two operation should stop and return error.
2018-12-12 11:30:06 +08:00
5634d13dd2 Release push_lock when schema change failed (#407)
Release push_lock when schema change failed
2018-12-10 18:39:19 +08:00
530bdec020 Fix bug that null value will be loaded to non-nullable column (#401)
* Fix bug that null value will be loaded to non-nullable column

* Optimize performance
2018-12-06 19:55:55 +08:00
e913e45343 Fix bug that null value will be loaded to non-nullable column (#397) 2018-12-06 10:09:34 +08:00
6b4049e21c Unify Slice code path (#380) 2018-12-03 18:11:47 +08:00
ff95f23615 Remove OLAP_LOG_DEBUG AND OLAP_LOG_TRACE log format (#378)
Use VLOG(3) and VLOG(10) instead
2018-12-03 10:08:21 +08:00
5dea8bd3e6 Remove OLAP_LOG_FATAL log format. Use LOG(FATAL) instead (#376) 2018-12-01 19:26:08 +08:00
3d324e38ea Remove OLAP_LOG_INFO log format. Use LOG(INFO) instead (#372) 2018-11-30 20:59:40 +08:00
49302955c8 Revert "Remove OLAP_LOG_INFO log format. Use LOG(INFO) instead (#370)" (#371)
This reverts commit a816925776de06dc7503ea7429802cad9042d0e4.
2018-11-30 20:56:51 +08:00
a816925776 Remove OLAP_LOG_INFO log format. Use LOG(INFO) instead (#370)
* Remove unused row-oriented format flags

* Remove unused row-oriented format flags

* Remove OLAP_LOG_INFO log format. Use LOG(INFO) instead
2018-11-30 20:36:58 +08:00
85d0996b35 Rename Rowset to SegmentGroup (#364)
* Rename Rowset to SegmentGroup

* Modify protobuf related rowset to SegmentGroup
2018-11-29 17:30:41 +08:00
aa27d0e056 Fix snapshot's making header bug (#362) 2018-11-28 18:58:21 +08:00
477b6b3d2a Remove unused row-oriented format flags (#357) 2018-11-27 16:34:22 +08:00
dedfccfaf5 Optimize the publish logic of streaming load (#350)
1. Only collect all error replicas if publish task is timeout.
2. Add 2 metrics to monitor the success of failure of txn.
3. Change publish timeout to Config.load_straggler_wait_second
2018-11-26 19:01:50 +08:00
7ac011571f Improve the Backend's disk info report performance (#349)
1. Use the data size save in tablet's header to calculate the disk used capacity.
2. Decrease the default interval of disk and tablet report, from 10 min to 1 min.
2018-11-26 13:42:58 +08:00
33873f2446 Fix wrong query result when column value is Null (#344) 2018-11-26 13:36:23 +08:00
be6e9c393d Fix bug of using symbolic link dir as storage path (#340)
* Fix bug of #307

There is a bug to use symbolic link directory as storage root path.
It is a problem that whether the path is canonical

In DownloadAction, checking fails by comparing canonical path with non-canonical path.
So fix the bug by convert all path to canonical path before comparison
2018-11-23 10:12:26 +08:00
8fbc043f93 Change ByteBuffer to StorageByteBuffer in olap/byte_buffer.h (#341)
Class definition of ByteBuffer duplicates between olap/byte_buffer.h and util/byte_buffer.h.
All of the two classes has a function names as remaining().
Some place which want to call remaining() of util/byte_buffer.h is linked to the other remaining() function of olap/byte_buffer.h
2018-11-22 23:18:33 +08:00
cec1b971f6 Add special add_column_statistics method for linked_schema_change (#337) 2018-11-22 15:39:00 +08:00
fbde6a8636 Support for custom build toolchains (#330)
* Support for custom build toolchains

ISSUE #297

Locate the header file path of DORIS_GCC_HOME, add them to the CPLUS_INCLUDE_PATH environment variable.
So clang can find it.

* Compatible with custom toolchain for clang

ISSUE #297

1. Find default header file paths of DORIS_GCC_HOME and set as enviroment variable CLANG_COMPATIBLE_FLAGS.
2. Explicitly add these paths to CLANG_INCLUDE_FLAGS.
2018-11-21 15:58:12 +08:00
9a2ad18428 Add path info of replica in catalog (#327)
Add path info of replica in catalog

Also fix a bug that when calling check_none_row_oriented_table,
store is null, it cannot be used to create table.
Instead, OLAPHeader can be used to get storage type information.
2018-11-19 17:42:46 +08:00
fec3c58655 Change log verbose level to vlog(3) (#325)
* Transform row-oriented table to columnar-oriented table

* Transform row-oriented table to columnar-oriented table

* change log verbose level
2018-11-16 17:17:39 +08:00
1ba8a4ee4e Transform row-oriented table to columnar-oriented table (#311) 2018-11-16 16:03:56 +08:00
0aea149c0b Fix core local value UT failed (#324)
Issue: #323
2018-11-16 15:27:16 +08:00
f1b673503e Fix lose of be's meta data bug (#318)
Modify rocksdb's WriteOptions.sync to true
to force sync WAL to disk every write operation
2018-11-15 19:23:42 +08:00
a2b299e3b9 Reduce UT binary size (#314)
* Reduce UT binary size

Almost every module depend on ExecEnv, and ExecEnv contains all
singleton, which make UT binary contains all object files.

This patch seperate ExecEnv's initial and destory to anthor file to
avoid other file's dependence. And status.cc include debug_util.h which
depend tuple.h tuple_row.h, and I move get_stack_trace() to
stack_util.cpp to reduce status.cc's dependence.

I add USE_RTTI=1 to build rocksdb to avoid linking librocksdb.a

Issue: #292

* Update
2018-11-15 16:17:23 +08:00
3a878fcf09 Improve build script (#305) 2018-11-14 14:10:46 +08:00
c6df9b1e2d Make BE UT success (#304) 2018-11-13 19:17:51 +08:00
063f7d7a9a Fix code LICENSE for file modified from LevelDB. (#300) 2018-11-12 16:09:40 +08:00
ae8d16c81e Fix failed cases in regression test (#299) 2018-11-12 11:15:39 +08:00
e8b4ebd4c5 Correct TxnKey comment (#295) 2018-11-10 15:49:10 +08:00
2081b7fea5 Be compatible with old RPC (#296)
Add palo.PInternalService which can server old version palo's client.

Issue: #293
2018-11-10 15:46:45 +08:00
6f206ae9c6 Fix some license (#290) 2018-11-09 14:30:09 +08:00
a1a795442a Fix be build error (#289)
#288
2018-11-09 10:39:03 +08:00
0c4edc2b3c Fix BE can't be grayscale upgraded (#285) 2018-11-07 09:34:39 +08:00
370e73ce5d Fix truncation error in CastExpr (#283) 2018-11-06 18:57:13 +08:00
9c3620047a Fix BE can't be started by start_be.sh (#281) 2018-11-05 11:21:51 +08:00