database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
Files
0430714ca9b0d910ba9b290276da05adfc752f8f
doris/be/src/util
History
Yingchun Lai 72f3082358 [Metrics] Add some metrics for container size in BE (#3246) 
We can observe the workload of BE, and also it's a way to check
whether there is any problem in BE, like some container increase
too large and lead to OOM.

This patch add the following metrics:
```
Name                                   Description
rowset_count_generated_and_in_use      The total count of rowset id generated and in use since BE last start
unused_rowsets_count                   The total count of unused rowset waiting to be GC
broker_count                           The total count of brokers in management
data_stream_receiver_count             The total count of data stream receivers in management
fragment_endpoint_count                The total count of fragment endpoints of data stream in management, should always equal to data_stream_receiver_count
active_scan_context_count              The total count of active scan contexts
plan_fragment_count                    The total count of plan fragments in executing
load_channel_count                     The total count of load channels in management
result_buffer_block_count              The total count of result buffer blocks for queries, each block has a limited queue size (default 1024)
result_block_queue_count               The total count of queues for fragments, each queue has a limited size (default 20, by config::max_memory_sink_batch_count)
routine_load_task_count                The total count of routine load tasks in executing
small_file_cache_count                 The total count of cached small files' digest info
stream_load_pipe_count                 The total count of stream load pipes, each pipe has a limited buffer size (default 1M)
tablet_writer_count                    The total count of tablet writers
brpc_endpoint_stub_count               The total count of brpc endpoints
```
2020-04-25 16:13:39 +08:00
..
arrow
Add ext_unix_timestamp for date < 1970-01-01 and > 2038-01-19 (#2161)
2019-11-08 21:19:26 +08:00
debug
[LSAN] Fix some memory leak detected by LSAN (#3326)
2020-04-22 22:59:44 +08:00
minizip
Add zip util(#2348) (#2441)
2019-12-27 10:10:21 +08:00
aes_util.cpp
Remove my aes and fix palo ns to doris (#277)
2018-11-02 17:05:48 +08:00
aes_util.h
Remove my aes and fix palo ns to doris (#277)
2018-11-02 17:05:48 +08:00
aligned_new.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
alignment.h
Enable auto convert when check in (#1926)
2019-10-09 22:31:27 +08:00
barrier.h
[util] Import ThreadPool and Thread from KUDU (#2915)
2020-02-17 11:22:09 +08:00
batch_process_thread_pool.hpp
Check all tablet using partition tablet map during publish version (#1619)
2019-08-14 18:24:52 +08:00
bfd_parser.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
bfd_parser.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
bit_stream_utils.h
Enable auto convert when check in (#1926)
2019-10-09 22:31:27 +08:00
bit_stream_utils.inline.h
Enable auto convert when check in (#1926)
2019-10-09 22:31:27 +08:00
bit_util.h
Add rle encoding (#1326)
2019-06-18 14:48:33 +08:00
bitmap_value.h
Support 64 bits integers for BITMAP type (#2772)
2020-01-17 14:13:38 +08:00
bitmap.cpp
Support 64 bits integers for BITMAP type (#2772)
2020-01-17 14:13:38 +08:00
bitmap.h
Support 64 bits integers for BITMAP type (#2772)
2020-01-17 14:13:38 +08:00
block_compression.cpp
[LSAN] Fix some memory leak detected by LSAN (#3326)
2020-04-22 22:59:44 +08:00
block_compression.h
Support page compression and checksum in BetaRowset (#1646)
2019-08-19 09:40:47 +08:00
blocking_priority_queue.hpp
Check all tablet using partition tablet map during publish version (#1619)
2019-08-14 18:24:52 +08:00
blocking_queue.hpp
Add path info of replica in catalog (#327)
2018-11-19 17:42:46 +08:00
broker_load_error_hub.cpp
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
broker_load_error_hub.h
Enhance the usability of Load operation (#490)
2019-01-03 19:07:27 +08:00
brpc_stub_cache.h
[Metrics] Add some metrics for container size in BE (#3246)
2020-04-25 16:13:39 +08:00
buffer_builder.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
byte_buffer.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
cgroup_util.cpp
[UT]Fix unit test for cgroup_util (#3094)
2020-03-12 22:59:40 +08:00
cgroup_util.h
Use cgroups memory limit and cpu cores in container (#2710)
2020-01-10 00:45:50 +08:00
cidr.cpp
Replace all remaining boost::split() with strings::split() (#2302)
2019-11-26 22:22:14 +08:00
cidr.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
CMakeLists.txt
[Code Refactor] Using block manager to unify the data file access. (#3189)
2020-03-25 20:39:07 +08:00
coding.cpp
Fix code's license (#1715)
2019-08-28 18:08:26 +08:00
coding.h
Support bitmap index for more type (#2630)
2020-01-31 21:09:29 +08:00
condition_variable.cpp
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
condition_variable.h
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
container_util.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
core_local.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
core_local.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
countdown_latch.h
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
counter_cond_variable.hpp
[Load] Fix several races in stream load that could cause BE crash (#2414)
2019-12-10 21:59:05 +08:00
cpu_info.cpp
Fix incompatibility with arm architecture in util and gutil (#2650)
2020-01-06 18:39:31 +08:00
cpu_info.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
crc32c.cpp
Use crc32c checksum for segment v2 (#1753)
2019-09-06 15:23:57 +08:00
crc32c.h
Use crc32c checksum for segment v2 (#1753)
2019-09-06 15:23:57 +08:00
date_func.cpp
Use double when converting TIME type value (#1722)
2019-08-29 21:19:19 +08:00
date_func.h
Use double when converting TIME type value (#1722)
2019-08-29 21:19:19 +08:00
debug_counters.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
debug_util.cpp
Show BE version in 'show backends;' (#3074)
2020-03-12 22:15:13 +08:00
debug_util.h
Show BE version in 'show backends;' (#3074)
2020-03-12 22:15:13 +08:00
defer_op.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
disk_info.cpp
use canonical path in DiskInfo::get_disk_devices() (#3000)
2020-02-27 11:00:50 +08:00
disk_info.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
doris_metrics.cpp
[Metrics] Add some metrics for container size in BE (#3246)
2020-04-25 16:13:39 +08:00
doris_metrics.h
[Metrics] Add some metrics for container size in BE (#3246)
2020-04-25 16:13:39 +08:00
dummy_runtime_profile.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
dynamic_util.cpp
[LSAN] Fix some memory leak detected by LSAN (#3326)
2020-04-22 22:59:44 +08:00
dynamic_util.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
errno.cpp
Add Env for file operation (#1321)
2019-06-17 10:18:16 +08:00
errno.h
Add Env for file operation (#1321)
2019-06-17 10:18:16 +08:00
error_util.cc
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
error_util.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
fake_lock.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
faststring.cc
Port latest faststring (#2403)
2019-12-06 20:39:56 +08:00
faststring.h
Port latest faststring (#2403)
2019-12-06 20:39:56 +08:00
file_cache.cpp
Add file cache for v2 (#2782)
2020-02-04 00:16:01 +08:00
file_cache.h
Add file cache for v2 (#2782)
2020-02-04 00:16:01 +08:00
file_utils.cpp
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
file_utils.h
Replace boost canonicalize (#2209)
2019-11-19 17:57:37 +08:00
filesystem_util.cc
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
filesystem_util.h
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
frame_of_reference_coding.cpp
Support bitmap index for more type (#2630)
2020-01-31 21:09:29 +08:00
frame_of_reference_coding.h
Support bitmap index for more type (#2630)
2020-01-31 21:09:29 +08:00
hash_util_ir.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
hash_util.hpp
Add bitmap_hash function (#2439)
2019-12-12 16:55:07 +08:00
internal_queue.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
json_util.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
json_util.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
load_error_hub.cpp
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
load_error_hub.h
Enhance the usability of Load operation (#490)
2019-01-03 19:07:27 +08:00
logging.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
lru_cache.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
md5.cpp
Add Md5Digest to util (#420)
2018-12-12 20:06:35 +08:00
md5.h
Add Md5Digest to util (#420)
2018-12-12 20:06:35 +08:00
mem_info.cpp
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
mem_info.h
Use cgroups memory limit and cpu cores in container (#2710)
2020-01-10 00:45:50 +08:00
mem_range.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
mem_util.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
metrics.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
metrics.h
[config] Support to modify configs when BE is running without restarting (#3264)
2020-04-08 11:17:47 +08:00
monotime.cpp
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
monotime.h
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
murmur_hash3.cpp
Get rid of choosing one tablet by compaction (#433)
2018-12-24 16:55:39 +08:00
murmur_hash3.h
Get rid of choosing one tablet by compaction (#433)
2018-12-24 16:55:39 +08:00
mutex.cpp
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
mutex.h
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
mysql_global.h
Add bitamp_to_string function (#2731)
2020-01-13 12:31:37 +08:00
mysql_load_error_hub.cpp
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
mysql_load_error_hub.h
Make MySQL support optional (#1248)
2019-06-05 12:28:15 +08:00
mysql_row_buffer.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
mysql_row_buffer.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
network_util.cpp
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
network_util.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
null_load_error_hub.cpp
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
null_load_error_hub.h
Make MySQL support optional (#1248)
2019-06-05 12:28:15 +08:00
once.h
Load Rowset only once in a thread-safe manner (#2022)
2019-10-21 16:05:12 +08:00
os_info.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
os_info.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
parse_util.cpp
[Uitls] ParseUtil::parse_mem_spec support K and T suffix (#2854)
2020-02-07 09:31:35 +08:00
parse_util.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
path_builder.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
path_builder.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
path_trie.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
path_util.cpp
Remove boost exception when parse store path (#2861)
2020-02-10 17:50:52 +08:00
path_util.h
Remove boost exception when parse store path (#2861)
2020-02-10 17:50:52 +08:00
perf_counters.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
perf_counters.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
pretty_printer.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
priority_thread_pool.hpp
Use ThreadPool to refactor MemTableFlushExecutor (#2931)
2020-02-18 18:39:04 +08:00
progress_updater.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
progress_updater.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
radix_sort.h
Optimize percentile_approx through radix sort (#2102) (#2107)
2019-11-05 09:25:47 +08:00
random.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
ref_count_closure.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
rle_encoding.h
Enable auto convert when check in (#1926)
2019-10-09 22:31:27 +08:00
runtime_profile.cpp
[Profile] Make running profile clearer and more intuitive to improve usability (#3365) (#3383)
2020-04-24 21:38:55 +08:00
runtime_profile.h
[Profile] Make running profile clearer and more intuitive to improve usability (#3365) (#3383)
2020-04-24 21:38:55 +08:00
scoped_cleanup.h
[util] Import util tools from KUDU (#2905)
2020-02-14 18:01:16 +08:00
semaphore.hpp
Optimize the load performance for large file (#1798)
2019-09-25 13:49:32 +08:00
simdutf8check.h
Validate the UTF-8 encode of loading data (#1457)
2019-07-11 09:46:38 +08:00
slice.cpp
Support Segment for BetaRowset (#1577)
2019-08-06 17:15:11 +08:00
slice.h
Add Bitmap index reader (#2319)
2019-12-03 23:01:40 +08:00
spinlock.cc
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
spinlock.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
sse_util.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
stack_util.cpp
Reduce UT binary size (#314)
2018-11-15 16:17:23 +08:00
stack_util.h
Reduce UT binary size (#314)
2018-11-15 16:17:23 +08:00
stat_util.hpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
static_asserts.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
stopwatch.hpp
Fix incompatibility with arm architecture in util and gutil (#2650)
2020-01-06 18:39:31 +08:00
streaming_sampler.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
string_parser.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
string_parser.hpp
Improve to_bitmap parse int performance (#2223)
2019-11-19 18:00:19 +08:00
string_util.cpp
Unify the type of path_hash to size_t (#2324)
2019-11-28 18:48:52 +08:00
string_util.h
Unify the type of path_hash to size_t (#2324)
2019-11-28 18:48:52 +08:00
symbols_util.cpp
Add object type (#1948)
2019-10-31 21:42:58 +08:00
symbols_util.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
system_metrics.cpp
Enhance the usabilities (#1100)
2019-05-07 15:55:04 +08:00
system_metrics.h
Add some pre-calculated metrics (#1079)
2019-04-30 11:12:23 +08:00
tdigest.h
Optimize percentile_approx through radix sort (#2102) (#2107)
2019-11-05 09:25:47 +08:00
template_util.h
Change license to Apache License 2.0 (#262)
2018-11-01 09:06:01 +08:00
thread.cpp
[util] Add Apache License 2.0 to Thread (#2928)
2020-02-18 15:36:49 +08:00
thread.h
Remove unused KUDU codes (#3175)
2020-03-24 13:54:05 +08:00
threadpool.cpp
Fix the flush_status bug in flush-executor (#2933)
2020-02-19 20:23:19 +08:00
threadpool.h
Fix the flush_status bug in flush-executor (#2933)
2020-02-19 20:23:19 +08:00
thrift_client.cpp
Use SleepFor() instead of usleep() (#3211)
2020-03-29 14:18:19 +08:00
thrift_client.h
[Thrift] Fix a bug when ThriftClientImpl close() error (#3128)
2020-03-17 15:53:44 +08:00
thrift_rpc_helper.cpp
Refactor FrontendHelper (#1888)
2019-09-27 13:21:14 +08:00
thrift_rpc_helper.h
[config] Support to modify configs when BE is running without restarting (#3264)
2020-04-08 11:17:47 +08:00
thrift_server.cpp
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
thrift_server.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
thrift_util.cpp
Use SleepFor() instead of usleep() (#3211)
2020-03-29 14:18:19 +08:00
thrift_util.h
Uniform Status (#1317)
2019-06-14 23:38:31 +08:00
time.cpp
Fix license issue of gutil (#471)
2018-12-27 13:53:27 +08:00
time.h
[Bug] Avoid compacting recengly added rowset (#3271)
2020-04-08 18:58:12 +08:00
tuple_row_compare.cpp
Remove unused LLVM related codes of directory:be/src/util,be/src/udf (#2910) (#2968)
2020-02-21 20:42:42 +08:00
tuple_row_compare.h
Remove unused LLVM related codes of directory:be/src/util,be/src/udf (#2910) (#2968)
2020-02-21 20:42:42 +08:00
types.h
[Bug] Fix int128 bloom filter write bug (#2995)
2020-03-05 09:15:11 +08:00
uid_util.cpp
Add show load of loadv2 (#1113)
2019-05-09 10:27:30 +08:00
uid_util.h
Improve unique id performance (#1911)
2019-09-29 18:20:02 +08:00
url_coding.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
url_coding.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
url_parser.cpp
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
url_parser.h
Change variable and namespace name in BE (#268)
2018-11-02 10:22:32 +08:00
utf8_check.cpp
Fix incompatibility with arm architecture in util and gutil (#2650)
2020-01-06 18:39:31 +08:00
utf8_check.h
Fix incompatibility with arm architecture in util and gutil (#2650)
2020-01-06 18:39:31 +08:00
uuid_generator.h
Improve unique id performance (#1911)
2019-09-29 18:20:02 +08:00
zip_util.cpp
Add zip util(#2348) (#2441)
2019-12-27 10:10:21 +08:00
zip_util.h
Add zip util(#2348) (#2441)
2019-12-27 10:10:21 +08:00