database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,956 Commits 1 Branch 0 Tags
db0724dfe0c39f71ccc4b9d44b6f9f030dac721f
Commit Graph

1705 Commits

Author SHA1 Message Date
wangbo
357394bb3e [branch-2.1]Reset io limit default value (#39898) 
pick #39842
 2024-08-26 14:27:35 +08:00
TengJianPing
2dea859bdb [debug](rpc) debug rpc time consumption problem (#39852) 
## Proposed changes

Issue Number: close #xxx

Add detail RPC time info for each channel, sorted by max rpc time of
channels:
```
                     DATA_STREAM_SINK_OPERATOR  (id=1,dst_id=1):
                          -  Partitioner:  Crc32HashPartitioner(64)
                          -  BlocksProduced:  74
                          -  BrpcSendTime:  2.689us
                          -  BrpcSendTime.Wait:  0ns
                          -  BytesSent:  89.35  KB
                          -  CloseTime:  680.152us
                          -  CompressTime:  0ns
                          -  ExecTime:  160.663ms
                          -  InitTime:  263.608us
                          -  InputRows:  32.512K  (32512)
                          -  LocalBytesSent:  0.00  
                          -  LocalSendTime:  0ns
                          -  LocalSentRows:  0
                          -  MemoryUsage:  
                              -  PeakMemoryUsage:  80.00  KB
                          -  MergeBlockTime:  0ns
                          -  OpenTime:  4.113ms
                          -  OverallThroughput:  0.0  /sec
                          -  PendingFinishDependency:  41.179ms
                          -  RowsProduced:  32.512K  (32512)
                          -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
 2024-08-24 19:59:39 +08:00
Xinyi Zou
ae4d747c13 [branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes (#39834) 
pick #39611
 2024-08-24 09:21:35 +08:00
Kaijie Chen
37443aa7e1 [improve](move-memtable) reuse connection in load_stream_stub (#39231) (#39762) 
backport #39231
 2024-08-23 22:46:28 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
1367f74e7a [branch-2.1](memory) Optimize ClearCacheActionimplementation (#39796) 
pick #38438
 2024-08-23 01:51:14 +08:00
zhiqiang
06a0b35704 [chore] Comment for tv_nsec (#39752) 
just comment.
 2024-08-22 22:16:21 +08:00
Xinyi Zou
13b882a4cc [branch-2.1](memory) Add memory metrics to bvar (#39763) 
pick #38391
 2024-08-22 17:34:30 +08:00
Xinyi Zou
8ce8887b75 [branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760) 
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
 2024-08-22 17:33:11 +08:00
zhiqiang
56cc9cc304 [fix](cancel)) Fix pipeline task leak cancel (#39697) 
pick #39737
 2024-08-22 00:40:22 +08:00
Yongqiang YANG
610f69432a [improvement](segmentcache) limit segment cache by fd limit or memory… (#39689) 
… (#39658)

remove a useless config.
 2024-08-21 15:19:52 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
zhiqiang
830f250a80 [opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537) 
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
 2024-08-19 14:33:59 +08:00
meiyi
eea3676791 [fix](group commit) fix group commit insert rpc may stuck (#39391) (#39458) 
pick https://github.com/apache/doris/pull/39391
 2024-08-16 13:19:00 +08:00
Pxl
6f1d9812bb [Bug](brpc) fix sync_filter_size/apply_filterv2 has wrong closure (#39299) 
pick from #39155
 2024-08-13 19:01:22 +08:00
Xin Liao
1ef42dd94b [Fix](load) The value of the index id printed in the log is incorrect #38790 (#39131) 
cherry pick from #38790
 2024-08-09 12:31:33 +08:00
Jerry Hu
0571342538 [fix](sink) The issue with 2GB limit of protocol buffer (#37990) (#39112) 
```
Fail to serialize doris.PFetchDataResult
```

If the size of `PFetchDataResult` is greater than 2G, protocol buffer
cannot serialize the message.

pick #37990
 2024-08-09 04:01:56 +08:00
meiyi
2ec1a6a7e7 [fix](group commit) Modify group commit commit/abort txn timeout as stream load (#39003) (#39069) 
pick https://github.com/apache/doris/pull/39003
 2024-08-08 14:36:29 +08:00
meiyi
749c9f7b56 [fix](group commit) fix repaly wal check label status (#38883) (#38997) 
pick https://github.com/apache/doris/pull/38883
 2024-08-07 22:06:59 +08:00
meiyi
3abb222064 [fix](group commit) Fix test_group_commit_async_wal_msg_fault_injection case (#35313) (#38911) 
pick https://github.com/apache/doris/pull/35313
 2024-08-06 17:57:22 +08:00
zclllhhjj
8fa0710cb3 [branch-2.1](load) fix miss writer in concurrency incremental open (#38605) (#38793) 
pick https://github.com/apache/doris/pull/38605
 2024-08-05 08:56:23 +08:00
meiyi
e8690b62ee [fix](group commit) Pick add debug log show why group commit not work; delete wal when replay success (#38611) (#38659) 
Pick https://github.com/apache/doris/pull/38611
 2024-08-01 16:59:54 +08:00
walter
182bf4d323 [chore](fe) Returns dropped tables in GetMeta request (#38541) 
Cherry-pick #38019
 2024-07-31 10:57:00 +08:00
Mryange
017dad8c54 [fix](type)support runtime predicate for time type (#38258) (#38465) 
## Proposed changes
https://github.com/apache/doris/pull/38258
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-31 10:27:36 +08:00
Xin Liao
e9f12fac47 [fix](load) fix no error url for stream load #38325 (#38417) 
cherry pick from #38325
 2024-07-28 19:06:57 +08:00
meiyi
91f0301b43 [fix](group commit) Pick some group commit pr (#38320) 
Pick https://github.com/apache/doris/pull/38292,
https://github.com/apache/doris/pull/34021,
https://github.com/apache/doris/pull/38228, some modify of
https://github.com/apache/doris/pull/37260, some modify of
https://github.com/apache/doris/pull/37595
 2024-07-25 17:32:44 +08:00
Kaijie Chen
54787f51c7 [log](load) print reason when LoadPathMgr failed to delete file (#38235) (#38340) 
backport #38235
 2024-07-25 17:28:09 +08:00
Qi Chen
a751372e76 [Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 (#37257) 
## Proposed changes

backport #37234
 2024-07-25 13:51:59 +08:00
abmdocrt
792bd7c74a [cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558" (#37830) 
Pick #35558
 2024-07-24 09:21:07 +08:00
Xin Liao
ce1c268743 [Fix](load) Fix uninitiated load_type (not a valid value for type) of StreamLoadContext when operate_txn_2pc #38033 (#38187) 
cherry pick from #38033
 2024-07-22 17:20:03 +08:00
wangbo
7b141ffde7 [pick]add min scan thread num for workload group's scan thread (#38123) 
## Proposed changes

pick #38096
 2024-07-19 18:43:05 +08:00
Gabriel
88d771d360 [pipeline](fix) Avoid to use a freed dependency when cancelled (#34584) (#38046) 
## Proposed changes

pick #34584
<!--Describe your changes.-->
 2024-07-18 15:27:10 +08:00
hui lai
1875267796 [fix](routine-load) fix routine load pause when Kafka data deleted after TTL (#37288) (#37983) 
pick (#37288)

When using routine load, After the data load is completed, the lag is
still a positive number:
```
  Lag: {"0":16,"1":15,"2":16,"3":16,"4":16,"5":16,"6":15,"7":16,"8":16,"9":16,"10":15,"11":16,"12":15,"13":15,"14":16,"15":16,"16":17,"17":15,"18":16,"19":15,"20":16,"21":16,"22":16,"23":16,"24":15,"25":17,"26":17,"27":16,"28":16,"29":16,"30":16,"31":17,"32":14,"33":16,"34":17,"35":16,"36":15,"37":15,"38":15,"39":16,"40":16,"41":16,"42":15,"43":15,"44":17,"45":16,"46":15,"47":15,"48":16,"49":17,"50":16,"51":15,"52":16,"53":15,"54":15,"55":17,"56":16,"57":17,"58":16,"59":16,"60":15,"61":15,"62":16,"63":16,"64":17,"65":16,"66":15,"67":16,"68":17,"69":16,"70":15,"71":17}
```
and the routing load is paused when the Kafka data reaches TTL and is
deleted, the error is `out of range`.

The reason why this happened is EOF has it offset which needed
statistics.

**note(important):**
After the bug is fixed, if you set 
```
"property.enable.partition.eof" = "false"
```
in your routine load job, it will meet the problem. For EOF has offset,
and the config is true in Doris default.
 2024-07-17 13:47:26 +08:00
Xinyi Zou
9861f81630 [branch-2.1](memory) Fix Jemalloc Cache Memory Tracker (#37905) 
pick #37464
 2024-07-16 19:01:31 +08:00
Xinyi Zou
1f779ba9de [branch-2.1](arrow-flight-sql) Open regression-test/pipeline/p0/arrow_flight_sql (#37727) 
pick #36854
 2024-07-16 16:23:43 +08:00
zhiqiang
de61887cdc [chore](log) reduce print warning msg during be starting up #36710 (#37780) 
cherry pick from #36710
 2024-07-15 14:46:54 +08:00
zhiqiang
79f6b647d5 [FIX] should check fe host standing when coordinator is not found. (#37772) 
fix https://github.com/apache/doris/pull/37707
 2024-07-15 12:27:31 +08:00
zclllyybb
2759383365 [branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) (#37269) 
pick https://github.com/apache/doris/pull/37062

1. revert https://github.com/apache/doris/pull/25097. we decide to rely
on OS. not maintain independent tzdata anymore to keep result
consistency
2. refactor timezone load. removed rwlock.

before:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (6.88 sec)
```
now:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (2.61 sec)
```
3. now don't support timezone offset format string like 'UTC+8', like we
already said in
https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage
4. support case-insensitive timezone parsing in nereids.
5. a bug when parse timezone using nereids. should check DST by input,
but wrongly by now before. now fixed.

doc pr: https://github.com/apache/doris-website/pull/810
 2024-07-15 10:56:48 +08:00
camby
9556c07a16 [mac](compile) fix compile error on mac (#37726) 2024-07-15 10:19:42 +08:00
Xinyi Zou
326b40cde2 [branch-2.1](memory) Add HTTP API to clear data cache (#37704) 
pick #36599

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
 2024-07-12 17:21:52 +08:00
Xinyi Zou
a61030215e [branch-2.1](memory) Support make all memory snapshots (#37705) 
pick #36679
 2024-07-12 16:21:37 +08:00
zhiqiang
035027f831 [fix](query cancel) Fix query is cancelled when it comes from follower FE #37662 (#37707) 
cherry pick from #37662
 2024-07-12 15:50:45 +08:00
Xinyi Zou
ef031c5fb2 [branch-2.1](memory) Fix reserve memory compatible with memory GC and logging (#37682) 
pick
#36307
#36412
 2024-07-12 11:43:26 +08:00
Xinyi Zou
cf2fb6945a [branch-2.1](memory) Refactor LRU cache policy memory tracking (#37658) 
pick 
#36235
#35965
 2024-07-11 21:04:01 +08:00
Xinyi Zou
62e0230523 [branch-2.1](memory) Add ThreadMemTrackerMgr BE UT (#37654) 
## Proposed changes

pick #35518
 2024-07-11 21:03:49 +08:00
Kaijie Chen
fed632bf4a [fix](move-memtable) check segment num when closing each tablet (#36753) (#37536) 
cherry-pick #36753 and #37660
 2024-07-11 20:33:44 +08:00
meiyi
1e3ab0ff8c [fix](group commit) Pick make group commit cancel in time (#36249) (#37404) 
pick https://github.com/apache/doris/pull/36249/
 2024-07-09 09:25:11 +08:00
meiyi
1a25270918 [fix](group commit) Pick Fix the incorrect group commit count in log; fix the core in get_first_block (#36408) (#37405) 
Pick https://github.com/apache/doris/pull/36408/
 2024-07-09 09:24:43 +08:00
walter
5280e277e7 [chore](be) Acquire and check MD5 digest of the file to download (#37418) 
Cherry-pick #35807, #36621, #36726
 2024-07-08 18:55:35 +08:00
meiyi
70f46c12b3 [improve](group commit) Pick Modify group commit case and modify cancel status (#35995) (#37398) 
Pick https://github.com/apache/doris/pull/35995
 2024-07-08 10:27:08 +08:00