From 1ca4d8bf10b08ed7046cfd12145fa823bd0badbb Mon Sep 17 00:00:00 2001 From: Petrichor <31833513+vinlee19@users.noreply.github.com> Date: Tue, 9 Jan 2024 10:36:32 +0800 Subject: [PATCH] [improve](docs) add Flink write metrics doc (#29578) --- .../docs/ecosystem/flink-doris-connector.md | 24 ++++++++++++++++++- .../docs/ecosystem/flink-doris-connector.md | 19 +++++++++++++++ 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/docs/en/docs/ecosystem/flink-doris-connector.md b/docs/en/docs/ecosystem/flink-doris-connector.md index 737f303f67..090f515bfc 100644 --- a/docs/en/docs/ecosystem/flink-doris-connector.md +++ b/docs/en/docs/ecosystem/flink-doris-connector.md @@ -390,6 +390,28 @@ ON a.city = c.city | TIME | DOUBLE | | HLL | Unsupported datatype | +## Flink write Metrics +Where the metrics value of type Counter is the cumulative value of the imported task from the beginning to the current time, you can observe each metric in each table in the Flink Webui metrics. + +| Name | Metric Type | Description | +| ------------------------- | ----------- | ------------------------------------------------------------ | +| totalFlushLoadBytes | Counter | Number of bytes imported. | +| flushTotalNumberRows | Counter | Number of rows imported for total processing | +| totalFlushLoadedRows | Counter | Number of rows successfully imported. | +| totalFlushTimeMs | Counter | Number of Import completion time. Unit milliseconds | +| totalFlushSucceededNumber | Counter | Number of times that the data-batch been successfully imported. | +| totalFlushFailedNumber | Counter | Number of times that the data-batch been failed. | +| totalFlushFilteredRows | Counter | Number of rows that do not qualify for data quality flushed | +| totalFlushUnselectedRows | Counter | Number of rows filtered by where condition flushed | +| beginTxnTimeMs | Histogram | The time cost for RPC to Fe to begin a transaction, Unit milliseconds. | +| putDataTimeMs | Histogram | The time cost for RPC to Fe to get a stream load plan, Unit milliseconds. | +| readDataTimeMs | Histogram | Read data time, Unit milliseconds. | +| writeDataTimeMs | Histogram | Write data time, Unit milliseconds. | +| commitAndPublishTimeMs | Histogram | The time cost for RPC to Fe to commit and publish a transaction, Unit milliseconds. | +| loadTimeMs | Histogram | Import completion time | + + + ## An example of using Flink CDC to access Doris ```sql SET 'execution.checkpointing.interval' = '10s'; @@ -759,6 +781,6 @@ You can search for the log `abort transaction response` in TaskManager and deter This problem is mainly caused by the conditional varchar/string type, which needs to be quoted. The correct way to write it is xxx = ''xxx''. In this way, the Flink SQL parser will interpret two consecutive single quotes as one single quote character instead of The end of the string, and the concatenated string is used as the value of the attribute. -15. **Failed to connect to backend: http://host:webserver_port, and Be is still alive** +15. **Failed to connect to backend: http://host:webserver_port, and BE is still alive** The issue may have occurred due to configuring the IP address of `be`, which is not reachable by the external Flink cluster.This is mainly because when connecting to `fe`, the address of `be` is resolved through fe. For instance, if you add a be address as '127.0.0.1', the be address obtained by the Flink cluster through fe will be '127.0.0.1:webserver_port', and Flink will connect to that address. When this issue arises, you can resolve it by adding the actual corresponding external IP address of the be to the "with" attribute:`'benodes'="be_ip:webserver_port,be_ip:webserver_port..."`.For the entire database synchronization, the following properties are available`--sink-conf benodes=be_ip:webserver,be_ip:webserver...`. diff --git a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md index 906f7ecd2d..c269d0931c 100644 --- a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md +++ b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md @@ -393,6 +393,25 @@ ON a.city = c.city | TIME | DOUBLE | | HLL | Unsupported datatype | +## Flink 写入指标 +其中Counter类型的指标值为导入任务从开始到当前的累加值，可以在Flink Webui metrics中观察各表的各项指标。 +| Name | Metric Type | Description | +| ------------------------- | ----------- | ------------------------------------------ | +| totalFlushLoadBytes | Counter | 已经刷新导入的总字节数 | +| flushTotalNumberRows | Counter | 已经导入处理的总行数 | +| totalFlushLoadedRows | Counter | 已经成功导入的总行数 | +| totalFlushTimeMs | Counter | 已经成功导入完成的总时间 | +| totalFlushSucceededNumber | Counter | 已经成功导入的次数 | +| totalFlushFailedNumber | Counter | 失败导入 的次数 | +| totalFlushFilteredRows | Counter | 数据质量不合格的总行数 | +| totalFlushUnselectedRows | Counter | 被 where 条件过滤的总行数 | +| beginTxnTimeMs | Histogram | 向Fe请求开始一个事务所花费的时间，单位毫秒 | +| putDataTimeMs | Histogram | 向Fe请求获取导入数据执行计划所花费的时间 | +| readDataTimeMs | Histogram | 读取数据所花费的时间 | +| writeDataTimeMs | Histogram | 执行写入数据操作所花费的时间 | +| commitAndPublishTimeMs | Histogram | 向Fe请求提交并且发布事务所花费的时间 | +| loadTimeMs | Histogram | 导入完成的时间 | + ## 使用FlinkSQL通过CDC接入Doris示例 ```sql -- enable checkpoint