在hadoop中,存在对应的counter计数器用于记录hadoop map/reduce job任务执行过程中自定义的一些计数器,其中hadoop任务中已经内置了一些计数器,例如CPU时间,GC时间等。
Storm中也存在类似counter的功能,metrics,详细介绍可以参考下面的文档:
Storm exposes a metrics interface to report summary statistics across the full topology. It's used internally to track the numbers you see in the Nimbus UI console: counts of executes and acks; average process latency per bolt; worker heap usage; and so forth.
所有的Metric都需要实现IMetric接口,该接口中只有一个方法用来表示取出现有的counter值,并将其清零。
public interface IMetric { public Object getValueAndReset(); }
以提供的实例CountMetric, MultiCountMetric和ReduceMetric为例,类图方式展示如下:
在CountMetric中,只是记录了一个long值,在每次incr和incrBy时进行递增记录;
MultiCountMetric中,其中内置了Map<String, CountMetric>用来记录多个CountMetric,scope方法用来以key的方式圈定范围,在每次getValueAndSet时,都会直接清掉Map中所有的CountMetric;
ReducedMetric是比较特殊的一个,因为它不仅仅记录了一个维度,还可以以reduce的方式来计算一段时间来的平均值,比如Storm中提供的实现MeanReducer,在ReducedMetric中的实现,其中_accumlator作为累积数据(从字面意思上理解),类型为Object,可以用任何类型来表示:
public class ReducedMetric implements IMetric { private final IReducer _reducer; private Object _accumulator; public ReducedMetric(IReducer reducer) { _reducer = reducer; _accumulator = _reducer.init(); } public void update(Object value) { _accumulator = _reducer.reduce(_accumulator, value); } public Object getValueAndReset() { Object ret = _reducer.extractResult(_accumulator); _accumulator = _reducer.init(); return ret; } }
MeanReducer中,就记录两个维度 count和总和,通过这两个维度,我们可以轻易计算出一段时间内的平均值。
class MeanReducerState { public int count = 0; public double sum = 0.0; } public class MeanReducer implements IReducer<MeanReducerState> { public MeanReducerState init() { return new MeanReducerState(); } public MeanReducerState reduce(MeanReducerState acc, Object input) { acc.count++; if(input instanceof Double) { acc.sum += (Double)input; } else if(input instanceof Long) { acc.sum += ((Long)input).doubleValue(); } else if(input instanceof Integer) { acc.sum += ((Integer)input).doubleValue(); } else { throw new RuntimeException( "MeanReducer::reduce called with unsupported input type `" + input.getClass() + "`. Supported types are Double, Long, Integer."); } return acc; } public Object extractResult(MeanReducerState acc) { if(acc.count > 0) { return acc.sum / (double) acc.count; } else { return null; } } }
所有的Metrics都需要在Spout/Bolt初始化之前记录,对应Spout.open(), Bolt.prepare方法,注册时需要指定指标的名称,对应实例,以及间隔时间(以秒为单位)。
context.registerMetric("execute_count", countMetric, 5); context.registerMetric("word_count", wordCountMetric, 60); context.registerMetric("word_length", wordLengthMeanMetric, 60);
IMetricsConsumer
注册metrics后,只是在定时进行记录metrics,但metrics该如何显示,这就取决于IMetricsConsumer,在Config中可以手动进行注册自定义的metricsConsumer,也可以直接使用storm中提供的记录日志的LoggingMetricsConsumer,该consumer会以日志的形式记录统计指标,下面是对其介绍:
Listens for all metrics, dumps them to log To use, add this to your topology's configuration: ```java conf.registerMetricsConsumer(org.apache.storm.metrics.LoggingMetricsConsumer.class, 1); ``` Or edit the storm.yaml config file: ```yaml topology.metrics.consumer.register: - class: "org.apache.storm.metrics.LoggingMetricsConsumer" parallelism.hint: 1
这表示,在config中可以通过手动注册的方式将LoggingMetricsConsumer注册上去,第二个参数为并行度:
config.registerMetricsConsumer(LoggingMetricsConsumer.class, 2);
此时Config对象(类似HashMap)会将topology.metrics.consumer.register属性注册,记录其class, parallelism.hint并行度,以及argument参数。
在Config中注册后,通过内置的特殊Bolt:MetricConsumerBolt来执行handleDataPoints方法,其中handleDataPoints赋给的两个参数taskInfo, dataPoints如下所示,给定了source task的一些状态,以及传输过来的汇总数据:
在应用后,就可以在storm的日志目录下查看到metrics日志文件:
/usr/local/apache-storm-1.0.1/logs/workers-artifacts/FirstTopo-46-1468485056/6703 -rw-rw-r-- 1 java java 55K 7月 14 18:47 gc.log.0 -rw-rw-r-- 1 java java 28K 7月 14 18:47 worker.log -rw-rw-r-- 1 java java 0 7月 14 16:31 worker.log.err -rw-rw-r-- 1 java java 1.2M 7月 14 18:47 worker.log.metrics -rw-rw-r-- 1 java java 0 7月 14 16:31 worker.log.out -rw-rw-r-- 1 java java 5 7月 14 16:31 worker.pid -rw-rw-r-- 1 java java 120 7月 14 16:31 worker.yaml
在worker.log.metrics中就可以查看到所有metrics的相关信息,注意不仅仅包含我们自定义的bolt类型,一些system类型的信息也会在上面显示出来:
2016-07-14 16:31:40,700 31721 1468485098 192.168.1.127:6702 6:bolt execute_count 5 2016-07-14 16:31:45,702 36723 1468485103 192.168.1.127:6702 6:bolt execute_count 5 2016-07-14 16:31:50,702 41723 1468485108 192.168.1.127:6702 6:bolt execute_count 5 2016-07-14 16:32:10,705 61726 1468485128 192.168.1.127:6702 6:bolt execute_count 5 2016-07-14 16:32:15,708 66729 1468485133 192.168.1.127:6702 6:bolt execute_count 5 2016-07-14 16:32:25,699 76720 1468485143 192.168.1.127:6702 6:bolt __ack-count {spout:default=60} 2016-07-14 16:32:25,701 76722 1468485143 192.168.1.127:6702 6:bolt __sendqueue {sojourn_time_ms=0.0, write_pos=10, read_pos=10, arrival_rate_secs=0.10267994660642776, overflow=0, capacity=1024, population=0} 2016-07-14 16:32:25,701 76722 1468485143 192.168.1.127:6702 6:bolt word_count {happy=18, angry=19, excited=14} 2016-07-14 16:32:25,702 76723 1468485143 192.168.1.127:6702 6:bolt __receive {sojourn_time_ms=817.6666666666666, write_pos=62, read_pos=61, arrival_rate_secs=1.222992254382389, overflow=0, capacity=1024, population=1}
相关推荐
< artifactId>storm-metrics-reporter < version>1.5.0 或者,如果您希望自己制作罐子: git clone https://github.com/staslev/storm-metrics-reporter.git cd storm-metrics-reporter mvn install 屏幕截图 ...
java运行依赖jar包
dropwizard metrics
yammer metrics-2.2.0 源码 yammer metrics-2.2.0 源码 yammer metrics-2.2.0 源码 yammer metrics-2.2.0 源码
docker load -i kube-state-metrics-v2.4.2.tar.gz docker images|grep k8s.gcr.io/kube-state-metrics/kube-state-metrics
赠送jar包:metrics-core-4.2.7.jar; 赠送原API文档:metrics-core-4.2.7-javadoc.jar; 赠送源代码:metrics-core-4.2.7-sources.jar; 赠送Maven依赖信息文件:metrics-core-4.2.7.pom; 包含翻译后的API文档:...
赠送jar包:metrics-graphite-3.1.5.jar; 赠送原API文档:metrics-graphite-3.1.5-javadoc.jar; 赠送源代码:metrics-graphite-3.1.5-sources.jar; 赠送Maven依赖信息文件:metrics-graphite-3.1.5.pom; 包含...
team which metrics it can use to objectively measure performance. You'll learn what data really counts, along with where to find it, how to get it, and how to analyze it. Because meaningful data may ...
metrics-server v0.6.1镜像包和安装文件
赠送jar包:hbase-metrics-1.4.3.jar; 赠送原API文档:hbase-metrics-1.4.3-javadoc.jar; 赠送源代码:hbase-metrics-1.4.3-sources.jar; 赠送Maven依赖信息文件:hbase-metrics-1.4.3.pom; 包含翻译后的API文档...
k8s.gcr.io/metrics-server/metrics-server:v0.4.1 components.yaml 博客地址:https://blog.csdn.net/shenhonglei1234/category_7260434.html
赠送jar包:metrics-jvm-3.1.5.jar; 赠送原API文档:metrics-jvm-3.1.5-javadoc.jar; 赠送源代码:metrics-jvm-3.1.5-sources.jar; 赠送Maven依赖信息文件:metrics-jvm-3.1.5.pom; 包含翻译后的API文档:...
docker load -i kube-state-metrics-v2.3.0.tar.gz 上传
PiggyMetrics 是springcloud的demo
赠送jar包:flink-metrics-core-1.14.3.jar; 赠送原API文档:flink-metrics-core-1.14.3-javadoc.jar; 赠送源代码:flink-metrics-core-1.14.3-sources.jar; 赠送Maven依赖信息文件:flink-metrics-core-1.14.3....
赠送jar包:metrics-core-3.1.5.jar; 赠送原API文档:metrics-core-3.1.5-javadoc.jar; 赠送源代码:metrics-core-3.1.5-sources.jar; 赠送Maven依赖信息文件:metrics-core-3.1.5.pom; 包含翻译后的API文档:...
HIS Metrics = Hersteller Initiative Software (HIS) [德国几大汽车OEM所倡议的软件] 按照一定的规则编写代码,例如MISRA-C++:2012 + HIS Metrics 软件静态测试C)
metrics-server yaml文件
赠送jar包:flink-metrics-core-1.10.0.jar; 赠送原API文档:flink-metrics-core-1.10.0-javadoc.jar; 赠送源代码:flink-metrics-core-1.10.0-sources.jar; 赠送Maven依赖信息文件:flink-metrics-core-1.10.0....
kubernetes性能监控工具metrics-server离线镜像, k8s.gcr.io/metrics-server-amd64:v0.3.6 ,使用方法:docker load -i metrics-server.tar