Outro Looks Prometheus http_requests_total{env=" prod" ,method=" post" ,code=" 200" } 1027 http_requests_total{env=" prod" ,method=" post" ,code=" 400" } 3 http_requests_total{env=" prod" ,method=" post" post" ,code=" 500" } 12 http_requests_total{env=" prod" ,method=" get" ,code=" 200" } 20 http_requests_total{env=" test" ,method=" post" ,code=" 200" } 372 http_requests_total{env=" test" ,method=" post" Outro Looks OpenMetrics http_requests_total{env=" prod" ,method=" post" ,code=" 200" } 1027 http_requests_total{env=" prod" ,method=" post" ,code=" 400" } 3 http_requests_total{env=" prod" ,method=" post"

system Black-box monitoring: Looking at a service from the outside (Does the server answer to HTTP requests?) White-box monitoring: Instrumention code from the inside (How much time does this subroutine Outro Exposition format http_requests_total{env=" prod" ,method=" post" ,code=" 200" } 1027 http_requests_total{env=" prod" ,method=" post" ,code=" 400" } 3 http_requests_total{env=" prod" ,method=" post" post" ,code=" 500" } 12 http_requests_total{env=" prod" ,method=" get" ,code=" 200" } 20 http_requests_total{env=" test" ,method=" post" ,code=" 200" } 372 http_requests_total{env=" test" ,method=" post"

化，那怎么 才能获取历史数据呢？可以使用 offset 关键字。 offset 后面跟一个时间段，比如 5m、1d、7d、1w，offset 要紧跟查询选择器，比如： sum(http_requests_total{method="GET"} offset 1d) 运算符 PromQL 支持基本的算术运算符和比较运算符，可以对不同的即时向量做运算，这为监控系统 带来了巨大的进步，算术运 te5m{method="post", code="404"} 21 method:http_requests:rate5m{method="get"} 600 method:http_requests:rate5m{method="del"} 34 method:http_requests:rate5m{method="post"} 120 ## promql method_cod method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m ## result {method="get"} 0.04 // 24 / 600 {method="post"} 0.05 // 6 / 120 group_left 和 group_right

predict_linear(node_filesystem_free{}[1h], 3 * 3600) < 0 异常检测 异常流量量 abs(requests - requests:holt_winters_rate1h offset 7d) > 0.3 * requests:holt_winters_rate1h offset 7d 告警规则: 预测业务A请求量量异常 异常响应 todo • 异常事件关联关系挖掘