http_requests_total{env=" prod" ,method=" post" ,code=" 200" } 1027 http_requests_total{env=" prod" ,method=" post" ,code=" 400" } 3 http_requests_total{env=" prod" ,method=" post" ,code=" 500" } 12 http_requests_total{env=" get" ,code=" 200" } 20 http_requests_total{env=" test" ,method=" post" ,code=" 200" } 372 http_requests_total{env=" test" ,method=" post" ,code=" 400" } 75 Richard Hartmann, RichiH@{freenode,OFTC,IRCnet} http_requests_total{env=" prod" ,method=" post" ,code=" 200" } 1027 http_requests_total{env=" prod" ,method=" post" ,code=" 400" } 3 http_requests_total{env=" prod" ,method=" post" ,code=" 500" } 12 http_requests_total{env="

http_requests_total{env=" prod" ,method=" post" ,code=" 200" } 1027 http_requests_total{env=" prod" ,method=" post" ,code=" 400" } 3 http_requests_total{env=" prod" ,method=" post" ,code=" 500" } 12 http_requests_total{env=" get" ,code=" 200" } 20 http_requests_total{env=" test" ,method=" post" ,code=" 200" } 372 http_requests_total{env=" test" ,method=" post" ,code=" 400" } 75 Richard Hartmann & Frederic Branczyk @TwitchiH observability Outro Post-Mortems Mistakes happen It is important to learn from mistakes so not to repeat them To write a good incident report, there must be no fear of retribution Blame-free post-mortems allow

code="501"} 3 method_code:http_errors:rate5m{method="post", code="500"} 6 3 1 2 3 1 2 3 1 2 3 4 5 method_code:http_errors:rate5m{method="post", code="404"} 21 method:http_requests:rate5m{method="get"} method="post"} 120 ## promql method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m ## result {method="get"} 0.04 // 24 / 600 {method="post"} 0.05 600 {method="get", code="404"} 0.05 // 30 / 600 {method="post", code="500"} 0.05 // 6 / 120 {method="post", code="404"} 0.175 // 21 / 120 比如针对 method="get" 的条目，右侧