You'll need to manage the process manually. This does not work well if your brokers are heterogenous or spread across data centers. Which is fine if your brokers are homogenous and co-located. For smaller or simpler setups, Kafka can generate an assignment plan for you that provides even distribution across brokers. A trend of larger message sizes over time suggests an unintended architectural change or an undesirable side effect of a change to another service causing it to produce larger messages.Įstablish a plan for scaling and capacityĪdding capacity is easy, but re-balancing topics/partitions across brokers can be quite hard. A sudden, positive change in message size could indicate a bubble in the pipeline or a fault in an upstream component. Bigger messages have a higher cost in performance because it takes a proportionally longer time to process those messages and can cause the pipeline to back up. Using messages in / sec and bytes in / sec, you can calculate bytes / msg. If there is a change in their standard deviations, you know that a broker(s) is overloaded. Messages and bytes in tell how well balanced your traffic is. It also suggests that an organization is more vulnerable to losing data if it has a master failure.Ĭhanges in these two metrics generally lead to looking at the other three metrics. Under-replicated partitions show that replication is not going as fast as configured, which adds latency as consumers don’t get the data they need until messages are replicated. When this number goes up, even 10ms going to 20ms, end-to-end latency balloons and all Splunk Infrastructure Monitoring is affected. You should monitor for changes in the 95th percentile. Log flush latency is important because the longer it takes to flush log to disk, the more the pipeline backs up and the worse our latency and throughput. Log flush latency and under replicated partitions tend to be the leading indicators regarding what’s going on and help you prepare to investigate a new bug or regression.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |