Kafka Consumers View

These displays provide detailed data for all consumers or for a particular consumer. Clicking Kafka Consumers from the left/navigation menu opens the Kafka Consumers Table display, which provides a view of all of your consumers and their associated metric data. The options available under Kafka Consumers are:

• Consumers Heatmap: Opens the Kafka Consumers Heatmap display, which allows you to view all consumers and their associated metrics in a particular cluster.
• Single Consumer Summary: Opens the Kafka Single Consumer Summary display, which contains current and historical metrics, as well as trend data, for a single consumer.
• Single Consumer JVM Runtime Summary: Opens the Kafka Single Consumer JVM Runtime Summary display, which contains current and historical JVM runtime metrics, as well as trend data, for a single consumer.
• Single Consumer Lag Summary: Opens the Kafka Single Consumer Lag Summary display, which displays the lag per topic in a bar graph format and lists the lag per topic for the consumer.

 

Kafka Consumers Table

The table in this display provides a view of all of your consumers and their associated metric data including connection, alert level, alert count, cluster name, client ID, and the current value of each gathered metric. Each row in the table contains data for a particular consumer. Click a column header to sort column data in ascending or descending order. Double-click on a table row to drill-down to the Kafka Single Consumer Summary display and view metrics for that particular consumer. Toggle between the commonly accessed displays by clicking the drop down list on the display title.

 

Note: Fields/columns with an asterisk (*) at the end of the field/column definition contain data that is provided by the selected cluster. Refer to KAFKA documentation for more information regarding these fields.

 

Filter By:

 

Cluster

Select the cluster for which you want to view data.

Consumers

The number of consumers found on the selected cluster, which are listed in the All Consumers Table.

Table

 

Cluster

The name of the cluster.

 

Consumer

The name of the consumer.

 

Alert Level

The current alert severity.

Red indicates that one or more metrics exceeded their ALARM LEVEL threshold.

Yellow indicates that one or more metrics exceeded their WARNING LEVEL threshold.

Green indicates that no metrics have exceeded their alert thresholds.

 

Alert Count

The total number of alerts for the host.

 

Client ID

The ID of the consumer.*

 

Consumed Bytes/s

The average number of bytes consumed per second.*

 

Avg Fetch Latency

The average time taken for a fetch request.*

 

Max Fetch Latency

The maximum time taken for a fetch request.*

 

Fetch Records/s

The number of fetch requests per second.*

 

Fetch Size Avg bytes

The average number of bytes fetched per request.*

 

Max Fetch Size bytes

The maximum number of bytes fetched per request.*

 

Fetch Throttle Avg Time ms

The average throttle time in milliseconds.*

 

Fetch Throttle Max Time ms

The maximum throttle time in milliseconds.*

 

Consumed Records/s

The average number of records consumed per second.*

 

Lag

The difference between the current consumer position in the partition and the end of the log.*

 

Lag Rate

The rate of consumer lag.

 

Records per Request Avg

The average number of records in each request.*

 

JMX Connection String

The JMX connection string.*

 

Kafka Version

The current version of Apache Kafka installed.*

 

Connected

Denotes whether or not the consumer is connected.*

 

Expired

When checked, performance data in the row has not been received within the time specified (in seconds) in the Expire Time field in the RTView Configuration Application > (KAFKAMON-LOCAL/Project Name) > Solution Package Configuration > Apache Kafka > DATA STORAGE > Duration > Expire Time property. The RTView Configuration Application > (KAFKAMON-LOCAL/Project Name) > Solution Package Configuration > Apache Kafka > DATA Storage > Duration > Delete Time property allows you to define the amount of time (in seconds) in which the row will be removed from the table if there is no response.

For example, if Expire Time was set to 120 and Delete Time was set to 3600, then the Expired check box would be checked after 120 seconds and the row would be removed from the table after 3600 seconds.

 

Timestamp

The date and time the row data was last updated.

 

 

 

Kafka Consumers Heatmap

Clicking Consumers Heatmap in the left/navigation menu opens the Kafka Consumers Heatmap, which provides an easy-to-view interface that allows you to quickly identify the current status of each of your consumers for each available metric. You can view the consumers in the heatmap based on the following metrics: the current alert severity, the current alert count, the bytes consumed rate, the fetch latency average, the fetch rate, the maximum consumer lag, and the records consumed rate. By default, this display shows the heatmap based on the Alert Severity metric.

Each rectangle in the heatmap represents a consumer. The rectangle color indicates the most critical alert state associated with the consumer. Choose a cluster from the drop-down menu to view all consumers for that cluster. You can use the Show Cluster check-box to include or exclude labels in the heatmap, and you can mouse over a rectangle to see additional metrics.

Drill-down and investigate a producer by clicking a rectangle in the heatmap to view details in the Kafka Single Consumer Summary display.

 

 

Filter By:

 

Cluster

Select the cluster for which you want to view data.

Consumers

The number of consumers found on the cluster and that are displayed in the heatmap.

Show Cluster

Select this check box to display the names of the clusters at the top of each rectangle in the heatmap.

Heatmap

 

 

Log Scale

Select this check box to enable a logarithmic scale. Use Log Scale to see usage correlations for data with a wide range of values. For example, if a minority of your data is on a scale of tens, and a majority of your data is on a scale of thousands, the minority of your data is typically not visible in non-log scale graphs. Log Scale makes data on both scales visible by applying logarithmic values rather than actual values to the data.

 

Auto Scale

Select to enable auto-scaling. When auto-scaling is activated, the color gradient bar's maximum range displays the highest value.

Note: Some metrics auto-scale automatically, even when Auto Scale is not selected.

 

Metric

Select the metric driving the heatmap display. The default is Alert Severity. Each Metric has a color gradient bar that maps values to colors. Mouse-over any rectangle to display the current values of the metrics for the consumer. Click on a rectangle to drill-down to the associated Kafka Single Consumer Summary display for a detailed view of metrics for that particular consumer.

 

 

Alert Severity

The current alert severity. Values range from 0 - 2, as indicated in the color gradient bar, where 2 is the highest Alert Severity:

Red indicates that one or more metrics exceeded their ALARM LEVEL threshold.

Yellow indicates that one or more metrics exceeded their WARNING LEVEL threshold.

Green indicates that no metrics have exceeded their alert thresholds.

 

 

Alert Count

The total number of critical and warning unacknowledged alerts in the adapters. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the maximum count of alerts in the heatmap. The middle value in the gradient bar indicates the average alert count.

 

 

Consumed Bytes/s

The rate of bytes being consumed (per second). The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaConsumer. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Fetch Latency Avg

The average time taken for fetch request. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaConsumer. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Fetch Rate

The number of fetch request per second. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaConsumer. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Consumer Lag

The maximum lag in the number of records for any partition. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaConsumer. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Consumer Lag Rate

The lag rate in the number of records for any partition. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaConsumer. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Consumed Records/s

The rate of records being consumed (per second). The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaConsumer. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Kafka Single Consumer Summary

Clicking Single Consumer Summary in the left/navigation menu opens the Kafka Single Consumer Summary display, which provides a view of the current and historical metrics for a single consumer.

Clicking on the information boxes at the top of the display takes you to the Kafka Consumers Table display, where you can view additional consumers data.

There are three options in the trend graph: Performance, Consumption Rates, and JVM Resource Utilization. In the Performance option on the trend graph, you can view trend data for the JVM CPU percentage, maximum record lag, consumer lag, and fetch rate over a selected time range. In the Consumption Rates option on the trend graph, you can view trend data for the consumed kilobyte rate and the consumed records rate over a selected time range. In the JVM Resource Utilization option on the trend graph, you can view trend data for JVM CPU percentage, maximum memory in megabytes, committed memory in megabytes, and used memory in megabytes over a selected time range.

Clicking the Critical/Warning link at the bottom of the display opens the Alerts Table by Component display.

 

 

Filter By:

 

Cluster

Select the cluster for which you want to show data in the display.

 

Consumer

Select the consumer for which you want to show data in the display.

Fields and Data

 

Consumed Bytes/s

The average number of bytes consumed per second.*

 

Consumed Records/s

The average number of records consumed per second.*

 

Avg Records/Request

The average number of records in each request.*

 

Avg Fetch Records/s

The average number of fetch requests per second.*

 

Avg/Max Fetch Latency

The average time taken for a fetch request, and the maximum amount of time taken for a fetch request.*

 

Consumer Lag

The maximum lag in number of records for any partition.*

Trend Graphs

Performance

JVM CPU % -- traces the CPU percentage being used by the JVM.

Max Record Lag -- traces the maximum record lag.

Consumer Lag -- traces the lag in number of records for any partition.

Fetch Rate -- traces the fetch rate.

Consumption Rate

Consumed KB/s -- traces the rate of kilobytes being consumed, per second.

Consumed Records/s -- traces the rate of records being consumed, per second.

JVM Resource Utilization

JVM CPU % -- traces the CPU percentage being used by the JVM.

Max Mem MB -- traces the maximum amount of available heap, in megabytes.

Committed Mem MB -- traces the amount of committed heap, in megabytes.

Used Mem MB -- traces the highest amount of heap used.

 

Log Scale

Select to enable a logarithmic scale. Use Log Scale to see usage correlations for data with a wide range of values. For example, if a minority of your data is on a scale of tens, and a majority of your data is on a scale of thousands, the minority of your data is typically not visible in non-log scale graphs. Log Scale makes data on both scales visible by applying logarithmic values rather than actual values to the data.

 

 

Time Settings

Select a time range from the drop down menu varying from 5 Minutes to Last 7 Days. By default, the time range end point is the current time.

 

To change the time range, deselect the now toggle, which displays some additional date fields. You can click the left and right arrow buttons to decrease the end time by one time period (the time selected in the Time range drop down) per click, or you can choose the date and time from the associated calendar and clock icons. You can also enter the date and time in the text field using the following format: MMM dd, YYYY HH:MM:ss. For example, Aug 21, 2018 12:24 PM. Click the now toggle to reset the time range end point to the current time.

 

 

Records Lag Max

The maximum lag in number of records for any partition.*

Fetch Throttle Max Time ms

The maximum throttle time, in milliseconds.*

Critical/Warning

The number of critical and warning alerts.

Fetch Throttle Avg Time ms

The average throttle time, in milliseconds.*

Fetch Records/s

The average number of fetch requests per second.*

Fetch Size Avg bytes

The average number of bytes fetched per request.*

Max Fetch Size bytes

The highest number of bytes fetched per request.*

Last Update

The date and time of the last data update.

 

 

Kafka Single Consumer JVM Runtime Summary

Clicking Single Consumer JVM Runtime Summary in the left/navigation menu opens the Kafka Single Consumer JVM Runtime Summary display, which provides a view of the current and historical JVM Runtime metrics for a single consumer.

Clicking on the information boxes at the top of the display takes you to the Kafka Consumers Table display, where you can view additional consumers data.

The Performance Trends trend graph shows trend data for the JVM CPU percentage, maximum memory in megabytes, committed memory in megabytes, and used memory in megabytes over a selected time range.

Clicking the Critical/Warning link at the bottom of the display opens the Alerts Table by Component display.

 

 

Filter By:

 

Cluster

Select the cluster for which you want to show data in the display.

 

Consumer

Select the consumer for which you want to show data in the display.

Fields and Data

 

CPU %

The percentage of CPU used by the consumer.*

 

Used Memory %

The percentage of memory used of this consumer as JVM.

 

Committed Mem MB

The committed heap memory, in megabytes, of this consumer as JVM.

 

Max Memory MB

The maximum heap memory, in megabytes, defined for this consumer.

 

Threads

The number of threads running on this consumer.

 

Peak Threads

The peak number of threads running on this consumer.

Performance Trends Graph

Traces the following:

CPU % -- traces the CPU percentage.

Max Mem MB -- traces the maximum amount of available heap, in megabytes.

Committed Mem MB -- traces the amount of committed heap, in megabytes.

Used Mem MB -- traces the highest amount of heap used, in megabytes.

 

Log Scale

Select to enable a logarithmic scale. Use Log Scale to see usage correlations for data with a wide range of values. For example, if a minority of your data is on a scale of tens, and a majority of your data is on a scale of thousands, the minority of your data is typically not visible in non-log scale graphs. Log Scale makes data on both scales visible by applying logarithmic values rather than actual values to the data.

 

 

Time Settings

Select a time range from the drop down menu varying from 5 Minutes to Last 7 Days. By default, the time range end point is the current time.

 

To change the time range, deselect the now toggle, which displays some additional date fields. You can click the left and right arrow buttons to decrease the end time by one time period (the time selected in the Time range drop down) per click, or you can choose the date and time from the associated calendar and clock icons. You can also enter the date and time in the text field using the following format: MMM dd, YYYY HH:MM:ss. For example, Aug 21, 2018 12:24 PM. Click the now toggle to reset the time range end point to the current time.

 

 

JMX Connection

The name of the JMX connection.*

OS Version

The version number of the operating systems.*

Process Name

The name of the process.*

Start Time

The date and time when the producer was started.*

Critical/Warning

The number of critical and warning alerts.

Architecture

The type of processor being used.*

JDK

The JDK version number.*

Uptime

The amount of time the producer has been up and running.*

Operating System

The operating system installed on the producer.*

Last Update

The date and time of the last data update.

 

 

 

 

Kafka Single Consumer Lag Summary

Clicking Single Consumer Lag Summary in the left/navigation menu opens the Kafka Single Consumer Lag Summary display, which displays the lag per topic in a bar graph format and lists the lag per topic for the consumer. Double-click on a bar graph to drill-down to the Kafka Single Consumer Summary display and view metrics for that particular consumer.

Each row in the table contains data for a particular topic. Click a column header to sort column data in ascending or descending order. Toggle between the commonly accessed displays by clicking the drop down list on the display title.

 

 

Filter By:

 

Cluster

Select the cluster for which you want to show data in the display.

 

Consumer

Select the consumer for which you want to show data in the display.

Lag Per Topic Bar Graph

Displays the lag per topic in a bar graph format.

Topics for Consumer Table

 

Topic

The name of the topic.

 

Lag

The difference between the current consumer position in the partition and the end of the log.*

 

Current Lag

The difference in the amount of lag from the previous polling period to the current polling period.*

 

Lag Rate

The rate of change in the amount of lag.*

 

Log Size

The current number of messages in the log.*

 

Partitions

The number of partitions containing the topic.

 

Time Stamp

The date and time the row data was last updated.