Kafka Producers View

These displays provide detailed data for all producers or for a particular producer. Clicking Kafka Producers from the left/navigation menu opens the Kafka Producers Table display, which provides a view of all of your producers and their associated metric data. The options available under Kafka Producers are:

Producers Heatmap: Opens the Kafka Producers Heatmap display, which allows you to view all producers and their associated metrics in a particular cluster.
Single Producer Summary: Opens the Kafka Single Producer Summary display, which contains current and historical metrics, as well as trend data, for a single producer.
Single Producer JVM Runtime Summary: Opens the Kafka Single Producer JVM Runtime Summary display, which contains current and historical JVM runtime metrics, as well as trend data, for a single producer.

 

 

Kafka Producers Table

The table in this display provides a view of all of your producers and their associated metric data including connection, alert level, alert count, cluster name, client ID, and the current value of each gathered metric. Each row in the table contains data for a particular producer. Click a column header to sort column data in ascending or descending order. Double-click on a table row to drill-down to the Kafka Single Producer Summary display and view metrics for that particular producer. Toggle between the commonly accessed displays by clicking the drop down list on the display title.

 

Note: Fields/columns with an asterisk (*) at the end of the field/column definition contain data that is provided by the selected cluster. Refer to KAFKA documentation for more information regarding these fields.

 

Filter By:

 

Cluster

Select the cluster for which you want to view data.

Producers

The number of producers found on the selected cluster, and that are listed in the All Producers Table.

Table:

 

Cluster

The name of the cluster.*

 

Producer

The name of the producer.

 

Alert Level

The current alert severity.

Red indicates that one or more metrics exceeded their ALARM LEVEL threshold.

Yellow indicates that one or more metrics exceeded their WARNING LEVEL threshold.

Green indicates that no metrics have exceeded their alert thresholds.

 

Alert Count

The total number of alerts for the host.

 

Client ID

The ID of the producer.*

 

Batch Size Average

The average batch size sent by the producer.*

 

Batch Size Max

The maximum number of messages that can be added to a batch before being sent to the event handler.*

 

Buffer Available Bytes

The number of available bytes in the buffer.*

 

Buffer Exhausted Rate

The average per-second number of record sends that are dropped due to buffer exhaustion.*

 

Buffer Total Bytes

The total number of bytes allowed in the buffer.*

 

Buffer Pool Wait Ratio

The fraction of time an appender waits for space allocation.*

 

Compression Avg Rate

The average compression rate of record batches.*

 

Connection Close Rate

The rate of connections being closed.*

 

Connections

The number of active connections.*

 

Connection Creation Rate

The rate of connections being created.*

 

Producer In Bytes/s

The average number of incoming bytes per second.*

 

IO Ratio

The rate of input/output operations.*

 

IO Time NS Avg

The average length of time the I/O thread spent waiting for a socket (in nanoseconds).*

 

IO Wait Ratio

The percent of time the producer was performing I/O operations while the CPU was idle.*

 

IO Wait Time ms Avg

The average length of time the I/O thread spent waiting for a socket (in milliseconds).*

 

Metadata Age

The age (in seconds) of the current producer metadata being used.*

 

Network IO Rate

The rate of input/output network operations.*

 

Producer Out Bytes/s

The average number of outgoing bytes per second.*

 

Produce Throttle Time Avg

The avg time (in milliseconds) a request was throttled by a broker.*

 

Produce Throttle Time Max

The maximum time (in milliseconds) a request was throttled by a broker.*

 

Record Errors/s

The average per-second number of record sends that resulted in errors for a topic.*

 

Avg Record Queue Time ms

The average time (in milliseconds) record batches spent in the record accumulator.*

 

Max Record Queue Time ms

The maximum time (in milliseconds) record batches spent in the record accumulator.*

 

Record Retry Rate

The average per-second number of retried record sends.

 

Producer Sent Records/s

The average number of records sent (per second) for a topic.*

 

Record Size Avg

The average record size.*

 

Record Size Max

The maximum record size.*

 

Records per Request Avg

The average number of records per request.*

 

Request Latency Avg

The average request latency (in milliseconds).*

 

Request Latency Max

The maximum request latency (in milliseconds).*

 

Request Rate

The average number of requests sent per second.*

 

Request Size Avg

The average request size.*

 

Request Size Max

The maximum request size.*

 

Requests In Flight

The current number of in-flight requests awaiting a response.*

 

Response Rate

The average number of responses received per second.*

 

Select Rate

The number of times the I/O layer checked for new I/O operations to perform per second.*

 

Waiting Threads

The number of user threads blocked waiting for buffer memory to enqueue their records.*

 

Jmx Connection

The JMX connection string.*

 

Kafka Version

The current version of Apache Kafka installed.*

 

Connected

Denotes whether or not the producer is connected.

 

Expired

When checked, performance data in the row has not been received within the time specified (in seconds) in the Expire Time field in the RTView Configuration Application > (KAFKAMON-LOCAL/Project Name) > Solution Package Configuration > Apache Kafka > DATA STORAGE > Duration > Expire Time property. The RTView Configuration Application > (KAFKAMON-LOCAL/Project Name) > Solution Package Configuration > Apache Kafka > DATA Storage > Duration > Delete Time property allows you to define the amount of time (in seconds) in which the row will be removed from the table if there is no response.

For example, if Expire Time was set to 120 and Delete Time was set to 3600, then the Expired check box would be checked after 120 seconds and the row would be removed from the table after 3600 seconds.

 

Time Stamp

The date and time the row data was last updated.

 

 

Kafka Producers Heatmap

Clicking Producers Heatmap in the left/navigation menu opens the Kafka Producers Heatmap, which provides an easy-to-view interface that allows you to quickly identify the current status of each of your producers for each available metric. You can view the producers in the heatmap based on the following metrics: the current alert severity, the current alert count, the incoming/outgoing byte rate, the IO wait time, the request latency, and the request/response rates. By default, this display shows the heatmap based on the Alert Severity metric.

Each rectangle in the heatmap represents a producer. The rectangle color indicates the most critical alert state associated with the producer. Choose a cluster from the drop-down menu to view all producers for that cluster. Mouse over a rectangle to see additional metrics.

Drill-down and investigate a producer by clicking a rectangle in the heatmap to view details in the Kafka Single Producer Summary display.

 

 

Filter By

 

Cluster

Select the cluster for which you want to view data.

Fields and Data

 

Producers

The number of producers found on the cluster and displayed in the heatmap.

 

Show Cluster

Select this check box to display the names of the clusters at the top of each rectangle in the heatmap.

Heatmap

 

Log Scale

Select this check box to enable a logarithmic scale. Use Log Scale to see usage correlations for data with a wide range of values. For example, if a minority of your data is on a scale of tens, and a majority of your data is on a scale of thousands, the minority of your data is typically not visible in non-log scale graphs. Log Scale makes data on both scales visible by applying logarithmic values rather than actual values to the data.

 

Auto Scale

Select to enable auto-scaling. When auto-scaling is activated, the color gradient bar's maximum range displays the highest value.

Note: Some metrics auto-scale automatically, even when Auto Scale is not selected.

 

Metric

Select the metric driving the heatmap display. The default is Alert Severity. Each Metric has a color gradient bar that maps values to colors. Mouse-over any rectangle to display the current values of the metrics for the producer. Click on a rectangle to drill-down to the associated Kafka Single Producer Summary display for a detailed view of metrics for that particular producer.

 

 

Alert Severity

The current alert severity. Values range from 0 - 2, as indicated in the color gradient bar, where 2 is the highest Alert Severity:

Red indicates that one or more metrics exceeded their ALARM LEVEL threshold.

Yellow indicates that one or more metrics exceeded their WARNING LEVEL threshold.

Green indicates that no metrics have exceeded their alert thresholds.

 

 

Alert Count

The total number of critical and warning unacknowledged alerts in the adapters. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the maximum count of alerts in the heatmap. The middle value in the gradient bar indicates the average alert count.

 

 

Incoming Bytes/s

The rate of incoming bytes (per second). The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaProducerIncomingByteRate. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Outgoing Bytes/s

The rate of outgoing bytes (per second). The color gradient bar shows the range of the value/color mapping. ated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaProducerOutgoingByteRate. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Avg IO Wait Time Avg ms

The average length of time the IO thread spent waiting for a socket (in milliseconds). The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaProducerIoWaitTimeMS. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Avg Request Latency

The average amount of time between when a producer is called and when the producer receives a response from the broker. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaProducerRequestLatency. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Requests/s

The average number of requests sent per second. The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaProducerRequestRate. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

Responses/s

The average number of responses received (per second). The color gradient bar, populated by the current heatmap, shows the value/color mapping. The numerical values in the gradient bar range from 0 to the defined alert threshold of KafkaProducerResponseRate. The middle value in the gradient bar indicates the middle value of the range.

When Auto Scale is checked, the numeric values in the color gradient bar show the range of the data being displayed rather than the default values. The middle value changes accordingly to indicate the color of the middle value of the range.

 

 

 

Kafka Single Producer Summary

Clicking Single Producer Summary in the left/navigation menu opens the Kafka Single Producer Summary display, which provides a view of the current and historical metrics for a single producer.

Clicking on the information boxes at the top of the display takes you to the Kafka Producers Table display, where you can view additional producers data.

There are two options in the trend graph: Performance and JVM. In the Performance option on the trend graph, you can view trend data for the requests rate, responses rate, maximum and average request latency, outgoing kilobyte rate, and average IO wait time over a selected time range. In the JVM option on the trend graph, you can view trend data for JVM CPU percentage, maximum memory in megabytes, committed memory in megabytes, used memory in megabytes, and number of live threads over a selected time range.

Clicking the Critical/Warning link at the bottom of the display opens the Alerts Table by Component display.

 

Note: Fields/columns with an asterisk (*) at the end of the field/column definition contain data that is provided by the selected cluster. Refer to KAFKA documentation for more information regarding these fields.

 

Filter By:

 

Cluster

Select the cluster for which you want to show data in the display.

 

Producer

Select the producer for which you want to show data in the display.

Fields and Data

 

Incoming/Outgoing KB/s

The rate of incoming and outgoing bytes (kilobytes per second).*

 

Requests/Responses/s

The average number of requests sent per second, and the average number of responses received per second.*

 

Network IO Rate

The rate of input/output network operations.*

 

Avg/Max Request Latency ms

The average request latency (in milliseconds), and the maximum request latency (in milliseconds).*

 

Avg/Max Batch Size KB

The average batch size sent by the producer, and the maximum number of messages that can be added to a batch before being sent to the event handler.*

 

Avg/Max Record Queue Time ms

The average time (in milliseconds) record batches spent in the record accumulator, and the maximum time (in milliseconds) record batches spent in the record accumulator.*

Trend Graphs

Performance

Requests/s -- traces the number of requests per second.

Responses/s -- traces the number of responses per second.

Max Request Latency ms -- traces the maximum request latency (in milliseconds).

Avg Request Latency ms -- traces the average request latency (in milliseconds).

Outgoing KBs/s -- traces the rate of outgoing bytes (kilobytes per second).

Avg IO Wait ms -- traces the average length of time the I/O thread spent waiting for a socket (in milliseconds).

JVM

JVM CPU % -- traces the CPU being used by the JVM.

Max Mem MB -- traces the maximum amount of available heap.

Committed Mem MB -- traces the amount of committed heap memory.

Used Mem MB -- traces the highest amount of heap used.

Live Threads -- traces the number of live threads.

 

 

Log Scale

Select to enable a logarithmic scale. Use Log Scale to see usage correlations for data with a wide range of values. For example, if a minority of your data is on a scale of tens, and a majority of your data is on a scale of thousands, the minority of your data is typically not visible in non-log scale graphs. Log Scale makes data on both scales visible by applying logarithmic values rather than actual values to the data.

 

 

Time Settings

Select a time range from the drop down menu varying from 5 Minutes to Last 7 Days. By default, the time range end point is the current time.

 

To change the time range, deselect the now toggle, which displays some additional date fields. You can click the left and right arrow buttons to decrease the end time by one time period (the time selected in the Time range drop down) per click, or you can choose the date and time from the associated calendar and clock icons. You can also enter the date and time in the text field using the following format: MMM dd, YYYY HH:MM:ss. For example, Aug 21, 2018 12:24 PM. Click the now toggle to reset the time range end point to the current time.

 

 

Client ID

The ID of the client.

Peak Threads

The highest number of threads running at one time during the current polling period.*

Critical/Warning

The number of critical and warning alerts.

Total Memory Available After GC MB

The amount of memory available after garbage collection takes place.

Last Update

The date and time of the last data update.

 

 

Kafka Single Producer JVM Runtime Summary

Clicking Single Producer JVM Runtime Summary in the left/navigation menu opens the Kafka Single Producer JVM Runtime Summary display, which provides a view of JVM runtime statistics and trend data for the selected producer.

Clicking on the information boxes at the top of the display takes you to the Kafka Producers Table display, where you can view additional producers data.

The Performance Trends trend graph provides trend data for the CPU percentage, maximum memory in megabytes, committed memory in megabytes, and the used memory in megabytes over a selected time range.

Clicking the Critical/Warning link at the bottom of the display opens the Alerts Table by Component display.

 

Note: Fields/columns with an asterisk (*) at the end of the field/column definition contain data that is provided by the selected cluster. Refer to KAFKA documentation for more information regarding these fields.

 

Filter By:

 

Cluster

Select the cluster for which you want to show data in the display.

 

Producer

Select the producer for which you want to show data in the display.

Fields and Data

 

CPU %

The percentage of CPU being used by producer.*

 

Used Memory %

The percentage of memory used by the producer.

 

Committed Mem MB

The amount of committed heap memory, in megabytes, by the producer.

 

Max Memory MB

The maximum amount of heap memory, in megabytes, defined for the producer.

 

Threads

The number of threads running on the producer.

 

Peak Threads

The peak number of threads running on the producer.

Performance Trends Graph

Traces the following:

CPU % -- traces the percentage of CPU being used by the JVM.

Max Mem MB -- traces the maximum amount of available heap, in megabytes.

Committed Mem MB -- traces the amount of committed heap, in megabytes.

Used Mem MB -- traces the amount of heap used, in megabytes.

JMX Connection

The name of the JMX connection.*

OS Version

The version number of the operating systems.*

Process Name

The name of the process.*

Start Time

The date and time when the producer was started.*

Critical/Warning

The number of critical and warning alerts.

Architecture

The type of processor being used.*

JDK

The JDK version number.*

Uptime

The amount of time the producer has been up and running.*

Operating System

The operating system installed on the producer.*

Last Update

The date and time of the last data update.