Solace

The following alerts are available with both the solution package and standalone versions for Solace. Default settings for warning and alarm thresholds, duration and whether the alert is enabled (true/false) are shown.

Alert

Warning

Level

Alarm Level

Duration

Enabled

SolBridgeInboundByteRateHigh

The number of inbound bytes per second across the bridge has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerBridge

8000000

10000000

30

FALSE

SolBridgeInboundMsgRateHigh

The number of inbound messages per second across the bridge as a whole has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerBridge

40000

50000

30

FALSE

SolBridgeIngressEgressRatesUnbalanced

This alert checks whether the ratio between the ingress and egress message rates is above 1.2 or smaller than 0.8, which indicates a difference between these metrics above 20%. The duration of this condition is set by default to 30 minutes.

Index Type: PerBridge

N/A

N/A

1800

FALSE

SolBridgeOutboundByteRateHigh

The number of outbound bytes per second across the bridge has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerBridge

8000000

10000000

30

FALSE

SolBridgeOutboundMsgRateHigh

The number of outbound messages per second across the bridge has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerBridge

40000

50000

30

FALSE

SolBrokerHbaLinkDown

This alert executes when the state of an HBA Fibre-Channel link contains "Down" (e.g. "Link Down")

Index Type: PerHbaLink

NaN

0

30

FALSE

SolBrokerEgressByteRateHigh

The egress rate (bytes/sec) for the message broker is excessive.

Index Type: PerAppliance

70

85

30

FALSE

SolBrokerIngressByteRateHigh

The ingress rate (bytes/sec) for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolBrokerEgressMsgRateHigh
The message egress rate (the number of egress messages per second) for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolBrokerIngressMsgRateHigh
The message ingress rate (the number of ingress messages per second) for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolBrokerNoQueueFound

This is an Event Alert. Event Alerts do not have duration or threshold settings.

A single alarm alert when there are discarded queues in the broker. (Delta of discard-queue-not-found is non-zero).

Note: This alert cannot be executed for Cloud Brokers. This request XML is a system level request which means that Cloud login credentials do not have permission to execute it.

Index Type: PerBroker

FALSE

SolBrokerNoSubscriptionMatch

This is an Event Alert. Event Alerts do not have duration or threshold settings.

A single alarm alert when there are no current subscription matches (Delta of no-subscription-match is non-zero).

Note: This alert cannot be executed for Cloud Brokers. This request XML is a system level request which means that Cloud login credentials do not have permission to execute it.

Index Type: PerBroker

FALSE

SolBrokerNoValidDestinationThis is an Event Alert. Event Alerts do not have duration or threshold settings.

A single alarm alert when invalid destinations exist in the broker. (Delta of discard-nodest is non-zero).

Note: This alert cannot be executed for Cloud Brokers. This request XML is a system level request which means that Cloud login credentials do not have permission to execute it.

Index Type: PerBroker

FALSE

SolBrokerRedundancyDown & SolBrokerRedundancyActivityStatusChanged

These alerts only pertain to brokers that are configured for redundancy.

These alerts execute when a redundancy misconfiguration is detected. Brokers qualify as being configured for redundancy if the Monitor either detects an associated mate broker name or a broker is explicitly configured for redundancy.

To verify whether the Monitor has detected all brokers configured for redundancy, go to the Admin>Cache Table display and select the _SolBrokerRedundancy cache. Verify that all brokers that are configured for redundancy have the IsHABroker flag checked. If the IsHABroker flag is NOT checked, use the RTView Configuration Application to configure the brokers for redundancy.

The SolBrokerRedundancyDown alert verifies that redundancy is configured properly by checking whether Redundancy Mode, Redundancy Status and Configuration Status are valid. That is, the Redundancy Mode is either Active/Active or Active/Standby, the Redundancy Status is Up and the redundancy Configuration Status is Enabled. If any of these conditions are not met, then a alarm alert will be raised with the following alert text: "<hostname> is not properly configured for redundancy or redundancy is down. Redundancy Status: <a> Configuration Status: <b>", where <hostname> is the hostname of the offending broker and <a> and <b> are the current Redundancy Status and Configuration Status of the broker respectively.

The SolBrokerRedundancyActivityStatusChanged alert checks whether the previous state of the Active-Standby Role, the Activity Statusof the Primary Virtual Router and the Activity Statusof the Backup Virtual Router is different from the current state. If they are different, that implies a change in the state of the redundancy status occurred and a warning alert will be triggered. As soon as the previous and the current redundancy state is stabilized, the warning alert automatically clears, indicating in the alert text the current and previous states being detected. The warning alert contains the following text: "<hostname> has changed its redundancy activity state. There might be untracked intermediate states from the ones that have been detected. Current state: <A> Previous state: <B>", where <A> and <B> are the concatenation of active-standby-role, primary-status-activity, and backup-status-activity separated by the character "-" for current and previous states.

Best Practices & Troubleshooting

It's possible to have multiple SolBrokerRedundancyActivityStatusChanged warning alerts when failing over if intermediate states have been collected. For instance, if the changes from Local Active to Local Inactive to Shutdown are detected, then two SolBrokerRedundancyActivityStatusChanged warning alerts will be executed in this broker and will have two warnings from one broker and one from the other broker if the intermediate state on the second broker was not gathered due to polling interval being longer than the time the broker changes its redundancy state. If you only want one warning alert per broker per failover operation, the recommended action is to increase the duration of the alert. This value will vary depending on data collection latency and is system dependent. On the other hand, if you need to keep track of all intermediate states of the failover operation, then you should decrease the polling interval for the show redundancy detail poller. This is not recommended as might overflow the data collector with requests that cannot be successfully completed or preventing sending other monitoring data regarding other aspects of the broker due to the existence of requests too-often repeated.

Due to SolBrokerRedundancyActivityStatusChanged warning alert being a transient alert which will be automatically cleared when the redundancy status is stabilized, enabling both alerts is recommended as SolBrokerRedundancyDown can stay uncleared if manual intervention for fixing redundancy misconfiguration or non-functioning is required.

By default, these alerts are disabled.

SolClientInboundByteRateHigh

The number of inbound bytes per second for the client has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerClient

8000000

10000000

30

FALSE

SolClientInboundMsgRateHigh

The number of inbound messages per second for the client as a whole has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerClient

40000

50000

30

FALSE

SolClientOutboundByteRateHigh

The number of outbound bytes per second for the client has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerClient

8000000

10000000

30

FALSE

SolClientOutboundMsgRateHigh

The number of outbound messages per second for the client as a whole has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerClient

40000

50000

30

FALSE

SolClientSlowSubscriber

One or more clients are consuming messages too slowly; endpoints may drop messages!

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerClient

1

N/A

30

FALSE

SolCspfNeighborDown

State is not “OK” for one or more CSPF neighbors.

Index Type: PerNeighbor

1

N/A

30

FALSE

SolEndpointBoundIdle

Alerts when the endpoint has a bind, messages spooled, and no egress rate. This combination indicates spooled messages are not being consumed.

Index Type: PerEndpoint

N/A N/A 30 FALSE

SolEndpointNoBridgeClient

This is an Event Alert. Event Alerts do not have duration or threshold settings.

A single alarm alerts for queue endpoints associated with a bridge that have a bind-count of zero.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerEndpoint

N/A

N/A

N/A

FALSE

SolEndpointNoBridgeTopic

This is an Event Alert. Event Alerts do not have duration or threshold settings.

A single alarm alert when there are no topics subscribed to the Queue (topic-subscription-count is zero).

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerEndpoint

N/A

N/A

N/A

FALSE

SolEndpointPendingMsgsHigh

The number of pending messages on a queue has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerEndpoint

8000

10000

30

FALSE

SolEndpointQuotaUtilExceeded

This alert will execute when the endpoint has a bind, an egress rate, and the percentage of quota spooled has been reached.

Index Type: PerEndpoint

10

15

30

FALSE

SolEndpointSpoolMemoryHWMUtilHigh

The High Water Mark of system memory consumption for spooling for this endpoint is too high.

Note: The metric is the percent of high-water-mark-in-mb relative to memory-used-in-mb, found in the SolEndpoints cache.

Index Type: PerEndpoint

10

20

30

FALSE

SolEndpointSpoolMemoryUtilHigh

The endpoint is currently consuming too much system memory for spooling.

Note: This alert does not trigger for VPNs with a standby replication state.

Note: The metric is the percent of high-water-mark-in-mb relative to quota-in-mb, found in the _SolEndpoints cache.

Index Type: PerEndpoint

10

20

30

FALSE

SolEndpointSpoolUsageHigh

The endpoint is consuming too much message broker memory for storing spooled messages. (Threshold units are megabytes.)

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerEndpoint

40

50

30

FALSE

SolEndpointSuffixedHasSpoolMsgs

This alert will execute when endpoints with a specified suffix have the specified number of messages spooled. The default suffixes are: .DMQ, .ERROR, .BACKOUT, and .DO

The endings that are alerted against can be edited in the Configuration Application under Solace > Data Collection > Endpoint Filter. The value of this property can be seen in the Alert Description under the Alert Administration table

Index Type: PerEndpoint

1   30 FALSE

SolEndpointUnackMsgsHigh

Alerts when the number of delivered unacknowledged messages of this endpoint is above its threshold.

Index Type: PerEndpoint

1 10 30 FALSE

SolEventModuleBrokerAlert
This is an Event Alert. Event Alerts do not have duration or threshold settings.

If the Solace Event Module is properly configured and running and this alert is enabled, all Syslog Events that are selected as alerts from the Message Brokers that were enabled for being monitored with Syslog will trigger this type of alert from the SYSTEM scope. Alerts of this type refer to Syslog events that can be clearable and non-clearable of SYSTEM scope. Therefore this alert can be clearable and non-clearable, depending on the event that triggered its execution.

FALSE

SolEventModuleClientAlert
This is an Event Alert. Event Alerts do not have duration or threshold settings.

If the Solace Event Module is properly configured and running and this alert is enabled, all Syslog Events that are selected as alerts from the Message Brokers that were enabled for being monitored with Syslog will trigger this type of alert from the CLIENT scope. Alerts of this type refer to Syslog events that can be clearable and non-clearable of CLIENT scope. Therefore this alert can be clearable and non-clearable, depending on the event that triggered its execution.

FALSE

SolEventModuleVpnAlert
This is an Event Alert. Event Alerts do not have duration or threshold settings.

If the Solace Event Module is properly configured and running and this alert is enabled, all Syslog Events that are selected as alerts from the Message Brokers that were enabled for being monitored with Syslog will trigger this type of alert from the VPN scope. Alerts of this type refer to Syslog events that can be clearable and non-clearable of VPN scope. Therefore this alert can be clearable and non-clearable, depending on the event that triggered its execution.

FALSE

SolGuaranteedMsgingHbaLinkDown

This alert executes when the broker is configured for Guaranteed Messaging and the state for an HBA Fibre-Channel contains "Down" (e.g. "Link Down")

Index Type: PerHbaLink

N/A

N/A

30

FALSE

SolGuaranteedMsgingMatePortDown

This alert executes when the broker is configured for Guaranteed Messaging and the ADB Link to Mate is down.

Note: Note that this alert does not clear until a fail-over occurs.

Index Type: PerADB

N/A

N/A

30

FALSE

SolGuaranteedMsgingNoMsgSpoolAdActive

This alert applies to a pair of brokers that are configured for redundancy as an HA pair. A single alert executes when neither broker in the HA pair has a message spool operational status of AD-Active.

Index Type: PerMsgRouter

N/A

N/A

30

FALSE

SolMsgRouterActiveDiskUtilHigh

The utilization of the active disk partition for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterConnectionUtilHigh

The connection utilization for the message broker (current number of connections divided by max allowed) is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterCpuTemperatureHigh

CPU temperature margin is above threshold.

Index Type: PerMsgRouterSensor

-30

-15

30

FALSE

SolMsgRouterCspfNeighborDown

Link-detect = no for CSPF neighbor.

Index Type: PerNeighbor

1

N/A

30

FALSE

SolMsgRouterDelvrdUnAckMsgUtilHigh

The delivered unacked messages as a percentage of all messages delivered for the application is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterExpired
This is an Event Alert. Event Alerts do not have duration or threshold settings.

The collection of monitoring data has stopped abruptly. Use this alert in conjunction with the SolMsgRouterNotConnected alert, or instead of it, if you don't need to be notified about a lack of connection when the Monitor starts up.

N/A

N/A

30

FALSE

SolMsgRouterFanSensorCheckFailed

The speed of one or more fans is low. (The default alert settings match the broker factory settings and should not be changed).

Note: The alert will be triggered when the fan speed drops below the low alert level (2657) and will be cleared when it exceeds 5000. This behavior matches the broker's on board fan speed alert behavior.

Index Type: PerMsgRouterSensor

-

2657

30

FALSE

SolMsgRouterInboundByteRateHigh

The number of inbound bytes per second for the message broker has reached its max threshold.

Index Type: PerMsgRouter

400000

500000

30

FALSE

SolMsgRouterInboundMsgRateHigh

The number of inbound messages per second for the message broker has reached its max threshold.

Index Type: PerMsgRouter

400000

500000

30

FALSE

SolMsgRouterIngressFlowUtilHigh

The ingress flow utilization (current flows divided by max allowed) for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterInterfaceDown

Link-detect = no for one or more enabled network interfaces.

Index Type: PerSolInterface

N/A

N/A

30

FALSE

SolMsgRouterMsgCountUtilHigh

The message count utilization for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterNABUsageHigh

Network Acceleration Blade memory usage is excessive.

Index Type: PerNAB

60

80

30

FALSE

SolMsgRouterNotConnected
The message broker is not ready for collecting performance monitoring data.

Index Type: PerMsgRouter

N/A

N/A

30

FALSE

SolMsgRouterOutboundByteRateHigh
The number of outbound bytes per second for the message broker has reached its max threshold.

Index Type: PerMsgRouter

400000

500000

30

FALSE

SolMsgRouterOutboundMsgRateHigh
The number of outbound messages per second for the message broker has reached its max threshold.

Index Type: PerMsgRouter

400000

500000

30

FALSE

SolMsgRouterPendingMsgsHigh
The total number of pending messages for this message broker has reached its maximum.

Index Type: PerMsgRouter

400000

500000

30

FALSE

SolMsgRouterPendingMsgsUtilHigh
The pending message count for this message broker is too high.

Note: The metric is the percent of num-messages-spooled relative to max-message-count-multiplied, found in the SolAppliances cache.

Index Type: PerMsgRouter

1

2

30

FALSE

SolMsgRouterPowerSupplyFailed
A power supply has failed.

Index Type: PerMsgRouter

0

N/A

30

FALSE

SolMsgRouterSpoolUtilization
The percentage of spool spaces used for storing spooled messages is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterStandbyDiskUtilHigh
The utilization of the standby disk partition for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterSubscriptionUtilHigh
The Topic Subscriptions on Queue utilization (current number of subscriptions divided by max allowed) for the message router is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterSwapUsedHigh
This alert strictly applies to software brokers (it does not execute for hardware brokers). The amount of swap space used by the message broker operating system is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterTemperatureSensorCheckFailed
A chassis temperature measurement is above threshold.

Index Type: PerMsgRouterSensor

40

45

30

FALSE

SolMsgRouterTranSessionCntUtilHigh
The transacted session count utilization for the message broker is excessive. The metrics are:

(transacted-sessions-used/ max-transacted-sessions)*100

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterTranSessionResUtilHigh
The transacted session resource utilization for the message broker is excessive.

Index Type: PerMsgRouter

70

85

30

FALSE

SolMsgRouterVoltageSensorCheckFailed
A power supply voltage is high or low.

Index Type: PerMsgRouterSensor

N/A

N/A

30

FALSE

SolSparseMessageSpoolFile
This is a Limits Alert that issues a Warning alert and is enabled by default.

Important: Do not modify thresholds for this alert as are set by Solace development.

A single warning alert (Severity 1) executes when the active-disk-partition-usage > 30.0

AND

disk-usage-mb/current-disk-usage >= 3.0.

This alert is defined to determine when there is a Sparse Message Spool File Condition. When disk space usage is several multiples of persistent store usage, then there is likely a large number of message spool files residing on the disk where each file contains few messages. This is referred to as a sparse message spool file condition, and requires urgent attention to mitigate and avoid the disk reaching capacity. For further information, refer to Solace documentation for diagnosing the sparse message spool file condition.

TRUE

SolVpnAmqpConnectionCountUtilHigh

The %age utilization of connections for the AMQP service in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnConnectionCountHigh

The number of connections to the server has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

60

80

30

FALSE

SolVpnConnectionCountUtilHigh

The utilization of VPN connections to this server is high.

Note: This alert does not trigger for VPNs with a standby replication state.

Note: The metric is the percent of connections relative to max-connections, found in the SolVpns cache.

Index Type: PerVPN

1

2

30

FALSE

SolVpnEndpointCountUtilHigh

The %age utilization of endpoints in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnInboundByteRateHigh

The number of inbound bytes per second for the VPN has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

8000000

10000000

30

FALSE

SolVpnInboundDiscardRateHigh

The number of discarded inbound messages per second for the server is excessive.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

1

5

30

FALSE

SolVpnInboundMsgRateHigh

The number of inbound messages per second for the VPN as a whole has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

40000

50000

30

FALSE

SolVpnMqttConnectionCountUtilHigh

The %age utilization of connections for the MQTT service in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnOutboundByteRateHigh

The number of outbound bytes per second for the VPN has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

8000000

10000000

30

FALSE

SolVpnOutboundDiscardRateHigh

The number of discarded outbound messages per second for the server is excessive.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

1

5

30

FALSE

SolVpnOutboundMsgRateHigh

The number of outbound messages per second for the server as a whole has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

40000

50000

30

FALSE

SolVpnPendingMsgsHigh

The total number of pending messages for this destination has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

8000000

10000000

30

FALSE

SolVpnRestInConnectionCountUtilHigh

The %age utilization of connections for the REST incoming service in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnRestOutConnectionCountUtilHigh

The %age utilization of connections for the REST outgoing service in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnSmfConnectionCountUtilHigh

The %age utilization of connections for the SMF service in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnSpoolMemoryUtilHigh

The utilization of system memory for spooling by this VPN is high.

Note: This alert does not trigger for VPNs with a standby replication state.

Note: The metric is the percent of current-spool-usage-mb relative to maximum-spool-usage-mb, found in the SolVpnSpoolStats cache.

Index Type: PerVPN

70

85

30

FALSE

SolVpnSubscriptionCountHigh

The number of endpoints in this VPN has reached its maximum.

Note: This alert does not trigger for VPNs with a standby replication state.

Index Type: PerVPN

8000

10000

30

FALSE

SolVpnSubscriptionsUtilHigh

The %age utilization of subscriptions in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE

SolVpnUsernameConnectionsHigh

The number of open connections of the client username in the alert text has reached its threshold.

Index Type: PerVPN

20 50 30 FALSE

SolVpnWebConnectionCountUtilHigh

The %age utilization of connections for the Web service in this VPN is high.

Index Type: PerVPN

70

85

30

FALSE