DataStax OpsCenter Documentation

Configuring events and alerts

The OpsCenter Event Log page displays a continuously updated list of events and alerts. The following list reflects the most detailed logging level available for Cassandra, DataStax Enterprise, and OpsCenter events. The available levels are DEBUG(0), INFO (1), WARN (2), ERROR (3), CRITICAL (4), ALERT (5).

Data for these events is stored in the events and events_timeline column families in the OpsCenter keyspace:

Event Code Description
COMPACTION 0 Major compaction has occurred.
CLEANUP 1 Unused keys have been removed or cleaned up.
REPAIR 2 A repair operation has been initiated.
FLUSH 3 Memtables have been flushed to disk.
DRAIN 4 The commit log has been emptied, or drained.
DECOMMISSION 5 A leaving node has streamed its data to another node.
MOVE 6 Like NODE_MOVE; a new token range has been assigned.
NODE_DOWN 13 A node has stopped responding.
NODE_UP 14 An unresponsive node has recovered.
NODE_LEFT 15 A node has left, or been removed from, the ring.
NODE_JOIN 16 A node has joined the ring.
NODE_MOVE 17 A node has been assigned a new token range (the token has moved).
OPSC_UP 18 OpsCenter has been started and is operating.
OPSC_DOWN 19 OpsCenter was stopped or stopped running.
GC 20 Java garbage collection has been initiated.

Optionally, you can configure OpsCenter to send alerts for selected levels of events. These alerts can be provided remotely by email, or through HTTP to a selected URL. Alerts are disabled by default.

Note

Alerts are triggered only by events from the OpsCenter API/UI. For example, a nodetool move operation submitted from the command line does not trigger an alert. However, a move operation launched using Dashboard > List View > Actions > Move controls in the OpsCenter does trigger an alert.

All alerts contain the following information about each event captured:

Field Description Example
api_source_ip IP that originally sent the request. 67.169.50.240
target_node Destination of a STREAMING action. 50.1.1.11
event_source Component that caused the event. OpsCenter (i.e., restart, start)
user OpsCenter user that caused the event. opscenter_user
time Normal timestamp for the event. 1311025650414527
action Type of event (see above table) 20
message Description of the event. Garbage Collecting node 50.1.1.13
level Numerical code for the log level. 1
source_node Node where the event originated. 50.1.1.13
level_str Logging level of the event. INFO

Enabling email alerts

OpsCenter can post alerts to selected email addresses. To enable email alerts, you must edit the email.conf file and provide valid SMTP server host and port information. This file is located in the following directories:

  • Package installations: /etc/opscenter/event-plugins
  • Binary tarball installations (Linux and Mac OSX): <install_location/opscenter/conf/event-plugins
  • Windows installations: Program Files (x86)\DataStax Community\opscenter\conf\event-plugins

To enable email alerts:

  1. Make sure that you have valid SMTP mail accounts to send and receive alerts.
  2. On the OpsCenter daemon host, open the email.conf file for editing.
  3. Set enabled to 1.
  4. Provide valid values for your SMTP host, port, user, and password.
  5. Enable Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocol on your system if you want secure communications. Typically, SSL is required.
  6. Provide valid values for the to_addr and from_addr email addresses. The to_addr value is the account that will receive alerts.
  7. Optionally, set the level of alerts to send and the desired subject line.
  8. Save email.conf and restart the OpsCenter daemon.

For example, in a system with email alerts enabled for critical and alert-level events,:file:email.conf looks like:

[email]
enabled=1
# levels can be comma delimited list of any of the following:
# DEBUG,INFO,WARN,ERROR,CRITICAL,ALERT
# If left empty, will listen for all levels
levels=CRITICAL,ALERT
smtp_host=smtp.gmail.com
smtp_port=465
smtp_user=mercury@gmail.com
smtp_pass=*********
smtp_use_ssl=1
smtp_use_tls=0
to_addr=cassandra_admin@acme.com
from_addr=mercury@gmail.com
subject=OpsCenter Event

To enable email alerts to multiple addresses:

Create a different email conf file with settings for each email address. All conf files are loaded so you can name them email1.conf, email2.conf, and so on.

Enabling alerts posted to a URL

OpsCenter can post alerts to a URL if you provide a correctly formatted POST script. For example, a simple PHP script containing print_r($_POST); should be sufficient for getting started.

To enable URL posting on the OpsCenter side, you must edit posturl.conf and provide a path to your script. This file is located in the following directories:

  • Package installations: /etc/opscenter/event-plugins
  • Binary tarball installations (Linux and Mac OSX): <install_location/opscenter/conf/event-plugins
  • Windows installations: Program Files (x86)\DataStax Community\opscenter\conf\event-plugins

To enable URL posting:

  1. Make sure your web server and posting script are configured to receive alerts.
  2. On the OpsCenter daemon host, open posturl.conf for editing.
  3. Set enabled to 1.
  4. For url, provide a valid path to your posting script. For example: url=http://50.1.1.11/postOPSCevents.php.
  5. Optionally, select the desired logging level. The default is to listen for all levels of events.
  6. Save posturl.conf and restart the OpsCenter daemon.

In a system with posting enabled for critical and alert-level events, posturl.conf looks like:

[posturl]
enabled=1
url=http://50.1.1.11/postOPSCevents.php
# levels can be comma delimited list of any of the following:
# DEBUG,INFO,WARN,ERROR,CRITICAL,ALERT
# If left empty, will listen for all levels
levels=CRITICAL,ALERT

You can set preferences to specify how the posting is handled on the receiving side.

To verify that events are posting correctly:

  1. Post events to a file such as /tmp/events on the web server host.

  2. Create a script. For example, create a PHP script http://50.1.1.11/postOPSCevents.php:

    <?php
      file_put_contents('/tmp/events', print_r($_POST,true), FILE_APPEND);
    ?>
    
  3. Deploy the script. You might need to restart the web server.

  4. Launch a logged event, such as an OpsCenter restart or garbage compaction from Dashboard > Cluster > List View. Output to /tmp looks something like this:

    Array
    (
      [api_source_ip] => 67.169.50.240
      [target_node] => None
      [event_source] => OpsCenter
      [user] => None
      [time] => 1311025598851602
      [action] => 20
      [message] => Garbage Collecting node 50.1.1.24
          [level] => 1
          [source_node] => 50.1.1.24
          [level_str] => INFO
    )