In some situations, the output provided by Cassandra's JMX MBeans and the nodetool utility are not enough to diagnose issues. If you find yourself in a place where you need more information about the runtime behavior of a specific Cassandra node, you can increase the logging levels to get more diagnostic information on specific portions of the system.
Cassandra uses a SLF4J to provide logging functionality; by default a log4j backend is used. Many of the core Cassandra classes have varying levels of logging output available which can be increased in one of two ways:
To update the via properties file, edit conf/log4j-server.properties to include the following two lines (warning - this will generate a lot of logging output on even a moderately trafficked cluster):
This will apply the DEBUG log level to all the classes in org.apache.cassandra.db package and below as well as to the StorageProxy class directly. Cassandra checks the log4j configuration file every ten seconds, and applies changes without needing a restart.
Note that by default, logging will go to /var/log/cassandra/system.log; the location of this log file can be changed as well - just update the log4j.appender.R.File path to where you would like the log file to exist, and ensure that the directory exists and is writable by the process running Cassandra.
Additionally, the default configuration will roll the log file once the size exceeds 20MB and will keep up to 50 backups. These values may be changed as well to meet your requirements.
To change logging levels via JMX, bring up the JConsole tool and attach it to the CassandraDaemon process. Locate the StorageService MBean and find setLog4jLevel under the Operations list.
This operation takes two arguments - a class qualifier and a logging level. The class qualifier can either be a full class name or any portion of a package name, similar to the log4j-server.properties configuration above except without the initial log4j.logger appender assignment. The level must be one of the standard logging levels in use by Log4j.
In keeping with our initial example, to adjust the logging output of StorageProxy to DEBUG, the first argument would be org.apache.cassandra.service.StorageProxy, and the second one DEBUG. On a system with traffic, you should see the effects of this change immediately.