Skip to content

Troubleshooting

Following logs will help to diagnose dSource/VDB issues related to this plugin.

Cassandra dSource/VDB log files: 

/{toolkit_directory}/Delphix_COMMON_{appliance_id}/plugin/sc:cassandra_{plugin-id}/Cassandra/log_{os_user}/delphixCassandraDebug.log

Cassandra log files: Cassandra can write messages to log files. The files are named using the following format and can be found at location mentioned in cassandra-env.sh file: 

Verify CASSANDRA_LOG_DIR variable in {Cassandra_install_path}/resources/cassandra/conf/cassandra-env.sh file

For Example: /{toolkit_directory}/Delphix_COMMON_{appliance_id}/plugin/sc:cassandra_{plugin-id}/.delphix/{<binary_folder_name>_delphix_<dataset_name>_n_<node_number>}/system.log
To make it easier to watch a Cassandra startup log without knowing the full filename, a symlink with the short filename cassandra.log is also created. This symlink points to the most recent log. The Cassandra startup log location on staging or target host can be found at below location: 
{delphix_mount_path}/<dataset_name>/<binary_folder_name>_delphix_<dataset_name>_n_<node_number>.log

Troubleshooting Queries:

Identify DataStax Cassandra Version: 

{Cassandra_install_path}/bin/dse -v
For example: 
/u02/cassandra/node1/bin/dse -v
6.8.25

Identify Open-Source Apache Cassandra Version: 

{Cassandra_install_path}/bin/cassandra -v
For example: 
/u01/cassandra/apache-cassandra-4.0.7/bin/cassandra -v
4.0.7

Start DataStax Enterprise Cassandra multi-node cluster: 

{Cassandra_install_path}/bin/dse cassandra

Stop DataStax Enterprise Cassandra multi-node cluster: 

{Cassandra_install_path}/bin/dse cassandra-stop -p {pid}

Start Open-Source Apache Cassandra multi-node cluster: 

{Cassandra_install_path}/bin/cassandra

Stop Open-Source Apache Cassandra multi-node cluster: 

Terminate the active cassandra process using:
kill -9 {pid}

Start interactive Cassandra CQL shell: 

{Cassandra_install_path}/bin/cqlsh {host} {port} --username {username} --password {password}"

Start interactive Cassandra CQL shell with ssl encryption: 

{Cassandra_install_path}/bin/cqlsh {host} {port} --ssl --username {username} --password {password}"

Configure a cqlshrc file that contains CQL shell session settings that are used when the CQL shell starts.

  1. Copy the cqlshrc.sample.ssl file to the ~/.cassandra/ directory. If the Cassandra base installation directory is located at /u02/cassandra/dse-6.8.33, then the sample cqlshrc file will be located under resources/cassandra/conf/cqlshrc.sample.ssl

  2. Rename the file to cqlshrc copied in Step 1 under ~/.cassandra/cqlshrc directory.

  3. Modify or update the cqlshrc file as per your cluster configuration. For example: 
    $ cat .cassandra/cqlshrc 
[authentication]
username = dlpxadmin
password = delphix

[connection]
hostname = 10.11.12.13
port = 9042
; factory = cqlshlib.ssl.ssl_transport_factory

[ssl]
certfile = /home/delphix/certs/rootca.crt
; optional - true by default.
validate = true
userkey = /home/delphix/certs/client_key.key
usercert = /home/delphix/certs/client_cert.crt_signed
  4. Restart the CQL shell. 
    $ bin/cqlsh --ssl
Connected to datastaxstg at 10.11.12.13:9042.
[cqlsh 6.8.0 | DSE 6.8.33 | CQL spec 3.4.5 | DSE protocol v2 | TLSv1_2]
Use HELP for help.
delphixadmin@cqlsh>

Verify Cassandra Cluster Status: 

{Cassandra_install_path}/bin/nodetool status
For example:
/u02/cassandra/node1_delphix_dsestg_n_0/bin/nodetool status
Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving/Stopped
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  127.0.0.1  185.97 KiB  1            61.6%             35a2a647-f3b8-4f2d-ad25-ab8376bc12ad  rack1
UN  127.0.0.2  148.97 KiB  1            44.9%             5f1b513e-6caf-4faa-bd26-4e9344689c80  rack1
UN  127.0.0.3  148.86 KiB  1            93.5%             cf4a0f6f-ffe5-40a4-b68d-fec5e449f341  rack1

Verify Cassandra Cluster setting across all nodes: 

{Cassandra_install_path}/bin/nodetool describecluster
For example:
/u02/cassandra/node1_delphix_dsestg_n_0/bin/nodetool describecluster
Cluster Information:
    Name: dsestg
    Snitch: com.datastax.bdp.snitch.DseSimpleSnitch
    DynamicEndPointSnitch: enabled
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        a100bba8-50a7-3c2f-959d-695342f1d061: [127.0.0.1, 127.0.0.2, 127.0.0.3]

Create Cassandra User to restrict access to cassandra cluster: 

create role {username} with SUPERUSER = true AND LOGIN = true AND PASSWORD = '{password}';

For more information on common cassandra commands, please refer to Cassandra official documentation