We can use the following steps to remove a datacenter in the Cassandra cluster.
- Check the network connection for TCP port 9042 with this command. netstat -anpl|grep 9042. This is to make sure all the clients are not connecting to this DC.
- Run a full repair for the data
- Check the replication factor for all keyspaces cqlsh> SELECT * from system.schema_keyspaces ;
- alter keyspace replication level to remove the decommissioning dc. ALTER KEYSPACE system_auth WITH replication = {‘class’: ‘NetworkTopologyStrategy’, ”:” } ;
- Run nodetool decommission on every node in the decommissioning dc
Table of Contents
How long time it takes for a datacenter removal?
On the bright side, the operation should be almost instantaneous (at least very quick), because this datacenter is not owning any data anymore from a Cassandra perspective. Thus, there should be no data to stream to other nodes. If streaming happens, we probably forgot about a keyspace using SimpleStrategy or NetworkTopologyStrategy that still uses the old data center. We should check it again.
Get Your Free Linux training!
Join our free Linux training and discover the power of open-source technology. Enhance your skills and boost your career! Learn Linux for Free!This should be fast, not to say immediate as this command should trigger no streaming at all due to the changes we made in the keyspaces replication configuration. This data center should not own any token ranges anymore as we removed the data center from all the keyspaces, in the previous step.
Tips of removing Datacenter
One thing to note is that decommission and removetoken both create quite a bit of load on the other data centers in the cluster. It wasn’t enough to noticeably move the read/write response times but it was enough that we wouldn’t do a bunch of nodes all at once. We can put them in a loop with a 10 minute sleep in between. Sure the script takes a long time but safety first in production.