getNewHostsWithInconsistentTopologiesAndRetry
public Set<CassandraServer> getNewHostsWithInconsistentTopologiesAndRetry(Map<CassandraServer,CassandraServerOrigin> newlyAddedHosts,
Map<CassandraServer,CassandraClientPoolingContainer> allHosts,
Duration waitTimeBetweenCalls,
Duration maxWaitTime)
Checks a set of new Cassandra servers against the current Casssandra servers
to ensure their topologies are matching. This is done to prevent user-led split-brain,
which can occur if a user accidentally provided hostnames for two different Cassandra clusters.
This is done by coming to a consensus on the topology of the pre-existing hosts,
and then subsequently returning any new hosts which do not match the present topology.
Of course, there is the base case of all hosts will be new. In this case, we simply check that all
new hosts are in consensus.
Servers that do not have support for the get_host_ids endpoint are always considered consistent,
even if we cannot come to a consensus on the hosts that do support the endpoint.
Consensus may be demonstrated independently by a set of nodes. In this case, we require that:
(1) A quorum of nodes (excluding those without `get_host_ids` support) are reachable.
(2) All reachable nodes have the same set of hostIds.
(3) All Cassandra nodes without get_host_ids support are considered to be matching.
The above should be sufficient to prevent user-led split-brain as:
(1) The initial list of servers validate that they've at least quorum for consensus of topology.
(2) All new hosts added then must match the set of pre-existing hosts topology.
Consensus may also be demonstrated and new hosts added without a quorum of nodes being reachable, if:
(4) New hosts support get_host_ids, and have the same set of hostIds as the most recent previous consensus
satisfied through conditions (1) - (3).
In this case, we know that a previous set of servers had quorum for a consensus, which we are also agreeing to.
Since we aren't agreeing on any new values, values that were agreed upon must have passed conditions (1) - (3)
at the time of their inception, and that required a quorum of nodes to agree.
There does exist an edge case of, two sets of Cassandra clusters being added (3 and 6 respectively).
On initialization, the Cassandra cluster with 6 will be used as the base case if the other 3 nodes
are down, as this will satisfy quorum requirements. However, the cluster of 6 could be the wrong
cluster, which means we're reading/writing from the wrong cluster! However, this then requires we
check all nodes, which then means we cannot handle Cassandra restarts, thus this is the best we can do.
- Parameters:
newlyAddedHosts - Set of new Cassandra servers you wish to validate.allHosts - All Cassandra servers which must include newlyAddedHosts.- Returns:
- Set of Cassandra servers which do not match the pre-existing hosts topology. Servers without
the get_host_ids endpoint will never be returned here.