Master Replacement

The steps outlined in this guide are theoretical only. Changing the IP addresses of the masters in a DC/OS cluster configured with static masters is not supported.

These steps have not been tested.

This document details how to migrate from one set of Master IPs to another set of Master IPs. Masters must be replaced one at a time; between each master replacement, all nodes must be updated to point at the new set of masters.

During/throughout this process, you must be aware of any services either external or internal to the cluster that may have hard-coded master IPs in them; for each of these, determine if you would prefer to update these settings prior to the migration, after each step of the migration, or after the migration is complete. (Ideally, minimal services should have hard-coded IPs; everything should use DNS names).

Also be aware of any changes that must be made to DNS itself.

Summary

At a high level, this process follows the following steps:

  1. Determine the correct order to replace the masters. If masters are replaced in the wrong order, you will experience outages during the migration process (you may still experience outages, but they should be minimal).
  2. Then, for each master replacement:
    1. Update the genconf/config.yaml with the new master (removing the old master)
    2. Generate a node upgrade script using dcos_generate_config.ee.sh --generate-node-upgrade-script <version>, where is the current version of DC/OS. This will automatically re-create the regular dcos_install.sh script.
    3. Install DC/OS on the new master, using dcos_install.sh master
    4. Stop DC/OS on the old master (the one that is being replaced), using sudo systemctl stop -- $(systemctl show -p Wants dcos.target | cut -d= -f2)
    5. For each existing master, complete the following, one at a time:
      1. Use the newly-generated upgrade script to upgrade the master
      2. Wait for zookeeper to settle and re-replicate (zxid should match)
      3. Wait for cockroachdb to settle and re-replicate (under-replicated should be 0)
      4. Wait for Mesos and Overlay to the replica log to re-replicate
      5. Validate that all checks work properly
      6. Optionally, wait an additional 15 minutes
      7. Continue to the next master
    6. Validate that all certificates have been properly propagated to the new master
    7. Validate that all checks complete successfully on the new master
    8. Update all of the private and public slaves to point to the new masters
      1. Validate that docker ps returns successfully
      2. Run the node upgrade script
      3. Verify that the node comes up healthy
    9. Decommission the old cockroach node
    10. Wait an hour to soak the changes
  3. Once all of the masters have been updated, ensure that all external DNS changes have been made
  4. Clean up cockroach node list
  5. Clean up navstar peer list

Determine Master Replacement Order

Determine Master Replacement Order:

The first step in replacing masters is determining the correct order in which to replace them - if you replace them in the wrong order, you risk exhibitor outages because different zk instances will have a non-matching server-id to server mapping.
Here’s the key point: we want to replace masters one at a time, following these three rules:

Every time you run the node upgrade script, a new order is generated - we don’t want this order to ever change, aside from the one node we’re changing at that time.

Note that if you replace masters out of the above order, the cluster will eventually recover, but you will have multiple outages until the cluster settles.

Exhibitor uses python’s ‘sort’ function to determine the order it will use for zk; this does a raw ASCII sort, disregarding the IP address format).

(The code used to generate this list is viewable in the calculate_exhibitor_static_ensemble command in the gen/calc.py file in the Github repository dcos/dcos: https://github.com/dcos/dcos/blob/master/gen/calc.py#L465)

  1. Determine the correct zookeeper order for the current list of IPs

  2. Determine the correct zookeeper order for the new list of IPs

  3. Choose a master to replace that is in the same position in both the new list and old list, such that replacing it will not disrupt the order (i.e., after replacing it and generating a new list, the list will still be in order). There may be multiple options here - any of them are fine, as long as they don’t violate this rule.

  4. Repeat step 3 until you come up with a valid order of masters to replace.

Assume we’re replacing IPs:: 10.10.0.236, 100.10.0.66, 10.10.0.158, 100.10.0.158, 10.10.0.149 with these IPs: 100.10.0.85, 100.10.0.147, 10.10.0.230, 10.10.0.184, 10.10.0.233 Then you’re starting with these zk server ids (as determined by python sort; note that ‘100.10.0.66’ comes after ‘100.10.0.158’ because ‘6’ is after ‘1’, and that ‘100.10.0.158’ comes after ‘10.10.0.236’ because ‘.’ comes before ‘0’ lexically, i.e., following ASCII character codes):

1: 10.10.0.149
2: 10.10.0.158
3: 10.10.0.236
4: 100.10.0.158
5: 100.10.0.66

and moving to:

1: 10.10.0.184
2: 10.10.0.230
3: 10.10.0.233
4: 100.10.0.147
5: 100.10.0.85

We can’t yet replace master 1, because that would put 1 and 2 out of lexical order:

1: 10.10.0.184 < - would come after 10.10.0.158, and generate a non-matching zk node id of 2
2: 10.10.0.158 < - would come before 10.10.0.184, and generate a non-matching zk node id of 1
3: 10.10.0.236
4: 100.10.0.158
5: 100.10.0.66

We must first replace master 2, because it doesn’t break lexical order (10.10.0.230 is after 10.10.0.149 and before 10.10.0.236):

1: 10.10.0.149
2: 10.10.0.230 < - was 10.10.0.158
3: 10.10.0.236
4: 100.10.0.158
5: 100.10.0.66

We can now replace master 1, because it will no longer break lexical order (10.10.0.184 is before the new 10.10.0.230):

1: 10.10.0.184
2: 10.10.0.230
3: 10.10.0.236
4: 100.10.0.158
5: 100.10.0.66

And so forth. After this, we can replace masters 3, 4, and 5, because this order will not change any zk server IDs.

A Python (2) script to achieve this can be found at https://github.com/justinrlee/dcos-toys/blob/master/master-replacement/order.py (Python 2 is used instead of 3 because it’s readily available on all *nix systems)