Application Deployments

Every change in the definition of applications or groups in Marathon is performed as a deployment. A deployment is a set of actions, that can:

  • Start/Stop one or more applications.
  • Upgrade one or more applications.
  • Scale one or more applications.

Deployments take time and are not instantly available. A deployment is considered active in Marathon until it has finished successfully. Marathon removes deployments after they have completed successfully.

You can perform multiple deployments at the same time as long as one application is changed only by one deployment. If you request a deployment that tries to change an application that is already being changed by another active deployment, Marathon will reject the new deployment.

Dependencies

Applications without dependencies can be deployed in any order without restriction. If there are dependencies between applications, then the deployment actions will be performed in the order the dependencies require.

In the example above, the application app is dependent on the application db.

  • Starting : If db and app are added to the system, db is started first and then app.
  • Stopping : If db and app are removed from the system, app is removed first and then db.
  • Upgrade : See the Rolling Restart section.
  • Scaling : If db and app get scaled, db is scaled first and then app.

Rolling Restarts

Marathon allows you to perform rolling restarts to deploy new versions of applications. In general, there are two phases to deploying a new version of an application: starting a set of processes with the new version and stopping the set of processes with the old version.

In Marathon, you can perform a rolling restart by defining an upgrade strategy with a minimumHealthCapacity at the application level.

The minimumHealthCapacity is a percentage which, when applied to the instance count, defines the number of healthy instances that a certain version of the application must have at all times during update. Number of healthy instances is rounded up (ceil).

  • minimumHealthCapacity == 0 : All old instances can be killed before the new version is deployed.
  • minimumHealthCapacity == 1 : All instances of the new version are deployed side by side before the old version is stopped.
  • minimumHealthCapacity between 0 and 1 : Scale the old version to minimumHealthCapacity and start the new version to minimumHealthCapacity side by side. If this is completed successfully, the new version is scaled to 100% and the old version is stopped. Number of healthy instances is rounded up (ceil). E.g. 3 instances and minimumHealthCapacity 0.7 gives us ⌈3 × 0.7⌉ = ⌈2.1⌉ = 3 so all instances will remain.

This gets a bit more complex if there are dependencies. In the example above, when the applications are updated, Marathon performs the following actions:

  1. Upgrade application db until all instances are replaced, ready, and healthy (taking upgradeStrategy into account).
  2. Upgrade application app until all instances are replaced, ready, and healthy (taking upgradeStrategy into account).

Note: Your cluster needs to have more capacity available for the update process if you choose a minimumHealthCapacity greater than 0.5. When minimumHealthCapacity is greater than 0.5, more than half of the instances of the same application are run side by side. These capacity constraints are added together if there are dependencies. In our example, we defined 0.6 for db and 0.8 for app. This means that when an update is performed there will be 12 instances of db (6 old and 6 new) and 32 instances of app (16 old and 16 new) running side by side.

Force a Deployment

An application can be changed by only one deployment at a time. Other changes to the application must wait until the first deployment has finished. You can break this rule by running a deployment with the force flag. The REST interface allows the force flag for all state-changing operations.

ATTENTION: The force flag should be used only in the case of a failed deployment!

If a force flag is set, all deployments that are affected by this deployment are cancelled. This action may leave the system in an inconsistent state. Specifically, if a deployment is cancelled when an app is in the middle of a rolling upgrade, it can end up in a state where some old and some new tasks are running. If the new deployment does not update that app, it will stay in that state until a future deployment is made for that app.

The only kind of deployments that can be force-updated safely are those that affect single apps only. The only good reason to force a deployment that affects multiple apps is to correct a failed deployment.

A failed Deployment

A deployment consists of steps executed one after the other. The next step is only executed if the previous step has finished successfully.

There are circumstances in which a step will never finish successfully. For example:

  • The new application does not start correctly.
  • The new application does not become healthy.
  • A dependency of the new application was not declared and is not available.
  • The capacity of the cluster is exhausted.
  • The app uses a Docker container and the configuration changes at Running Docker Containers on Marathon were not made.

The deployment in these cases would take forever. To heal the system, a new deployment must be applied to correct the problem with the current deployment.

The /v2/deployments endpoint

The list of running deployments can be accessed via the /v2/deployments endpoint. There are several items of information available for every deployment:

  • affectedApps: Which applications are affected by this deployment.
  • steps: The steps to perform for this deployment.
  • currentStep: Which step is currently being performed.

Every step can have several actions. The actions inside a step are performed concurrently. Possible actions are:

  • StartApplication: Start the specified application.
  • StopApplication: Stop the specified application.
  • ScaleApplication: Scale the specified application.
  • RestartApplication: Restart the specified application to the minimumHealthStrategy.
  • KillAllOldTasksOf: Kill the rest of tasks of the specified application.