Event Bus

Marathon has an internal event bus that captures all API requests and scaling events. By subscribing to the event bus, you can be informed about every event instantly, without pulling. The event bus is useful for integrating with any entity that acts based on the state of Marathon, like load balancers, or to compile statistics.

Events can be subscribed to by pluggable subscribers.

The event bus has two APIs:

  • The event stream. For more information, see the /v2/events entry in the Marathon REST API Reference.

  • Deprecated The callback endpoint, which POSTs events in JSON format to one or more endpoints.

The HTTP callback endpoint support is deprecated with Marathon 1.4 and will be removed in an upcoming version. Please use the event stream instead of the callback endpoint because:

  • It is easier to set up.

  • Delivery is faster because there is no request/response cycle.

  • The events are transferred in order.

Other subscribers are easy to add. See the code in marathon/event/http for guidance.

Subscription to Events via the Callback Endpoint

Add these command line options to configure events:

$ ./bin/start --master ... --event_subscriber http_callback --http_endpoints http://host1/foo,http://host2/bar

Both host1 and host2 will receive events.

Event Types

Below are example JSON bodies that are send by Marathon.

API Request

Fired every time Marathon receives an API request that modifies an app (create, update, delete):

{
  "eventType": "api_post_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "clientIp": "0:0:0:0:0:0:0:1",
  "uri": "/v2/apps/my-app",
  "appDefinition": {
    "args": [],
    "backoffFactor": 1.15,
    "backoffSeconds": 1,
    "cmd": "sleep 30",
    "constraints": [],
    "container": null,
    "cpus": 0.2,
    "dependencies": [],
    "disk": 0.0,
    "env": {},
    "executor": "",
    "healthChecks": [],
    "id": "/my-app",
    "instances": 2,
    "mem": 32.0,
    "ports": [10001],
    "requirePorts": false,
    "storeUrls": [],
    "upgradeStrategy": {
        "minimumHealthCapacity": 1.0
    },
    "uris": [],
    "user": null,
    "version": "2014-09-09T05:57:50.866Z"
  }
}

Status Update

Fired every time the status of a task changes:

{
  "eventType": "status_update_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "slaveId": "20140909-054127-177048842-5050-1494-0",
  "taskId": "my-app_0-1396592784349",
  "taskStatus": "TASK_RUNNING",
  "appId": "/my-app",
  "host": "slave-1234.acme.org",
  "ports": [31372],
  "version": "2014-04-04T06:26:23.051Z"
}

The possible values for taskStatus are:

  • TASK_STAGING
  • TASK_STARTING
  • TASK_RUNNING
  • TASK_FINISHED
  • TASK_FAILED
  • TASK_KILLING (only when the task_killing feature is enabled)
  • TASK_KILLED
  • TASK_LOST

where the last four states are terminal.

Framework Message

{
  "eventType": "framework_message_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "slaveId": "20140909-054127-177048842-5050-1494-0",
  "executorId": "my-app.3f80d17a-37e6-11e4-b05e-56847afe9799",
  "message": "aGVsbG8gd29ybGQh"
}

Event Subscription

Fired when a new http callback subscriber is added or removed:

{
  "eventType": "subscribe_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "clientIp": "1.2.3.4",
  "callbackUrl": "http://subscriber.acme.org/callbacks"
}
{
  "eventType": "unsubscribe_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "clientIp": "1.2.3.4",
  "callbackUrl": "http://subscriber.acme.org/callbacks"
}

Health Checks

{
  "eventType": "add_health_check_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "appId": "/my-app",
  "healthCheck": {
    "protocol": "HTTP",
    "path": "/health",
    "portIndex": 0,
    "gracePeriodSeconds": 5,
    "intervalSeconds": 10,
    "timeoutSeconds": 10,
    "maxConsecutiveFailures": 3
  }
}
{
  "eventType": "remove_health_check_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "appId": "/my-app",
  "healthCheck": {
    "protocol": "HTTP",
    "path": "/health",
    "portIndex": 0,
    "gracePeriodSeconds": 5,
    "intervalSeconds": 10,
    "timeoutSeconds": 10,
    "maxConsecutiveFailures": 3
  }
}
{
  "eventType": "failed_health_check_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "appId": "/my-app",
  "taskId": "my-app_0-1396592784349",
  "healthCheck": {
    "protocol": "HTTP",
    "path": "/health",
    "portIndex": 0,
    "gracePeriodSeconds": 5,
    "intervalSeconds": 10,
    "timeoutSeconds": 10,
    "maxConsecutiveFailures": 3
  }
}
{
  "eventType": "health_status_changed_event",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "appId": "/my-app",
  "instanceId": "my-app.instance-c7c311a4-b669-11e6-a48f-0ea4f4b1778c",
  "version": "2014-04-04T06:26:23.051Z",
  "alive": true
}
{
  "appId": "/my-app",
  "taskId": "my-app_0-1396592784349",
  "version": "2016-03-16T13:05:00.590Z",
  "reason": "500 Internal Server Error",
  "host": "localhost",
  "slaveId": "4fb620fa-ba8d-4eb0-8ae3-f2912aaf015c-S0",
  "eventType": "unhealthy_task_kill_event",
  "timestamp": "2016-03-21T09:15:10.764Z"
}

Deployments

{
  "eventType": "group_change_success",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "groupId": "/product-a/backend",
  "version": "2014-04-04T06:26:23.051Z"
}
{
  "eventType": "group_change_failed",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "groupId": "/product-a/backend",
  "version": "2014-04-04T06:26:23.051Z",
  "reason": ""
}
{
  "eventType": "deployment_success",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "id": "867ed450-f6a8-4d33-9b0e-e11c5513990b"
}
{
  "eventType": "deployment_failed",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "id": "867ed450-f6a8-4d33-9b0e-e11c5513990b"
}
{
  "eventType": "deployment_info",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "plan": {
    "id": "867ed450-f6a8-4d33-9b0e-e11c5513990b",
    "original": {
      "apps": [],
      "dependencies": [],
      "groups": [],
      "id": "/",
      "version": "2014-09-09T06:30:49.667Z"
    },
    "target": {
      "apps": [
        {
          "args": [],
          "backoffFactor": 1.15,
          "backoffSeconds": 1,
          "cmd": "sleep 30",
          "constraints": [],
          "container": null,
          "cpus": 0.2,
          "dependencies": [],
          "disk": 0.0,
          "env": {},
          "executor": "",
          "healthChecks": [],
          "id": "/my-app",
          "instances": 2,
          "mem": 32.0,
          "ports": [10001],
          "requirePorts": false,
          "storeUrls": [],
          "upgradeStrategy": {
              "minimumHealthCapacity": 1.0
          },
          "uris": [],
          "user": null,
          "version": "2014-09-09T05:57:50.866Z"
        }
      ],
      "dependencies": [],
      "groups": [],
      "id": "/",
      "version": "2014-09-09T05:57:50.866Z"
    },
    "steps": [
      {
        "action": "ScaleApplication",
        "app": "/my-app"
      }
    ],
    "version": "2014-03-01T23:24:14.846Z"
  },
  "currentStep": {
    "actions": [
      {
        "type": "ScaleApplication",
        "app": "/my-app"
      }
    ]
  }
}
{
  "eventType": "deployment_step_success",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "plan": {
    "id": "867ed450-f6a8-4d33-9b0e-e11c5513990b",
    "original": {
      "apps": [],
      "dependencies": [],
      "groups": [],
      "id": "/",
      "version": "2014-09-09T06:30:49.667Z"
    },
    "target": {
      "apps": [
        {
          "args": [],
          "backoffFactor": 1.15,
          "backoffSeconds": 1,
          "cmd": "sleep 30",
          "constraints": [],
          "container": null,
          "cpus": 0.2,
          "dependencies": [],
          "disk": 0.0,
          "env": {},
          "executor": "",
          "healthChecks": [],
          "id": "/my-app",
          "instances": 2,
          "mem": 32.0,
          "ports": [10001],
          "requirePorts": false,
          "storeUrls": [],
          "upgradeStrategy": {
              "minimumHealthCapacity": 1.0
          },
          "uris": [],
          "user": null,
          "version": "2014-09-09T05:57:50.866Z"
        }
      ],
      "dependencies": [],
      "groups": [],
      "id": "/",
      "version": "2014-09-09T05:57:50.866Z"
    },
    "steps": [
      {
        "action": "ScaleApplication",
        "app": "/my-app"
      }
    ],
    "version": "2014-03-01T23:24:14.846Z"
  },
  "currentStep": {
    "actions": [
      {
        "type": "ScaleApplication",
        "app": "/my-app"
      }
    ]
  }
}
{
  "eventType": "deployment_step_failure",
  "timestamp": "2014-03-01T23:29:30.158Z",
  "plan": {
    "id": "867ed450-f6a8-4d33-9b0e-e11c5513990b",
    "original": {
      "apps": [],
      "dependencies": [],
      "groups": [],
      "id": "/",
      "version": "2014-09-09T06:30:49.667Z"
    },
    "target": {
      "apps": [
        {
          "args": [],
          "backoffFactor": 1.15,
          "backoffSeconds": 1,
          "cmd": "sleep 30",
          "constraints": [],
          "container": null,
          "cpus": 0.2,
          "dependencies": [],
          "disk": 0.0,
          "env": {},
          "executor": "",
          "healthChecks": [],
          "id": "/my-app",
          "instances": 2,
          "mem": 32.0,
          "ports": [10001],
          "requirePorts": false,
          "storeUrls": [],
          "upgradeStrategy": {
              "minimumHealthCapacity": 1.0
          },
          "uris": [],
          "user": null,
          "version": "2014-09-09T05:57:50.866Z"
        }
      ],
      "dependencies": [],
      "groups": [],
      "id": "/",
      "version": "2014-09-09T05:57:50.866Z"
    },
    "steps": [
      {
        "action": "ScaleApplication",
        "app": "/my-app"
      }
    ],
    "version": "2014-03-01T23:24:14.846Z"
  },
  "currentStep": {
    "actions": [
      {
        "type": "ScaleApplication",
        "app": "/my-app"
      }
    ]
  }
}