Practicing Continuous Delivery

James Ward


  1. Continuous Delivery
  2. Intro: Play Framework, Git, Heroku
  3. Micro Apps
  4. Checkout, Build, Run
  5. One Way Deployment
  6. Zero Intervention Deployment
  7. Isolated Schema Changes
  8. Ephemeral App Tier
  9. Logs as Streams
  10. Monitor & Notify

Continuous Delivery

Goals: Deliver Faster & Reduce Risk

Variable Deployment Process


Deploy to Prod

→ → → → → → → → → → → → → → → →



Deploy to Staging

Manual Test

Continuous Delivery at Heroku

  • 4000+ Heroku apps on Heroku
  • 200+ production deploys a day
  • Java, Scala, Ruby, Python, Elang, Go
  • 85 people
  • 21 teams

Variety of Technology Options

The examples here use:

  • Play Framework with Java
  • Git
  • Heroku
  • PostgreSQL

Play Framework

Play is based on a lightweight, stateless, web-friendly architecture and features predictable and minimal resource consumption (CPU, memory, threads) for highly-scalable applications - thanks to its reactive model, based on Iteratee IO.
Play is based on a lightweight, stateless, web-friendly architecture and features predictable and minimal resource consumption (CPU, memory, threads) for highly-scalable applications - thanks to its reactive model, based on Iteratee IO.

My Top 10 Favorite Features

  • 0) Simple
  • 1) URL Routing
  • 2) Class Reloading
  • 3) Share-Nothing
  • 4) Java & Scala Support
  • 5) Great Testing Support
  • 6) JPA/EBean Support
  • 7) NIO Server (Netty)
  • 8) Asset Compiler
  • 9) Instant Deployment on Heroku

Create a new Play App

$ play new foo

Git- Distributed SCM

Create a Git Repo:

$ git init
$ git add .
$ git commit -m init


Cloud Application Platform

1,871,268+ Apps Running on Heroku

Deploying on Heroku

Provision an App

$ heroku create


$ git push heroku master

Heroku Terminology

Practice: Micro Apps

Barrier: High Friction with Monolithic Apps

Does app startup take more than 1 minute?

  • Consume cloud services
  • Service oriented apps
  • Decouple through APIs (HTTP, Pub/Sub, etc)
  • Zero friction to create & deploy

Add a JSON Service


public static Result hello() {
    ObjectNode result = Json.newObject();
    result.put("message", "hello, world");
    return ok(result);


GET     /hello                    controllers.Application.hello()


$ git commit -am "Add JSON Service"
$ git push heroku master

Practice: Checkout, Build, Run

Barrier: Creating New Environments is Difficult

Does getting a new developer setup take more than 15 minutes?

  • It works on my machine.
  • Everything needed is defined & resolvable via a build tool
  • Environments only differ through configuration

Staging Environment

Create a Staging App

$ git remote rename heroku prod
$ heroku create -r staging

Push a New Feature to Staging

$ git checkout -b newfeature
$ # make a change
$ git commit -am "new feature"
$ git push staging newfeature:master

Practice: One Way Deployment

Barrier: Unknown Source to Deployment Correlation

Can I SSH in and make a change?

  • Production changes requires source changes
  • Correlation between code and running systems

Heroku Releases

$ # make a change
$ git commit -am "a change"
$ git push prod master

$ heroku releases
$ git log
$ heroku rollback

Practice: Zero Intervention Deployment

Barrier: Complicated Deployment Procedures

Are there any credentials stored in the source repo?

  • Each environment owns their config
  • External Resources own their config

Heroku Config

$ heroku config:add FOO=bar
$ heroku config
$ # use System.getenv("FOO")
$ git commit -am "add FOO use"
$ git push heroku master

External Resources Own Their Config

$ heroku config
$ heroku addons:add memcachier
$ heroku config

Practice: Isolated Schema Changes

Barrier: Combined Schema and Codes Changes

Do schema changes and code changes happen at the same time?

  • Versioned, Roll-Backable, etc
  • Long running schema changes don't impact runtime
  • Minimum Two Phase Migrations: First DB then Code

Setup Database


"postgresql" % "postgresql" % "9.1-901-1.jdbc4"



Create local Postgres Database

$ sudo -u postgres createuser -P f
$ createdb -U f -W -h localhost f
$ export DATABASE_URL=postgres://f:f@localhost/f

Create Schema


# --- !Ups
create table bar (
    id                        varchar(255) not null,
    name                      varchar(255),
    constraint pk_bar primary key (id));
create sequence bar_seq;

# --- !Downs
drop table if exists bar cascade;
drop sequence if exists bar_seq;

Apply Schema Changes

Apply Schema Changes Locally and Test

$ psql -U f -W -h localhost f

Apply Schema Changes on Heroku and Test

$ git add conf/evolutions/default/1.sql
$ git commit -am "add schema v1"
$ git push prod master
$ heroku pg:psql 

Use the New Schema


package models;

import play.db.ebean.Model;
import javax.persistence.Entity;
import javax.persistence.Id;

public class Bar extends Model {
    @Id public String id;
    public String name;

Test then Deploy

$ git commit -am "add Bar entity"
$ git push prod master

Add a Column

Update the Schema (conf/evolutions/default/2.sql)

# --- !Ups
alter table bar add column rating integer;

# --- !Downs
alter table bar delete column rating;

Test then Deploy

Update the Code (app/models/

public Integer rating;

Test then Deploy

Practice: Ephemeral App Tier

Barrier: State in the App

Does a Chaos Monkey cause chaos?

  • Local disk and memory are not persistent stores
  • Server affinity prevents Continuous Delivery
  • All persistent state is external to the app tier

Scaling on Heroku

$ heroku scale web=50
$ heroku ps

Stateless Web Tier

  • Add Session to App
  • Deploy & Run
  • Consistency Across Instances
  • Restart Instances

Zero-Error Change-Over

Practice: Logs as Streams

Barrier: Decentralized Logging

Are logs written to local disk?

  • Externalize & Centralize Logging
  • One place to search
  • Log rotation and disk filling are barriers

Logging on Heroku

  • All STDOUT, Router Logs, Buildpack Logs goes to Logplex
  • View the logs for an app:
    $ heroku logs
  • Tail the logs for an app:
    $ heroku logs -t

Practice: Monitor & Notify

Barrier: Reactive Feedback Loop

Do your users know before you when something goes wrong?

  • Service Checks
  • Log Monitoring
  • Performance Monitoring

Service Checks

Log Monitoring


$ heroku addons:add papertrail
$ heroku addons:open papertrail

Performance Monitoring

New Relic

$ heroku addons:add newrelic
$ heroku addons:open newrelic

Further Learning