We (Alchemy Video) found ourselves in an interesting position a few months ago. One of our Video SSP (sell side platform) clients required a strong presence in Australia for the launch of a new advertising product.

At the time we were heavily using Digital Ocean to deliver all the core assets mainly in EMEA / North America. We had written our own provider specific autoscaling technology which interfaced via Digital Ocean's API. With no presence in Australia, the closest region Digital Ocean had to serve from was Singapore - this was the equivalent of serving content from New York to London. The latencies are very noticeable and the peering between Singapore and Australia is not great either.

The solution was to abstract our provider specific auto-scaling technology, but this was a little harder than expected. The first challenge was to rethink our entire build process. On Digital Ocean we created a single instance from a stock image (Ubuntu) and then configured this, creating a template which we could then image and distribute over required regions and subsequently manage scaling up and down via the API. This type of set up would obviously not scale across multiple providers. We needed consistency in our image builds across all providers. We had already scripted our image builds in Python, but stock OS distribution builds differ between providers and not all providers enable the upload of a custom image. In the case of Digital Ocean, the response was:

You can create a snapshot of your machine and then use that snapshot as a base image for the new droplet.

After some time trying out a number of possible solutions we finally found ourselves seriously considering Docker. The concept of containers was a little bizarre at first and there were concerns regarding load overhead from what is effectively a multi-virtualization environment. Digging deeper into docker we found it was incredibly well written and thought out - there were still limitations (I will cover this in another blog post) and added complexity in managing and deploying, but we ended up with a set up that could be run in any environment we wanted.

Docker enabled us to build in stages, we could take the stock Ubuntu image and customize it. We have a bespoke Ubuntu build using FFMpeg and this takes around two hours to compile with all the necessary libraries and configuration options - this is not something you wish to carry out on each build! Using Docker Hub we could easily version, share and commit these builds and manage multiple images across several private repositories.

Docker Repos

Our build script for docker was relatively straight forward: -

#Use Latest FFMPEG Ubuntu Build (as base image)
FROM venatus/ubuntu-ffmpeg:latest

MAINTAINER Chris Beck <chrisb@venatusmedia.com>

ENV DEBIAN_FRONTEND noninteractive

# Update, Upgrade and Install Basics
RUN apt-get -y update && apt-get -y upgrade && apt-get -y install vim git apt-utils supervisor curl memcached cron wget

# Install PHP
RUN apt-get -y install php7.0-fpm php7.0-cli php-pear php7.0-curl php7.0-dev php7.0-gd php-memcached php-xdebug php-mbstring

# Configure PHP FPM
RUN sed -i -e "s/;daemonize\s*=\s*yes/daemonize = no/g" /etc/php/7.0/fpm/php-fpm.conf  
RUN sed -i -e "s/;catch_workers_output\s*=\s*yes/catch_workers_output = yes/g" /etc/php/7.0/fpm/php-fpm.conf

# Install Nginx
RUN apt-get -y install nginx-full

# Add Start Script
ADD scripts/start.sh /start.sh  
RUN chmod 755 /start.sh

# ...
# Futher Configuration ...
# ...

#Run Start Script At Boot
CMD ["/bin/bash", "/start.sh"]  

Our start script enabled us to perform a final post-launch configuration. Every provider we run on has slightly different hardware underneath and there are tweaks that need to be done at boot - Example of ./start.sh: -

#!/bin/bash

# ...

# Optimize nginx workers
sed -i -e "s/worker_processes 1/worker_processes $(cat /proc/cpuinfo |grep processor | wc -l)/" /etc/nginx/nginx.conf

# ...

This is just a quick example that allows us to set worker processes of Nginx to number of available cores of host machine before we start our main Nginx process.

$ -> docker images
REPOSITORY              TAG                 IMAGE ID  
alchemy-edge            2.8                 2c45c5e36fab  
alchemy-edge            latest              2c45c5e36fab  
alchemy-edge            2.7                 d9e46056613c  
...
venatus/alchemy-edge    latest              eec4aec4791e  
venatus/ubuntu-ffmpeg   latest              2c179b8c0ff9  

With our docker containers built, tested and ready to deploy we had to create docker host images for all providers. For spin-up time on these providers, we didn't want to pull from docker hub for each boot - we needed to spin-up in under 60 seconds to meet demand. Our provider build scripts were quickly written in Python using the providers API SDK's. Each script differs considerably as there is no standard process for spinning, configuring host machine template, imaging and distributing over multiple regions.

Here is our build script for Digital Ocean:-

...

DO_TOKEN = '<token>'  
DO_BUILD_REGION = '<build-region>'  
DO_BUILD_FROM_SNAPSHOT = '<base-snapshot>'  
DO_BUILD_SIZE = '512mb'  
DO_SSH_KEY_ID = '<ssh-key>'

@command('build')
def build(version):  
    """Build Image.
    Arguments:
      version - Version.
    """

    ...

    rtn = client.droplets.create( ... )

    running = False
    instanceIp = None
    while(not running):
        ...

    isOnline = False
    while(not isOnline):
        if(configure.hasAccess(instanceIp,user='root')):
            ...

    if(configure.configure(instanceIp)):
        isPowerOff = False
        while(not isPowerOff):
            ...
                client.droplets.power_off(dropletId)


        client.droplets.snapshot( ... )
        ...

        while(not snapshotCreated):
            ...

        for region in regions:
            if(region != DO_BUILD_REGION):
                client.images.transfer(snapshotId, region)

        print('Droplet snapshot id: '+ str(snapshotId))

    else:
        print('Server -> Failed To Configure Server @ '+instanceIp)


if __name__ == '__main__':  
    Run()

The configure script referenced here is shared between all providers and logs-in to the new machine via SSH and configures the server once it comes online.

...
commands.append(["docker", "pull", "venatus/alchemy-edge"])  
commands.append(["docker", "run", "--restart=always", "-p", "80:80", "-p", "443:443", "-e", "PENV=Live", "-d", "-t", "venatus/alchemy-edge:latest"])  
...

Once we have the snapshot id for the build, this can be added to our bespoke auto-scaling provider config file: -

{
  "regions": {
    ...
    "lon": {
      "state": "active",
      "zones": {
        "lon1": {
          "state": "active",
          "min_servers": 10,
          "max_servers": 100
        }
      },
      "name": "London"
    },
    "nyc": {
      "state": "inactive",
      "zones": {
        "nyc3": {
          "state": "active",
          "min_servers": 10,
          "max_servers": 100
        }
      },
      "name": "New York"
    }
    ...
  },
  "image_id": "<snapshot-id>",
  ...
}

Finally we implement our required provider interface:-

<?php  
namespace project\lib\cdn;  
class DigitalOcean implements \project\lib\cdn\ProviderTemplate {  
  ...
  public function spinDownNode($nId){..}
  public function spinUpNodeWithImage($id, $imgId, $rgn, ..){..}
  public function getPublicIPAddress($nId){..}
  public function isImageAvailable($rgn, $imgId){..}
  ...
}

For us as a very small development team, responsible for managing hundreds of servers across several providers with 30+ regions, the only way to achieve such a goal was to give our system near-full autonomy, only working within certain limits to achieve a healthy service.

For it to work effectively it had to implement the following:-

  • Cycle old servers. If there was a newer image, phase the old servers out gradually and introduce new ones in every region over a period of time with no impact on service in any region.
  • Check the health of each machine. Decide what a healthy machine looked like. The server was responsible for providing a private page detailing information that could be used to measure health.
  • Apply minimum number of nodes in each region as per the provider configuration files
  • Auto-scale in each region independently. For this it had to understand combined server data (load, requests per second, request latencies, network IO, disk IO etc).
  • Check new nodes and make sure they are coming online and are correctly configured
  • Terminate any requested nodes - These could be nodes that are no longer required, but making sure the node is safe to terminate (logs sent to S3 etc.)

The main auto-scaling class (methods):-

...
const SCALE_UP_LOAD_GREATER_THAN = 0.7;  
const SCALE_DOWN_LOAD_LESS_THAN = 0.3;  
const UNHEALTHY_LOAD_GREATER_THAN = 4;  
const SCALE_UP_INCREASE_SERVERS_BY = 2;  
const SCALE_DOWN_DECREASE_SERVERS_BY = 1;  
const TIME_BETWEEN_AUTO_SCALE_EVENTS_SECS = 60;  
const FAILURES_IN_PERIOD = 30;  
const FAILURE_ROLLING_PERIOD_HOURS = 6;  
const AUTO_SCALE_LOAD_AVERAGING_MINS = 10;  
const SCALE_UP_NEW_CACHE_REQUESTS = 10;  
const SCALE_UP_NEW_CACHE_REQUEST_SERVERS_BY = 1;  
const FORCE_TERMINATE_AFTER_SECONDS = 60;  
...
private static $logs = [];  
private static $configs = [];  
private static $providers = [  
  'aws' => 'Amazon AWS',
  'digitalocean' => 'Digital Ocean',
  ...
];

final private function __construct(){}  
final private function __clone(){}

private static function log(..){..}  
private static function getConfig(..){..}  
private static function setLocationForNode(..){..}  
private static function createDNSRecordsForNode(..){..}  
public static function configureNewNodes(..){..}  
private static function spinUpIn(..){..}  
private static function requestTermination(..){..}  
private static function terminateNode(..){..}  
private static function applyRegionNodeMinimumsFor(..){..}  
private static function updateLatestImageVersion(..){..}  
public static function cycleOld(..){..}  
public static function applyRegionNodeMinimums(..){..}  
public static function checkNewNodes(..){..}  
private static function maxRemoveNodesInZone(..){..}  
public static function autoScaleInZones(){..}  
private static function spinNodeDown(..){..}  
public static function terminateRequestedNodes(..){..}  
public static function requestTerminateAll(..){..}  
private static function retireCheck(..){..}  
public static function healthCheck(..){..}  

We used the health check to provide detailed data around each node and with far greater granularity than AWS and easy access to our data, we are able to make far more accurate decisions about nodes in each region/zone within our network. All the raw data is currently pumped into MongoDB for direct access and Google's BigQuery for dashboard creation.

There will be a follow up post in the coming weeks to describe how we routed traffic from the client to the correct server without the support of load balancers inside a number of providers. As the dashboard shows, AWS has high RPS, but very low bandwidth usage with digital ocean taking over to deliver content.

Any questions, please reach out via LinkedIn or Github (links are at the top of the page via mouseover).