Managing V2 Backups and Application Data

V2Master | V2HighLevelDesign

• Contents

  1. Managing V2 Backups and Application Data
  2. Introduction
     1. Definitions
     2. Services
  3. Requirements
     1. Application Data Management Requirements
     2. Backup Requirements
  4. Server Backups
     1. Backups Tiers
     2. Backup Paths
  5. Application Data Management
     1. Rails Application Directory Structure
        1. Rails Application
        2. Rails Application Directory Structure
        3. Rails Data Directory Structure
        4. Rails Backup Directory Structure
        5. Rails Application Backup Job
     2. MoinMoin Wiki Directory structure
  6. Backup Directory Structures
     1. Application Backup Directories
     2. VM Backup Directories
     3. Backup Server Backup Directories
  7. Summary of Backup Systems

Introduction

This page describes:

1. How application data (e.g. all application data and state) for individual V2 applications are managed
2. How backups for individual V2 applications are managed
3. How backups for VMs are managed

Definitions

1. Application - An application that manages its state in a database and a set of data files

2. VM - A virtual machine, in which supporting services and application are run

   1. One or more Applications run on a VM, limited by the capacity of the VM

3. Host System - A hardware/software platform supporting a set of services and application VMs

   1. One or more VMs run on a Host System, limited by the capacity of the Host System

4. Service - A host or network service that supports Applications and their underlying infrastructure (see Services)

The entire infrastructure can be scaled by adding additional VMs and Host Systems

Services

Each Host System must have:

1. DNS server in a VM that resolves the internal names and IP address of the VMs

2. forwarding-only DNS server that forward DNS requests to the DNS server VM

3. rinetd daemon that forwards incoming TCP traffic to the appropriate VM based on the port number. Typically

   1. All web traffic (ports 80 and 443) is forwarded to the Reverse Proxy server

   2. Note that DNS traffic, which uses both TCP and UDP protocols, is managed by the Host System's forwarding-only DNS server
   3. If the Host System has a VM with a Puppet server then port 8140 is forwarded to it

4. Reverse Proxy, an Apache module in a web server VM, that forward web requests to the appropriate VM based on the hostname in the HTTP request

Additional services that might be found on a Host System:

1. Outward facing DNS
2. Puppet master server

Requirements

Application Data Management Requirements

1. To ensure good application/service performance given a host environment consisting of:
   1. A limited amount of RAM, typically 16-32 GB
   2. A limited amount of high-speed SSD-based disk space, typically between 128-256 GB SSD drives
   3. A large amount of lower-speed RAID1 hard-disk space, typically 1-3 TB mirrored SATA hard disks

Recommendation:

1. Store the base operating system and application support service (web-server, database server, etc.) on SSD
2. Store the minimum application footprint, e.g. the actively accessed part of the application (code, configuration, database, logs) on SSD
3. Store the applications revision and attachment files on the RAID hard disk

4. Issue where to store thumbnail images?

Backup Requirements

1. To guarantee that the data required to completely restore or duplicate an instance of a application is always available. At a minimum the data to be backed up includes:
   1. The application's files
   2. The application's database
2. Backups should be made automatically on a nightly basis
3. The backup data should be stored:
   1. On one or more backup servers
   2. On one or more off-site storage devices (typically USB drives)
4. There should not a lot of effort required to re-configure backups if an application is moved from one VM or Host Machine to another

Server Backups

Backups Tiers

1. Three are multiple backups tiers. Starting from the lowest tier:

   1. Application - Individual application and service backups, hosted on virtual machines

   2. VM - Virtual machine backups, hosted on host systems

   3. Local Backups - Backups of multiple VMs on stored locally on a host system

   4. Remote Backups - Backups of multiple local backups. E.g. backup of multiple host systems

2. There is a naming convention that provides unique paths to backups, so that all backup data can be found given:

   1. The application's or service's virtual host name and relative path to the application/service

   2. The name of the VM on which it runs

   3. The fully qualified domain name of the Host System when the VM is running

3. The directory structures of the backups, at each backup level are structured such that it is possible to make:
   1. Remote backups of local backups by copying the entire local backup, without needing to know the details of the VMs and application/service backups
   2. On VMs backups of application/services can be made to a well know directory, mounted from the host system, with out knowing the details of how remote and local ba

4. TODO For monitoring: Backup tasks should create a backup history file, containing:

   1. File name: backup-{name}-{yyyymmdd}, containing:
      1. Number of files
      2. Number of MB of data
   2. Stored in a {history} directory

Backup Paths

1. Local and Remote backups:

   1. partition = Absolute path to the remote backup directory. Typically: /v01/backup

   2. server = The fully qualified domain name where the local backup is located

   3. Paths to remote and local backup directories:

      {partition}/remote/{server}            E.g. /v01/backup/remote/zg-1.softxs.ch
      {partition}/local                      E.g. /v01/backup/local

2. Application and Service backups:

   1. relpath = The relative path to the application/service backup directory. Composed of:

      1. virtual host name - The fully qualified domain name of the virtual host over which the application/service is provided

      2. relative path - The relative path used to address the application/service

   2. Example: The rails application http://demo.softxs.ch/hydro (hosted on the server zg-3.softxs.ch) has the relpath v2.softxs.ch/hydro, and its remote and local backups would be located at:

      {partition}/remote/{server}/{relname}   E.g. /v01/backup/zg-3.softxs.ch/v2.softxs.ch/hydro
      {partition}/local/{relpath}             E.g. /v01/backup/local/v2.softxs.ch/hydro

3. Local backups are performed on VMs (or servers, when there is no VM) by individual application/service jobsthat push their backup data to the local server's backup directory

   1. The details of what is backed up and how the backup is structured depends on type of application/service being backed up

   2. See TODO (link to section below) for more details

4. Remote backups are made by backup servers that pull data from the remote server's local backup directory to their remote backup directory

   1. Example of remote backup jobs:

      rsync -va server1:/v01/backup/local/ /v01/remote/backup/server1
      rsync -va server2:/v01/backup/local/ /v01/remote/backup/server2
      ...

Application Data Management

Rails Application Directory Structure

1. Capistrano notes:
   1. Rails applications are deployed by Capistrano, which imposes a well-defined directory structure
      1. Supports keeping the codebase of older revisions, and the ability to rollback releases
         1. Provided you can rollback the database migration.

      2. Supports a common directory containing the Rails bundle and other information doesn't change between releases

2. Local Capistrano extensions

Rails Application

1. usr = rails application user. Typically v2

2. name' = application installation name

   /home/{user}/rails/{name}              Symbolic link to {name}-app/current/public
   /home/{user}/rails/{name}-app          Application installation directory

Rails Application Directory Structure

1. name = application installation name

2. yymmddhhmmss = release timestamp

   {name}                                  Symbolic link to {name}-app/current/public
   {name}-app
     current                               Symbolic link to releases/{yymmhhmmhhss}. E.g. the current release
     releases
       yymmddhhmmss                        Root rails application directory. Approx. 16 MB. We backup older releases
         app
         config
         ...
       yymmddhhmmss
       yymmddhhmmss
       ...
     shared
       assets                              Small? Appears to be empty
       backup                              Small, only contains backups made during software updates
       bundle                              Large, 130 MB. On SSD
       log                                 Can get large, needs log rotation. On SSD
       pids                                Small
       system                              Empty
       var                                 Symbolic link to data directory. Very large. Contains all file_assets. On HD

Rails Data Directory Structure

1. The rails application var, which contains all the file_assets is stored on the hard disk

2. A symbolic link at {name}-app/shared/var points to the applications data directory

   /v01/rails/data/{relpath}/{name}/var

Rails Backup Directory Structure

1. vm = VM directory name. E.g. /home/v2/rails/vms/vm

2. path = Virtual hostname and relative path. E.g. v2.softxs.ch/v2p0-jk1

   /v01/backup/{vm}/{path}

Rails Application Backup Job

1. Dump database with timestamp to the backup/db directory

2. Rsync the following

   1. releases --> backup/releases

   2. shared --> backup/shared

   3. /data/var --> backup/shared

MoinMoin Wiki Directory structure

Backup Directory Structures

Application Backup Directories

VM Backup Directories

Backup Server Backup Directories

Summary of Backup Systems

• SoftXS Systems
  Description	Server	VM	Local Path	Relpath	Application Type	Status
  SoftXS Web Site	zg-3.softxs.ch	web	/home/v2/rails/www.softxs.ch	www.softxs.ch	rails	open
  Hydro Demo	zg-3.softxs.ch	web	/home/v2/rails/demo.softxs.ch/hydro	demo.softxs.ch/hydro	rails	open
  Tracking System	zg-3.softxs.ch	intern	/home/v2/rails/intern.softxs.ch/tracking	intern.softxs.ch/tracking	rails	open
  Wiki	loki.softxs.ch	wiki	/home/wiki/www/html	wiki/softxs.ch	MoinMoin	open
  Prototype Systems
  Description	Host	VM	Local Path	Relpath	Application Type	Status
  TODO	zg-3.softxs.ch	v2	/home/v2/rails/?	v2.softxs.ch/?	rails	open
  Customer Systems
  Description	Host	VM	Local Path	Relpath	Application Type	Status
  Roadnotes PMS	zg-3.softxs.ch	v0402	/home/v2/rails/intern.softxs.ch/tracking	intern.softxs.ch/tracking	rails	open
  DrawMGT Systems
  Description	Host	VM	Local Path	Relpath	Application Type	Status
  Cardenillo	zg-1.softxs.ch	n/a	/home/cardenillo/html/prod/var	cardenillo.softxs.ch	DrawMGT	open
  Coya II	zg-1.softxs.ch	n/a	/home/coya/html/prod/var	coya.softxs.ch	DrawMGT	open
  DryTech	zg-1.softxs.ch	n/a	/home/drytech/html/prod/var	drytech.softxs.ch	DrawMGT	open
  HCMC2	zg-1.softxs.ch	n/a	/home/hcmc2/html/prod/var	hcmc2.softxs.ch	DrawMGT	open
  IDP	zg-1.softxs.ch	n/a	/home/idp/html/prod/var	idp.softxs.ch	DrawMGT	open
  Ilulissat	zg-1.softxs.ch	n/a	/home/ilulissat/html/prod/var	ilulissat.softxs.ch	DrawMGT	open
  KEJV	zg-1.softxs.ch	n/a	/home/kejv/html/arch/var ?	kejv.softxs.ch	DrawMGT	open
  Lagarfoss	zg-1.softxs.ch	n/a	/home/lagarfoss/html/prod/var	lagarfoss.softxs.ch	DrawMGT	open
  Sisimiut	zg-1.softxs.ch	n/a	/home/sisimiut/html/prod/var	sisimiut.softxs.ch	DrawMGT	open

See also V2VirtualServers and V2DeploymentInstances.