V2 Backups
Introduction
This page describes:
- How application data (e.g. all application data and state) for individual V2 applications are managed
- How backups for individual V2 applications are structured
- How backups for VMs are structured
- How multiple VM are consolidated on the host server
- How multiple server backups consolidated to a backup server
Definitions
Application - An application or service that manages its state in a database and/or a set of data files
VM - A virtual machine, in which (supporting) services and/or applications are run
One or more Applications run on a VM. The limit is set by the capacity of the VM
Server - A hardware/software platform supporting a set of services and application VMs
One or more VMs run on a Server, limited by the capacity of the Server
Service - A host or network service that supports Applications and their underlying infrastructure (see Services)
The entire infrastructure can be scaled by adding additional VMs and Servers
Services
Each Host System must have:
DNS server in a VM that resolves the internal names and IP address of the VMs
Forwarding-only DNS server that forward DNS requests to the DNS server VM
rinetd daemon that forwards incoming TCP traffic to the appropriate VM based on the port number. Typically
All web traffic (ports 80 and 443) is forwarded to the Reverse Proxy server
- Note that DNS traffic, which uses both TCP and UDP protocols, is managed by the Host System's forwarding-only DNS server
- If the Host System has a VM with a Puppet server then port 8140 is forwarded to it
Reverse Proxy, an Apache module in a web server VM, that forward web requests to the appropriate VM based on the hostname in the HTTP request
Additional services that might be found on a Host System:
- Outward facing DNS
- Puppet master server
Requirements
Application Data Management Requirements
- To ensure good application/service performance given a server consisting of:
- RAM, typically 16-32 GB
- SSD, typically 128-256 GB
- RAID1 HD, typically two 1-3 TB
Recommendation:
- On SSD:
- Base OS and application support services (web-server, database server, etc.) on SSD
- Application footprint, reduced to the actively accessed data (application code, libraries, configuration, database, logs)
- On HD
- Applications data, revision and attachment files on the RAID hard disk
Issue where to store thumbnail images?
Backup Requirements
- To guarantee that the data required to completely restore or duplicate an instance of a application is always available. At a minimum the data to be backed up includes:
- The application's data files
- The application's database
- Backups should be made automatically on a nightly basis
- The backup data should be stored:
- On one or more backup servers
- On one or more off-site storage devices (typically USB drives)
- There should not a lot of effort required to re-configure backups if an application is moved from one VM or server to another
Server Backups
Backups Tiers
- There are multiple backups tiers. They are, starting from the lowest tier:
Application - Individual application and service backups, hosted on virtual machines
VM - Virtual machine backups, hosted on servers
Local Backups - Backups of multiple VMs on running locally on the server
Remote Backups - Backups of other servers' local backups. E.g. multiple local backups copied from other servers
Off-Site Backups - Backups stored on off-line devices, typically external USB drives, stored in secure locations
- There is a naming convention that provides unique paths to backups, such that all backup data can be found given:
The application's or service's relpath, e.g. virtual host name and relative path to the application
The VM name on which it runs
The fully qualified domain name of the Server that hosts the VM
- The directory structures of the backups, at each backup level, are defined such that it is possible to make:
- Remote backups of local backups, by copying the entire local backup, without needing to know the details of the VMs and application backups
- On VMs, backups of an application can be made to a well known directory, mounted from the host system, without needing to know the details of remote and local backups
TODO For monitoring: Backup tasks should create backup history files containing:
File name: backup-{name}-{yyyymmdd}, containing:
- Number of files
- Number of MB of data
- Size of the latest DB backup file(s)
Stored in a {history} directory
Backup Paths
Local and Remote backups:
partition = Absolute path to the remote backup directory. Typically: /v01/backup
server = The fully qualified domain name where the local backup is located
- Paths to remote and local backup directories:
{partition}/remote/{server} E.g. /v01/backup/remote/zg-1.softxs.ch {partition}/local E.g. /v01/backup/local
Application and Service backups:
relpath = The relative path to the application backup directory. Composed of:
virtual host name - The fully qualified domain name of the virtual host over which the application is provided
relative path - The relative path used to address the application
Example: The rails application http://demo.softxs.ch/hydro, hosted on the server zg-3.softxs.ch in the VM web, which has the relpath v2.softxs.ch/hydro, and its remote and local backups would be located at:
- As a local backup:
{partition}/local/{vm}/{relpath} E.g. /v01/backup/local/web/v2.softxs.ch/hydro
- As a remote backup:
{partition}/remote/{server}/{vm}/{relname} E.g. /v01/backup/zg-3.softxs.ch/v2.softxs.ch/hydro
- As a local backup:
Local Backups are performed on VMs (or directly on servers when there is no VM) by individual application jobs that push their backup data to the local server's (e.g. the VM host machine) backup directory
- The details of what is backed up and how the backup is structured depends on type of application being backed up
See TODO (link to section below) for more details
Remote Backups are made by backup servers that pull data from the remote server's local backup directory to their remote backup directory
- Example of commands that could be used as remote backup jobs:
rsync -a zg-3.softxs.ch:/v01/backup/local/ /v01/remote/backup/zg-3.softxs.ch/ rsync -a odin.softxs.ch:/v01/backup/local/ /v01/remote/backup/odin.softxs.ch/ ...
- Example of commands that could be used as remote backup jobs:
Application Data Management
Backup Types
Rails applications (specifically applications installed by Capistano)
- DrawMGT applications
- Wiki systems
- CVS and GIT repositories
- Email repositories
WordPress systems
Rails Applications
Rails Application Directory Structure
This is limited to rails applications installed by Capistrano
- Capistrano notes:
- Rails applications are deployed by Capistrano, which imposes a well-defined directory structure
- Supports keeping the codebase of older revisions, and the ability to rollback releases
- Provided you can rollback the database migration.
Supports a common directory containing the Rails bundle and other information doesn't change between releases
- Supports keeping the codebase of older revisions, and the ability to rollback releases
- Rails applications are deployed by Capistrano, which imposes a well-defined directory structure
- Local Capistrano extensions:
TODO backup directory setup
TODO delayed job management
Rails Application Installation Directories
usr = rails application user. Typically v2
name' = application installation name
/home/{user}/rails/{name} Symbolic link to {name}-app/current/public /home/{user}/rails/{name}-app Application installation directory
Rails Application Internal Directory Structure
name = application installation name
yymmddhhmmss = release timestamp
{name} Symbolic link to {name}-app/current/public {name}-app current Symbolic link to releases/{yymmhhmmhhss}. E.g. the current release releases yymmddhhmmss Root rails application directory. Approx. 16 MB. We backup older releases app config ... yymmddhhmmss yymmddhhmmss ... shared assets Small? Appears to be empty backup Small, only contains backups made during software updates bundle Large, 130 MB. On SSD log Can get large, needs log rotation. On SSD pids Small system Empty var Symbolic link to data directory. Very large. Contains all file_assets. On HD
Rails Data Directory Structure
The rails application var, which contains all the file_assets is stored on the hard disk
A symbolic link at {name}-app/shared/var points to the applications data directory
/v01/rails/data/{relpath}/{name}/var
Rails Backup Directory Structure
vm = VM directory name. E.g. /home/v2/rails/vms/vm
path = Virtual hostname and relative path. E.g. v2.softxs.ch/v2p0-jk1
/v01/backup/{vm}/{path}
Rails Application Backup Job
Dump database with timestamp to the backup/db directory
- Rsync the following
releases --> backup/releases
shared --> backup/shared
/data/var --> backup/shared
MoinMoin Wiki Directory structure
Backup Directory Structures
Application Backup Directories
VM Backup Directories
Backup Server Backup Directories
Summary of Backup Systems
SoftXS Systems
Description
Server
VM
Local Path
Relpath
Application Type
Status
SoftXS Web Site
zg-3.softxs.ch
web
/home/v2/rails/www.softxs.ch
www.softxs.ch
MAPS
done
Hydro Demo
zg-3.softxs.ch
web
/home/v2/rails/demo.softxs.ch/hydro
demo.softxs.ch/hydro
V2
done
PMS Demo
zg-3.softxs.ch
web
/home/v2/rails/demo.softxs.ch/pms
demo.softxs.ch/pms
V2
done
Tracking System
zg-3.softxs.ch
intern
/home/v2/rails/intern.softxs.ch/tracking
intern.softxs.ch/tracking
V2
done
Wiki
loki.softxs.ch
wiki
/home/wiki/www/html
wiki/softxs.ch
MoinMoin
done
Prototype Systems
Description
Host
VM
Local Path
Relpath
Application Type
Status
Prototype MAPS
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/maps
v2.softxs.ch/map
MAPS
n/a
Prototype MAPS AH
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/maps-ah
v2.softxs.ch/map-ah
MAPS
n/a
Prototype V2 AH
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-ah
v2.softxs.ch/v2p0-ah
V2
n/a
Prototype V2 DS
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-ds
v2.softxs.ch/v2p0-ds
V2
n/a
Prototype V2 JK
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-jk
v2.softxs.ch/v2p0-jk
V2
n/a
Prototype V2 JK 1
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-jk1
v2.softxs.ch/v2p0-jk1
V2
n/a
Prototype V2 JK 2
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-jk2
v2.softxs.ch/v2p0-jk2
V2
n/a
Prototype V2 JK 3
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-jk3
v2.softxs.ch/v2p0-jk3
V2
n/a
Prototype V2 TN
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-tn
v2.softxs.ch/v2p0-tn
V2
n/a
Prototype V2 TN 1
zg-3.softxs.ch
v2
/home/v2/rails/v2.softxs.ch/v2p0-tn1
v2.softxs.ch/v2p0-tn1
V2
n/a
Customer Systems
Description
Host
VM
Local Path
Relpath
Application Type
Status
Customer Signon
zg-3.softxs.ch
v0402
/home/v2/rails/v3.softxs.ch/sso
v3.softxs.ch/sso
MAPS
done
Roadnotes PMS
zg-3.softxs.ch
v0402
/home/v2/rails/roadnotes.softxs.ch/pms
roadnotes.softxs.ch/pms
V2
done
Poyry Uma Oya System
zg-3.softxs.ch
v0402
/home/v2/rails/poyry.softxs.ch/umaoya
poyry.softxs.ch/umaoya
V2
done
DrawMGT Systems
Description
Host
VM
Local Path
Relpath
Application Type
Status
Cardenillo
zg-1.softxs.ch
n/a
/home/cardenillo/html/prod/var
cardenillo.softxs.ch/prod
DrawMGT
wait
Coya II
zg-1.softxs.ch
n/a
/home/coya/html/prod/var
coya.softxs.ch/prod
DrawMGT
wait
DryTech
zg-1.softxs.ch
n/a
/home/drytech/html/prod/var
drytech.softxs.ch/prod
DrawMGT
wait
HCMC2
zg-1.softxs.ch
n/a
/home/hcmc2/html/prod/var
hcmc2.softxs.ch/prod
DrawMGT
wait
IDP
zg-1.softxs.ch
n/a
/home/idp/html/prod/var
idp.softxs.ch/prod
DrawMGT
wait
Ilulissat
zg-1.softxs.ch
n/a
/home/ilulissat/html/prod/var
ilulissat.softxs.ch/prod
DrawMGT
wait
KEJV
zg-1.softxs.ch
n/a
/home/kejv/html/arch/var
kejv.softxs.ch/arch
DrawMGT
wait
Lagarfoss
zg-1.softxs.ch
n/a
/home/lagarfoss/html/arch/var ?
lagarfoss.softxs.ch/arch ?
DrawMGT
wait
Sisimiut
zg-1.softxs.ch
n/a
/home/sisimiut/html/prod/var
sisimiut.softxs.ch/prod
DrawMGT
wait
See also V2VirtualServers and V2DeploymentInstances.