= V2 Backups = [[V2Master]] | [[V2HighLevelDesign]] <> = Introduction = This page describes: 1. How application data (e.g. all application data and state) for individual V2 applications are managed 1. How backups for individual V2 applications are structured 1. How backups for VMs are structured 1. How multiple VM are consolidated on the host server 1. How multiple server backups consolidated to a backup server == Definitions == 1. '''Application''' - An application or service that manages its state in a database and/or a set of data files 1. '''VM''' - A virtual machine, in which (supporting) services and/or applications are run 1. One or more ''Applications'' run on a ''VM''. The limit is set by the capacity of the ''VM'' 1. '''Server''' - A hardware/software platform supporting a set of services and application VMs 1. One or more ''VMs'' run on a ''Server'', limited by the capacity of the ''Server'' 1. '''Service''' - A host or network service that supports Applications and their underlying infrastructure (see [[#Services|Services]]) The entire infrastructure can be scaled by adding additional ''VMs'' and ''Servers'' == Services == Each Host System must have: 1. '''DNS server''' in a VM that resolves the internal names and IP address of the VMs 1. '''Forwarding-only DNS''' server that forward DNS requests to the DNS server VM 1. '''rinetd''' daemon that forwards incoming TCP traffic to the appropriate VM based on the port number. Typically 1. All web traffic (ports 80 and 443) is forwarded to the ''Reverse Proxy'' server 1. Note that DNS traffic, which uses both TCP and UDP protocols, is managed by the Host System's forwarding-only DNS server 1. If the Host System has a VM with a Puppet server then port 8140 is forwarded to it 1. '''Reverse Proxy''', an Apache module in a web server VM, that forward web requests to the appropriate VM based on the hostname in the HTTP request Additional services that might be found on a Host System: 1. Outward facing DNS 1. Puppet master server = Requirements = == Application Data Management Requirements == 1. To ensure good application/service performance given a server consisting of: 1. RAM, typically 16-32 GB 1. SSD, typically 128-256 GB 1. RAID1 HD, typically two 1-3 TB Recommendation: 1. On SSD: 1. Base OS and application support services (web-server, database server, etc.) on SSD 1. Application footprint, reduced to the actively accessed data (application code, libraries, configuration, database, logs) 1. On HD 1. Applications data, revision and attachment files on the RAID hard disk 1. '''Issue''' where to store thumbnail images? == Backup Requirements == 1. To guarantee that the data required to completely restore or duplicate an instance of a application is always available. At a minimum the data to be backed up includes: 1. The application's data files 1. The application's database 1. Backups should be made automatically on a nightly basis 1. The backup data should be stored: 1. On one or more backup servers 1. On one or more off-site storage devices (typically USB drives) 1. There should not a lot of effort required to re-configure backups if an application is moved from one VM or server to another = Server Backups = == Backups Tiers == 1. There are multiple backups tiers. They are, starting from the lowest tier: 1. '''Application''' - Individual application and service backups, hosted on virtual machines 1. '''VM''' - Virtual machine backups, hosted on servers 1. '''Local Backups''' - Backups of multiple VMs on running locally on the server 1. '''Remote Backups''' - Backups of other servers' local backups. E.g. multiple local backups copied from other servers 1. '''Off-Site Backups''' - Backups stored on off-line devices, typically external USB drives, stored in secure locations 1. There is a naming convention that provides unique paths to backups, such that all backup data can be found given: 1. The application's or service's '''relpath''', e.g. '''virtual host name''' and '''relative path''' to the application 1. The '''VM name''' on which it runs 1. The '''fully qualified domain name''' of the '''Server''' that hosts the VM 1. The directory structures of the backups, at each backup level, are defined such that it is possible to make: 1. Remote backups of local backups, by copying the entire local backup, without needing to know the details of the VMs and application backups 1. On VMs, backups of an application can be made to a well known directory, mounted from the host system, without needing to know the details of remote and local backups 1. '''TODO''' For monitoring: Backup tasks should create backup history files containing: 1. File name: '''backup-{name}-{yyyymmdd}''', containing: 1. Number of files 1. Number of MB of data 1. Size of the latest DB backup file(s) 1. Stored in a '''{history}''' directory == Backup Paths == 1. '''Local''' and '''Remote''' backups: 1. '''partition''' = Absolute path to the ''remote backup'' directory. Typically: '''/v01/backup''' 1. '''server''' = The fully qualified domain name where the local backup is located 1. Paths to remote and local backup directories: {{{ {partition}/remote/{server} E.g. /v01/backup/remote/zg-1.softxs.ch {partition}/local E.g. /v01/backup/local }}} 1. '''Application''' and '''Service''' backups: 1. '''relpath''' = The relative path to the application backup directory. Composed of: 1. '''virtual host name''' - The fully qualified domain name of the virtual host over which the application is provided 1. '''relative path''' - The relative path used to address the application 1. Example: The rails application http://demo.softxs.ch/hydro, hosted on the server zg-3.softxs.ch in the VM ''web'', which has the ''relpath'' '''v2.softxs.ch/hydro''', and its remote and local backups would be located at: 1. As a local backup: {{{ {partition}/local/{vm}/{relpath} E.g. /v01/backup/local/web/v2.softxs.ch/hydro }}} 1. As a remote backup: {{{ {partition}/remote/{server}/{vm}/{relname} E.g. /v01/backup/zg-3.softxs.ch/v2.softxs.ch/hydro }}} 1. '''Local Backups''' are performed on VMs (or directly on servers when there is no VM) by individual application jobs that '''''push''''' their backup data to the local server's (e.g. the VM host machine) backup directory 1. The details of what is backed up and how the backup is structured depends on type of application being backed up 1. See '''TODO (link to section below)''' for more details 1. '''Remote Backups''' are made by backup servers that '''''pull''''' data from the remote server's local backup directory to their ''remote'' backup directory 1. Example of commands that could be used as remote backup jobs: {{{ rsync -a zg-3.softxs.ch:/v01/backup/local/ /v01/remote/backup/zg-3.softxs.ch/ rsync -a odin.softxs.ch:/v01/backup/local/ /v01/remote/backup/odin.softxs.ch/ ... }}} = Application Data Management = == Backup Types == 1. Rails applications (specifically applications installed by ''Capistano'') 1. DrawMGT applications 1. Wiki systems 1. CVS and GIT repositories 1. Email repositories 1. Word``Press systems == Rails Applications == === Rails Application Directory Structure === This is limited to rails applications installed by ''Capistrano'' 1. Capistrano notes: 1. Rails applications are deployed by Capistrano, which imposes a well-defined directory structure 1. Supports keeping the codebase of older revisions, and the ability to rollback releases 1. Provided you can rollback the database migration. 1. Supports a ''common'' directory containing the Rails ''bundle'' and other information doesn't change between releases 1. Local Capistrano extensions: 1. '''TODO''' backup directory setup 1. '''TODO''' delayed job management ==== Rails Application Installation Directories ==== 1. '''usr''' = rails application user. Typically '''v2''' 1. '''name'''' = application installation name {{{ /home/{user}/rails/{name} Symbolic link to {name}-app/current/public /home/{user}/rails/{name}-app Application installation directory }}} ==== Rails Application Internal Directory Structure ==== 1. '''name''' = application installation name 1. '''yymmddhhmmss''' = release timestamp {{{ {name} Symbolic link to {name}-app/current/public {name}-app current Symbolic link to releases/{yymmhhmmhhss}. E.g. the current release releases yymmddhhmmss Root rails application directory. Approx. 16 MB. We backup older releases app config ... yymmddhhmmss yymmddhhmmss ... shared assets Small? Appears to be empty backup Small, only contains backups made during software updates bundle Large, 130 MB. On SSD log Can get large, needs log rotation. On SSD pids Small system Empty var Symbolic link to data directory. Very large. Contains all file_assets. On HD }}} ==== Rails Data Directory Structure ==== 1. The rails application ''var'', which contains all the file_assets is stored on the hard disk 1. A symbolic link at {name}-app/shared/var points to the applications ''data'' directory {{{ /v01/rails/data/{relpath}/{name}/var }}} ==== Rails Backup Directory Structure ==== 1. '''vm''' = VM directory name. E.g. /home/v2/rails/vms/'''vm''' 1. '''path''' = Virtual hostname and relative path. E.g. '''v2.softxs.ch/v2p0-jk1''' {{{ /v01/backup/{vm}/{path} }}} === Rails Application Backup Job === 1. Dump database with timestamp to the ''backup/db'' directory 1. Rsync the following 1. ''releases'' --> ''backup/releases'' 1. ''shared'' --> ''backup/shared'' 1. ''/data/var'' --> ''backup/shared'' == MoinMoin Wiki Directory structure == = Backup Directory Structures = == Application Backup Directories == == VM Backup Directories == == Backup Server Backup Directories == = Summary of Backup Systems = ||<-7 #cddeee> '''SoftXS Systems''' || ||<#e0e0e0> '''Description''' ||<#e0e0e0> '''Server''' ||<#e0e0e0> '''VM''' ||<#e0e0e0> '''Local Path''' ||<#e0e0e0> '''Relpath''' ||<#e0e0e0> '''Application Type''' ||<#e0e0e0> '''Status''' || || SoftXS Web Site || zg-3.softxs.ch || web || /home/v2/rails/www.softxs.ch || www.softxs.ch || MAPS ||<#ccff66> done || || Hydro Demo || zg-3.softxs.ch || web || /home/v2/rails/demo.softxs.ch/hydro || demo.softxs.ch/hydro || V2 ||<#ccff66> done || || PMS Demo || zg-3.softxs.ch || web || /home/v2/rails/demo.softxs.ch/pms || demo.softxs.ch/pms || V2 ||<#ccff66> done || || Tracking System || zg-3.softxs.ch || intern || /home/v2/rails/intern.softxs.ch/tracking || intern.softxs.ch/tracking || V2 ||<#ccff66> done || || Wiki || loki.softxs.ch || wiki || /home/wiki/www/html || wiki/softxs.ch || Moin``Moin ||<#ccff66> done || ||<-7 #cddeee> '''Prototype Systems''' || ||<#e0e0e0> '''Description''' ||<#e0e0e0> '''Host''' ||<#e0e0e0> '''VM''' ||<#e0e0e0> '''Local Path''' ||<#e0e0e0> '''Relpath''' ||<#e0e0e0> '''Application Type''' ||<#e0e0e0> '''Status''' || || Prototype MAPS || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/maps || v2.softxs.ch/map || MAPS ||<#e0e0e0> ''n/a'' || || Prototype MAPS AH || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/maps-ah || v2.softxs.ch/map-ah || MAPS ||<#e0e0e0> ''n/a'' || || Prototype V2 AH || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-ah || v2.softxs.ch/v2p0-ah || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 DS || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-ds || v2.softxs.ch/v2p0-ds || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 JK || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-jk || v2.softxs.ch/v2p0-jk || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 JK 1 || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-jk1 || v2.softxs.ch/v2p0-jk1 || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 JK 2 || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-jk2 || v2.softxs.ch/v2p0-jk2 || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 JK 3 || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-jk3 || v2.softxs.ch/v2p0-jk3 || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 TN || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-tn || v2.softxs.ch/v2p0-tn || V2 ||<#e0e0e0> ''n/a'' || || Prototype V2 TN 1 || zg-3.softxs.ch || v2 || /home/v2/rails/v2.softxs.ch/v2p0-tn1 || v2.softxs.ch/v2p0-tn1 || V2 ||<#e0e0e0> ''n/a'' || ||<-7 #cddeee> '''Customer Systems''' || ||<#e0e0e0> '''Description''' ||<#e0e0e0> '''Host''' ||<#e0e0e0> '''VM''' ||<#e0e0e0> '''Local Path''' ||<#e0e0e0> '''Relpath''' ||<#e0e0e0> '''Application Type''' ||<#e0e0e0> '''Status''' || || Customer Signon || zg-3.softxs.ch || v0402 || /home/v2/rails/v3.softxs.ch/sso || v3.softxs.ch/sso || MAPS ||<#ccff66> done || || Roadnotes PMS || zg-3.softxs.ch || v0402 || /home/v2/rails/roadnotes.softxs.ch/pms || roadnotes.softxs.ch/pms || V2 ||<#ccff66> done || || Poyry Uma Oya System || zg-3.softxs.ch || v0402 || /home/v2/rails/poyry.softxs.ch/umaoya || poyry.softxs.ch/umaoya || V2 ||<#ccff66> done || ||<-7 #cddeee> '''DrawMGT Systems''' || ||<#e0e0e0> '''Description''' ||<#e0e0e0> '''Host''' ||<#e0e0e0> '''VM''' ||<#e0e0e0> '''Local Path''' ||<#e0e0e0> '''Relpath''' ||<#e0e0e0> '''Application Type''' ||<#e0e0e0> '''Status''' || || Cardenillo || zg-1.softxs.ch || ''n/a'' || /home/cardenillo/html/prod/var || cardenillo.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || || Coya II || zg-1.softxs.ch || ''n/a'' || /home/coya/html/prod/var || coya.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || || Dry``Tech || zg-1.softxs.ch || ''n/a'' || /home/drytech/html/prod/var || drytech.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || || HCMC2 || zg-1.softxs.ch || ''n/a'' || /home/hcmc2/html/prod/var || hcmc2.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || || IDP || zg-1.softxs.ch || ''n/a'' || /home/idp/html/prod/var || idp.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || || Ilulissat || zg-1.softxs.ch || ''n/a'' || /home/ilulissat/html/prod/var || ilulissat.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || || KEJV || zg-1.softxs.ch || ''n/a'' || /home/kejv/html/arch/var || kejv.softxs.ch/arch || DrawMGT ||<#e0e0e0> wait || || Lagarfoss || zg-1.softxs.ch || ''n/a'' || /home/lagarfoss/html/arch/var ? || lagarfoss.softxs.ch/arch ? || DrawMGT ||<#e0e0e0> wait || || Sisimiut || zg-1.softxs.ch || ''n/a'' || /home/sisimiut/html/prod/var || sisimiut.softxs.ch/prod || DrawMGT ||<#e0e0e0> wait || See also V2VirtualServers and V2DeploymentInstances.