V2 Deployment

V2Master

Introduction

This page describes:

  1. The server architecture for deploying publicly accessible MAPS and V2 systems
  2. The software infrastructure that supports MAPS and V2 deployment
  3. How to install publicly accessible beta instances of V2 and MAPS

See Also

  1. [V2SetupDevelopment] - Deployment and configuration of MAPS & V2 in development environments

  2. [V2SetupProduction] - Deployment and configuration of MAP & V2 in production/prototype environments

  3. [V2InstancesList] - List of static (e.g. non-generic) production/prototype instances

Server Architecture

Overview

  1. Access to one or more physical servers, accessible via a limited number of public IP addresses
    1. Use virtual hosts (DNS aliases) for addressing services and application instances
    2. Use a reverse proxy server to dispatch to the appropriate server, virtual machine
  2. The physical servers will collectively support multiple services:
    1. Rails-based MAPS and V2 systems
    2. PHP-based DrawMGT systems
    3. Wiki systems
    4. Web servers
    5. DNS servers
    6. CVS and Git servers
  3. Services hosted in virtual machines:
    1. Easy to migrate VMs to alternate physical servers
    2. Host OS configuration does not need to be changed based on changing application requirements
    3. Scaling to additional physical or could-based machines is possible
  4. Automated deployment of V2 systems:
    1. Creation of an instance of a largely pre-configured virtual machine
    2. Provisioning of the virtual machine (VM) with all site and instance specific configuration
    3. Deployment of the V2 application instance, including the site an instance specific configuration
      1. Initially each instance of a V2 application will reside in its own VM
  5. Automated monitoring:
    1. MAPS and V2 instances
    2. Virtual machines
    3. Physical servers
    4. Supporting infrastructure (DNS servers, reverse-proxy servers, etc.)

Supporting Technologies

Server operating system

CentOS 6.4

Virtual Machines

Vagrant, which runs on top of VirtualBox

VM provisioning

Puppet, a Ruby-based system configuration tool

Application deployment

Capistrano, a Ruby Gem for deploying Rails applications

Reverse proxy server

mod_proxy, an Apache module

Monitoring

Nagios and RRDTool

  1. Selected CentOS because:
    1. Acceptable to corporate and enterprise clients

    2. Supports Vagrant and VirtualBox (FreeBSD is currently unable to host

  2. The purpose of the reverse proxy server is to route incoming web requests to the appropriate server and virtual machine
  3. Selected Apache mod_proxy because:
    1. We have experience configuring and using Apache
    2. We do not have high performance requirements and therefore don't need to be particularly concerned about choosing the best performing reverse proxy server.
      1. Our highest traffic DrawMGT sites, which have 500-800 users, generate at most 25,000 requests per week, meaning (given a five day week and a 10 hour day) an average only 500 requests per hour. E.g. significantly less than one per second
    3. Apache and mod_proxy can be swapped out and replaced with something else, without affecting the rest of the infrastructure

Server and DNS Architecture

The following shows the DNS and IP address configuration for a system hosting multiple applictation VMs:

Notes:

  1. Dir is the directory where the VM resides on the virtual host. It is a located in /home/vms

  2. Notation:

    1. n - The application instance number, a decimal number from 1 to 255

    2. xxxx - The application instance number plus 0x0400, expressed in (lower case) Hex.

    3. x.y - The applictaion instance number expressed as two integers suitable for an IP address

    4. Examples:

      • Host

        Name

        IP Address

        App-1

        vm0401

        172.16.4.1

        App-163

        vm04a3

        172.16.4.163

        App-258

        vm0502

        172.16.5.2

  3. The VM Host's ip address:
    1. For vh01, the IP address is set to 172.16.2.4, but can but this can be changed to any IP address appropriate to its environment
    2. Should be accessible, possibly via NAT, from the public Internet
  4. Additional VM Hosts can be added by incrementing
    1. The same 172.16.x.y subnets can be used on every VM Host as they are not accesible out side the VM Host

  5. A DNS server must run on the virtual host, which must provide DNS lookup for, at a minumum, the vh01 sub-domain

    1. The vh01.softxs.ch sub-domain is only available to the vh01 host and it's VMs
    2. The external DNS names for the VMs all map to the vh01 host, E.g. 192.168.2.4
  6. Pre-configuration of DNS entries and Reverse Proxy configuration: The following should be pre-configured (e.g. generated by scripts):

    1. DNS entries for the vmxxxx.vh01.softxs sub-domain to 172.16.4.x address mappings

    2. DNS entries for the vmxxxxvh01.softxs name to 172.16.2.4 address mappings

    3. Reversy proxy configuration files mapping the external vmxxxxvh01.softxs name to the 172.16.4.x addresses

      1. With Ubuntu the configuration files can be placed in an available-sites directory and then symbolic links can be created in an activated-sites directory in order to enable them (which requires restarting Apache)

  7. Subnets 172.16.3.0/24 and 192.168.4.0/24 are Vagrant hostonly networks and are not directly accessible from outside the vh01 host

    1. Subnet 172.16.1.x is reserved for individually managed servers. E.g. DNS, mail, Web, Blog, etc. servers
    2. Subnet 172.16.4.x is reserved for automatically managed application servers
  8. Additional subnets above 172.16.4.x can be added in case more than 256 VMs are hosted in a single host (e.g. 172.16.5.x)

  9. For white labeling (custom customer specified domain names):

    1. Create a DNS aliases that maps to the VM host. Note that for non-softxs.ch domains, the customer will be responsible for the creation of DNS entry

    2. Create a virtual domain configuration file for the reverse proxy in the gateway machine that maps the domain name to the internal ip address

      1. See: http://ubuntuguide.org/wiki/Apache2_reverse_proxies

Server Deployment

General Recommendations

  1. Have spare hardware capacity
    1. Do not have idle backup servers, but use extra servers in production roles
    2. Run multiple identical servers with load split between them
  2. Regularly migrate services to different servers to ensure:
    1. It is possible and that there are no hidden problems
    2. We can rapidly and reliably restore services in the event of a failure

Hardware

  1. 64-bit Intel architecture
  2. Multi-core, fast CPU(s)
  3. Lots of RAM
  4. SSD (non-mirrored) disk for host operating system and the active part of the guest VMs (e.g. the guest OS, Rails, web and DB servers)
  5. Mirror/RAID hard disk for application data and backup staging

Note that Virtualization must be enabled in the underlying PC BIOS in order for Vagrant and VirtualBox VMs to function

Infrastructure Services

Puppet Based Server Configuration

  1. All servers should be managed using Puppet

  2. The configuration information, for our entire infrastructure should be stored in a single Git repository

    1. Located at: http://git.softxs.ch/home/git/gitroot/v2infra.git

  3. All configuration changes should be done on the Puppet Server and propagated to the target server

Provisioning of Applications Created by MAPS

Applications sold by MAPS will be provisioned and managed automatically:

  1. DNS entries
  2. Reverse proxy configuration
  3. VM provisioning - Network configuration and hostname
  4. Application deployment - Site and instance configuration
  5. Backup - Configuration and execution
  6. Monitoring

New Servers

The new servers should be implemented in VMs on new servers.

  1. MAPS Server - Master Application and Payment System server, including public web site and sales font-end

  2. Puppet Server - Automated server configuration via Puppet

  3. Reverse Proxy Server - Routing incoming web requests to the back-end VMs and applications responsible for serving them

Existing Servers

The existing servers are currently running on dedicated servers and should be migrated to separate VMs.

  1. DNS Servers - Public DNS servers. Currently residing in Zug (zg-1.softxs.ch) and Budapest (bp-1.softxs.hu) Partially DONE

  2. Mail Server - Incoming and outgoing email and IMAP based email access. Handling mail aliases for DrawMGT customer systems

  3. Git and CVS Servers - Source control systems

  4. Wiki Server - MoinMoin wikis DONE

  5. Web Server - Static web pages DONE

  6. Blog Server - DrawMGT customer MoinMoin based Wikis and WordPress based blog system

  7. DrawMGT Server - Host for DrawMGT PHP based applications

Virtual Machines

  1. VMs are created and initilized (e.g. network configuration) by Vagrant

  2. Initially, deploy one VM per application instance. E.g. each application instance has its own VM
    1. For paid applications, the application's VM will be located on an SSD
    2. For free (or low-cost) applications, the application's VM will be located on a normal hard disk
  3. File sharing from the VM to the host system:

    1. Puppet configuration files - Not sure if this needed, given a Puppet server

    2. Backup staging area - For application documents, database backups and log files
    3. Application documents

VM Provisioning

  1. VM boxes, which are templates for creating new VMs, are stored in a directory: /home/vagrant/boxes

    1. Standard boxes VM boxes can be downloaded from: http://www.vagrantbox.es

  2. We will deploy a CentOS 6.4 box, with Rails infrastructure, Apache, Passenger and MySQL already installed (but not completely configured)

  3. VM instances, which can be either running, suspended or halted, are stored in a directory: /home/vagrant/VirtualBox VMs

  4. VMs are created on the host system:

    1. In one directory per VM: Suggest /home/vm/{name}

    2. The VM directory contains Vagrantfile that defines the VMs basic properties:

      1. The base box upon which it is based
      2. Network and hostname configuration
      3. Directories shared with the host OS
      4. Configuration options: memory limits, etc.
      5. Note that Vagrant allows multiple VMs to be defined in a single Vagrant file. We will not use this feature

    3. The host's vagrant user is used to access the VM from the command line

  5. Create and save an SSH key for accessing the machines?

Tips

  1. Vagrant uses VirtualBox, by Oracle, to manage VMs

    1. Use the VBoxManage command to monitor VMs. Must be run as user vagrant (running displays an empty list!)

      1. VBoxManage list [-l] vms - Lists all VM instances that have been defined

      2. VBoxManage list [-l] runningvms - Lists all VM instances that are running

  2. Always shutdown a VM before attempting to destroy it. Destroying a running VM will sometimes hang

VM Provisioning

Application Deployment

  1. Includes MAPS and V2 deployment

  2. To be completed - describe:

    • Application issues
      • Gemfile.lock
    • Apache config
    • Site/instance Git repository
      • Structure of repository
      • Deployment files
        • database.yml
    • Capistrano
  3. System deployment
  4. Application upgrade

V2 and MAPS Application Deployment

  1. See V2CapistranoDeployment

V2 and MAPS System Setup

See the following pages:

  1. V2SetupDevelopment - Describes how to setup and bootstrap MAPS and V2 in a development or prototype environment

  2. V2SetupProduction - Describes how to setup and bootstrap MAPS and V2 in a production environment

Deployment Types

  1. Predefined systems, based on an application, site and instance.

    1. Use cases: personal development systems, prototype systems, production MAPS system, large-customer systems, white labelled systems
    2. Requires manual configuration, some of which might be automatically generated
    3. Deployment is automated, provided the configuration is completely defined

  2. Generated systems, based on a product, which determines an application and a pre-defined configuration

    1. Use cases: v2 instances, trial systems, demo systems
    2. Requires a configuration template, which is created manually
    3. The configuration template:
      1. Makes use of other configuration elements (e.g. host machine for VMs, DNS entries, proxy server configuration, etc.), which must be prepared in advance
      2. Allows multiple instances of pre-configured applications to be automatically deployed

Deployment Steps

The complete end-to-end list of deployment steps:

  1. Pre-configuration:
    1. Virtual Machine Host
    2. DNS
    3. Reverse Proxy
  2. Virtual machine setup
    1. Setup directories on virtual host for VM
    2. Vagrant file creation
    3. Bring up VM
    4. Setup NHS shares to host machine
      1. File system for documents
      2. File system for backups
    5. Provision VM
  3. Application deployment
    1. Setup installation directories on VM
    2. Checkout from Git and deploy on VM
    3. Migration:
      1. Database
      2. Documents and static file
    4. Setup automated backups

Pre-Configuration

The following items must be prepared in advance. Typically the configured is either performed by hand, or defined by hand in Puppet configuration files.

  1. Virtual Host (VH) preparation - A system that hosts virtual machines

    1. Base server hardware and operating system prepared by hand
    2. System provisioning performed by Puppet

  2. DNS entry preparation - DNS entries for accessing virtual machines

    1. Sets of entries for multiple instances prepared in advance by hand in Puppet configuration files
    2. Deployment performed by puppet

  3. Reverse Proxy configuration - Defines forwarding rules for external web access to application instances on virtual machines

    1. ets of entries for multiple instances prepared in advance by hand in Puppet configuration files
    2. Deployment performed by puppet

Application Deployment Configuration Variables

Notes

  1. vmsshport - Must be a unique value from within a VM. Suggest using the x.y value

DMZ VM System

This section describes the configuration on the Rails deployment environment on the loki.softxs.ch server, located in the DMZ in the AH server room.

Current Services Provided in DMZ

All these services should be migrated to VM based servers.

  1. idun.softxs.ch - email and DNS server
    1. Incoming and outgoing SMPT mail transfer
    2. IMAPS access to email
    3. Storage of email
    4. DNS server for DMZ, not used publicly
  2. honir.softxs.ch - web server
    1. Private web pages
      1. Mackay family tree
      2. Robert L. Mackay Diaries
      3. Innovation web site
      4. Salandra
      5. Sho Takahashi
      6. Alan
      7. Venture internal pages
    2. Business web pages
      1. Demo systems
        1. Hydro-2007
        2. ITA-Demo
        3. LTF-Demo
        4. SpecMGT-Demo

    3. Web based access to email, via SquirrelMail

    4. Wikis, via MoinMoin

      1. SoftXS public web pages
      2. DrawMGT documentation in English and German
      3. Internal Wiki
      4. Customer Wikis
        1. IDP
        2. Cardenillo
        3. MET
        4. NDD
        5. Sisimiut
        6. HCMC2
    5. AHYTRA - hydrodynamic modeling software and associated web-based plotting and display

      1. Available via http://lu.softxs.ch/venture/ahytra

  3. ymir.softxs.ch - Source control and blog server
    1. CVS repositories
    2. Git repositories

      1. Git repository web access, via gitweb

    3. Blogs, via WordPress

      1. Salandra - AH private blog. http://blog.salandra.ch

      2. SoftXS - Unused. http://blog.softxs.ch

    4. DrawMGT test systems (can be discarded)

VM Server #1

Hardware

  1. CPU: Intel Core i5-3330 CPU @ 3.00GHz, Quad Core, SandyBridge

  2. Main memory: 16 GB
  3. Disk:
    1. 1 x 128 GB SSD, KINGSTON SH103S3120G
    2. 2 x 2 TB Western Digital Caviar Green, WDC WD20EARS-00MVWB0
  4. Network:
    1. Onboard 1 GB/Sec ethernet -- unusable
    2. PCI card with 10/100 MB/sec ethernet

Basic Configuration

Network Access Configuration

  1. ADSL router configuration
    1. Hardware: Cisco 826
    2. NAT Forwarding 
      •   Pro Inside global         Inside local          Outside local         Outside global
  tcp 81.221.23.36:22       192.168.1.36:22       ---                   ---  # SSH
  tcp 81.221.23.36:25       192.168.1.36:25       ---                   ---  # SMTP
  tcp 81.221.23.36:53       192.168.1.36:53       ---                   ---  # DNS TCP
  udp 81.221.23.36:53       192.168.1.36:53       ---                   ---  # DNS UDP
  tcp 81.221.23.36:80       192.168.1.36:80       ---                   ---  # HTTP
  tcp 81.221.23.36:443      192.168.1.36:443      ---                   ---  # HTTPS
  tcp 81.221.23.36:465      192.168.1.36:465      ---                   ---  # SMTPS
  tcp 81.221.23.36:993      192.168.1.36:993      ---                   ---  # IMAPS
  2. Firewall configuration

    1. Hardware: Soekris net4501

    2. Firewall software: mon0wall

    3. NAT forwarding configuration. Includes 
      •   IF    Proto    Src  Dest IP     Source IP            Dst   Description
  WAN   TCP       22  192.168.2.4 (ext. 192.128.1.36)   22   Incoming SSH to loki
  WAN   TCP/UDP   53  192.168.2.4 (ext. 192.128.1.36)   53   Incoming DNS to loki
  WAN   TCP       80  192.168.2.4 (ext. 192.128.1.36)   80   Incoming HTTP to loki
  WAN   TCP      443  192.168.2.4 (ext. 192.128.1.36)  443   Incoming HTTPS to loki

File Systems

TODO The 2 x TB WD disks have not been formatted as mirrored disks.

User Configuration

  1. Vagrant
  2. Puppet

VM Configuration

Summary of VMs:

  1. Puppet Server VM
  2. DNS and DHCP Server VM
  3. Reverse Proxy VM
  4. V2 Server VMs

  5. Web Server VM -- DONE

  6. Wiki Server VM -- DONE

  7. Blog Server VM -- TODO

Puppet Server VM

Items to document for each VM:

  1. Location on host files system
  2. Guest BOX
  3. Local port forwarding
  4. File system shares

DNS and DHCP Server VM

Reverse Proxy VM

V2 Server VMs

ZG3 Beta Systems

This sections describes the configuration of the Rails environment on the zg-3.softxs.ch server, located in the Datawire datacenter in Cham.

Environment

The V2 server is a virtual host on the zg-3.softxs.ch server.

The server has following software systems:

  1. FreeBSD 9.0-RELEASE FreeBSD 9.0-RELEASE #0
  2. Apache Server version: Apache/2.2.23 (FreeBSD)
  3. Phusion Passenger apache module
  4. Ruby: ruby 1.9.3p327 (2012-11-10 revision 37606) [amd64-freebsd9]
  5. Rails 3.2.11
  6. mysql Ver 14.14 Distrib 5.5.28, for FreeBSD9.0 (amd64) using 5.2
  7. And many Gems

The installation location for Rails applications:

  1. /v01/local/www/rails

For each application two items are required:

  1. In the rails directory the following is required:
    1. A directory with the path {app}-app where the git repository is cloned
    2. A symbolic link {app} which points to the {app}-app/public directory

  2. In /usr/local/etc/apache22/httpd.conf there must be a RackBaseUri defined. See below

Example directory structure for the V2pp and MAPS installations:

  1. Note that the V2p0 and MAPS applications are currently configured for the development environment 
    • $ cd /v01/local/www/rails
$ ls -l
lrwxr-xr-x   1 alan  www    16 Jan 24 13:00 maps -> maps-app/public
drwxrwxr-x  13 root  www    21 Jan 24 13:00 maps-app
lrwxr-xr-x   1 root  www    15 Jan 17 16:20 v2p0 -> v2p0-app/public
drwxrwxrwx  15 alan  www    23 Jan 24 17:14 v2p0-app

Apache Configuration

The following is configured in usr/local/etc/apache22/httpd.conf

Procedure

  1. Log into zg-3.softxs.ch
    1. You must be a member of group www
  2. cd to the {app}-app dircetory
  3. Git pull/fetch. Typically: 
    • git pull origin master
  4. Bundle install 
    • bundle install
      Use 'sudu gem install' as necessary to install any missing Gems. Note that it appears that bundle install detects the presence of sudo and asks for a password for the bundle install if new gems must be installed. If any Gems are installed then you need to restart Apache: 
      • cd /usr/local/etc/rc.d
sudo ./apache22 restart
      Then check in /var/log/httpd-error.log to make sure there were no errors when Apache restarted.
  5. Run rake tasks as necessary. The typical list of rake tasks is: 
    • rake db:drop
rake db:create
rake db:migrate
rake db:seed_fu
rake db:populate
  6. If you need to perform any tweaks in the database, use the following user/password to access the DB server: 
    • mysql -uroot -psqladmin
  7. Test the result. Links:

    1. MAPS http://v2.softxs.ch/maps

    2. V2p0 http://v2.softxs.ch/v2p0

V2Deployment (last edited 2013-11-08 18:12:23 by 172)

Copyright 2008-2014, SoftXS GmbH, Switzerland