= V2 Deployment = [[V2Master]] <> = Introduction = This page describes: 1. The server architecture for deploying publicly accessible MAPS and V2 systems 1. The software infrastructure that supports MAPS and V2 deployment 1. How to install publicly accessible beta instances of V2 and MAPS == See Also == 1. '''[V2SetupDevelopment]''' - Deployment and configuration of MAPS & V2 in development environments 1. '''[V2SetupProduction]''' - Deployment and configuration of MAP & V2 in production/prototype environments 1. '''[V2InstancesList]''' - List of static (e.g. non-generic) production/prototype instances = Server Architecture = == Overview == 1. Access to one or more physical servers, accessible via a limited number of public IP addresses 1. Use virtual hosts (DNS aliases) for addressing services and application instances 1. Use a reverse proxy server to dispatch to the appropriate server, virtual machine 1. The physical servers will collectively support multiple services: 1. Rails-based MAPS and V2 systems 1. PHP-based DrawMGT systems 1. Wiki systems 1. Web servers 1. DNS servers 1. CVS and Git servers 1. Services hosted in virtual machines: 1. Easy to migrate VMs to alternate physical servers 1. Host OS configuration does not need to be changed based on changing application requirements 1. Scaling to additional physical or could-based machines is possible 1. Automated deployment of V2 systems: 1. Creation of an instance of a largely pre-configured virtual machine 1. Provisioning of the virtual machine (VM) with all site and instance specific configuration 1. Deployment of the V2 application instance, including the site an instance specific configuration 1. Initially each instance of a V2 application will reside in its own VM 1. Automated monitoring: 1. MAPS and V2 instances 1. Virtual machines 1. Physical servers 1. Supporting infrastructure (DNS servers, reverse-proxy servers, etc.) == Supporting Technologies == || Server operating system || [[http://wiki.centos.org|CentOS 6.4]] || || Virtual Machines || [[http://www.vagrantup.com|Vagrant]], which runs on top of [[https://www.virtualbox.org|VirtualBox]] || || VM provisioning || [[https://puppetlabs.com/|Puppet]], a Ruby-based system configuration tool || || Application deployment || [[https://github.com/capistrano/capistrano|Capistrano]], a Ruby Gem for deploying Rails applications || || Reverse proxy server || [[http://httpd.apache.org/docs/2.2/mod/mod_proxy.html|mod_proxy]], an Apache module || || Monitoring || [[http://www.nagios.org/|Nagios]] and [[http://oss.oetiker.ch/rrdtool/|RRDTool]] || 1. Selected CentOS because: 1. Acceptable to corporate and enterprise clients 1. Supports Vagrant and Virtual``Box (FreeBSD is currently unable to host 1. The purpose of the reverse proxy server is to route incoming web requests to the appropriate server and virtual machine 1. Selected Apache mod_proxy because: 1. We have experience configuring and using Apache 1. We do not have high performance requirements and therefore don't need to be particularly concerned about choosing the best performing reverse proxy server. 1. Our highest traffic DrawMGT sites, which have 500-800 users, generate at most 25,000 requests per week, meaning (given a five day week and a 10 hour day) an average only 500 requests per hour. E.g. significantly less than one per second 1. Apache and mod_proxy can be swapped out and replaced with something else, without affecting the rest of the infrastructure == Server and DNS Architecture == The following shows the DNS and IP address configuration for a system hosting multiple applictation VMs: || '''Desc.''' || '''Dir''' || '''Domain Name''' || '''External Name''' || '''IP Address''' || '''Description''' || || VM Host || ''n/a'' || vh01.softxs.ch || vh01.softxs.ch || 192.168.2.4 || Host for virtual machines || || Services || mgt || mgt.vh01.softxs.ch || puppet.softxs.ch || 172.16.1.1 || Puppet server || || Gateway || gw || gw.vh01.softxs.ch || ''none'' || 172.16.1.2 || Reverse Proxy to VMs || || Wiki || wiki || wiki.vh01.softxs.ch || ''none'' || 172.16.1.3 || Wiki server || || Web || www || www.vh01.softxs.ch || ''none'' || 172.16.1.4 || Web server || || Nagios || nagi || nagi.vh01.softxs.ch || nagi.softxs.ch || 172.16.1.5 || Nagios monitoring server || || Test-1 || vmt1 || ''none'' || ''none'' || 172.16.2.1 || VM test server || || App-1 || vm0401 || vm0401.vh01.softxs.ch || vm0401vh01.softxs.ch || 172.16.4.1 || Application VM || || App-2 || vm0402 || vm0402.vh01.softxs.ch || vm0402vh01.softxs.ch || 172.16.4.2 || Application VM || || App-'''n''' || vm'''xxxx''' || vm'''xxxx'''.vh01.softxs.ch || vm'''xxxx'''vh01.softxs.ch || 172.16.'''x.y''' || Application VM || Notes: 1. '''Dir''' is the directory where the VM resides on the virtual host. It is a located in '''/home/vms''' 1. Notation: 1. '''n''' - The application instance number, a decimal number from 1 to 255 1. '''xxxx''' - The application instance number plus 0x0400, expressed in (lower case) Hex. 1. '''x.y''' - The applictaion instance number expressed as two integers suitable for an IP address 1. Examples: || '''Host''' || '''Name''' || '''IP Address''' || || App-1 || vm0401 || 172.16.4.1 || || App-163 || vm04a3 || 172.16.4.163 || || App-258 || vm0502 || 172.16.5.2 || 1. The VM Host's ip address: 1. For vh01, the IP address is set to 172.16.2.4, but can but this can be changed to any IP address appropriate to its environment 1. Should be accessible, possibly via NAT, from the public Internet 1. Additional VM Hosts can be added by incrementing 1. The same 172.16.x.y subnets can be used on every VM Host as they are not accesible out side the VM Host 1. A DNS server must run on the virtual host, which must provide DNS lookup for, at a minumum, the '''vh01''' sub-domain 1. The vh01.softxs.ch sub-domain is only available to the vh01 host and it's VMs 1. The external DNS names for the VMs all map to the vh01 host, E.g. 192.168.2.4 1. Pre-configuration of DNS entries and Reverse Proxy configuration: The following should be pre-configured (e.g. generated by scripts): 1. DNS entries for the vm'''xxxx'''.vh01.softxs sub-domain to 172.16.4.x address mappings 1. DNS entries for the vm'''xxxx'''vh01.softxs name to 172.16.2.4 address mappings 1. Reversy proxy configuration files mapping the external vm'''xxxx'''vh01.softxs name to the 172.16.4.x addresses 1. With Ubuntu the configuration files can be placed in an ''available-sites'' directory and then symbolic links can be created in an ''activated-sites'' directory in order to enable them (which requires restarting Apache) 1. Subnets 172.16.3.0/24 and 192.168.4.0/24 are Vagrant ''hostonly'' networks and are not directly accessible from outside the vh01 host 1. Subnet 172.16.1.x is reserved for individually managed servers. E.g. DNS, mail, Web, Blog, etc. servers 1. Subnet 172.16.4.x is reserved for automatically managed application servers 1. Additional subnets above 172.16.4.x can be added in case more than 256 VMs are hosted in a single host (e.g. 172.16.5.x) 1. For ''white labeling'' (custom customer specified domain names): 1. Create a DNS aliases that maps to the VM host. Note that for non-''softxs.ch'' domains, the customer will be responsible for the creation of DNS entry 1. Create a virtual domain configuration file for the reverse proxy in the gateway machine that maps the domain name to the internal ip address 1. See: http://ubuntuguide.org/wiki/Apache2_reverse_proxies = Server Deployment = == General Recommendations == 1. Have spare hardware capacity 1. Do not have idle backup servers, but use extra servers in production roles 1. Run multiple identical servers with load split between them 1. Regularly migrate services to different servers to ensure: 1. It is possible and that there are no hidden problems 1. We can rapidly and reliably restore services in the event of a failure == Hardware == 1. 64-bit Intel architecture 1. Multi-core, fast CPU(s) 1. Lots of RAM 1. SSD (non-mirrored) disk for host operating system and the active part of the guest VMs (e.g. the guest OS, Rails, web and DB servers) 1. Mirror/RAID hard disk for application data and backup staging Note that ''Virtualization'' must be enabled in the underlying PC BIOS in order for Vagrant and Virtual``Box VMs to function == Infrastructure Services == === Puppet Based Server Configuration === 1. All servers should be managed using ''Puppet'' 1. The configuration information, for our ''entire'' infrastructure should be stored in a single Git repository 1. Located at: http://git.softxs.ch/home/git/gitroot/v2infra.git 1. All configuration changes should be done on the ''Puppet Server'' and propagated to the target server === Provisioning of Applications Created by MAPS === Applications sold by MAPS will be provisioned and managed automatically: 1. DNS entries 1. Reverse proxy configuration 1. VM provisioning - Network configuration and hostname 1. Application deployment - Site and instance configuration 1. Backup - Configuration and execution 1. Monitoring === New Servers === The new servers should be implemented in VMs on new servers. 1. '''MAPS Server''' - Master Application and Payment System server, including public web site and sales font-end 1. '''Puppet Server''' - Automated server configuration via ''Puppet'' 1. '''Reverse Proxy Server''' - Routing incoming web requests to the back-end VMs and applications responsible for serving them === Existing Servers === The existing servers are currently running on dedicated servers and should be migrated to separate VMs. 1. '''DNS Servers''' - Public DNS servers. Currently residing in Zug (zg-1.softxs.ch) and Budapest (bp-1.softxs.hu) '''Partially DONE''' 1. '''Mail Server''' - Incoming and outgoing email and IMAP based email access. Handling mail aliases for DrawMGT customer systems 1. '''Git and CVS Servers''' - Source control systems 1. '''Wiki Server''' - Moin``Moin wikis '''DONE''' 1. '''Web Server''' - Static web pages '''DONE''' 1. '''Blog Server''' - DrawMGT customer Moin``Moin based Wikis and Word``Press based blog system 1. '''DrawMGT Server''' - Host for DrawMGT PHP based applications == Virtual Machines == 1. VMs are created and initilized (e.g. network configuration) by ''Vagrant'' 1. Initially, deploy one VM per application instance. E.g. each application instance has its own VM 1. For paid applications, the application's VM will be located on an SSD 1. For free (or low-cost) applications, the application's VM will be located on a normal hard disk 1. File sharing from the VM to the host system: 1. Puppet configuration files - ''Not sure if this needed, given a Puppet server'' 1. Backup staging area - For application documents, database backups and log files 1. Application documents === VM Provisioning === 1. VM boxes, which are templates for creating new VMs, are stored in a directory: '''/home/vagrant/boxes''' 1. Standard boxes VM boxes can be downloaded from: [[http://www.vagrantbox.es]] 1. We will deploy a CentOS 6.4 box, with Rails infrastructure, Apache, Passenger and MySQL already installed (but not completely configured) 1. VM instances, which can be either ''running'', ''suspended'' or ''halted'', are stored in a directory: '''/home/vagrant/VirtualBox VMs''' 1. VMs are created on the host system: 1. In one directory per VM: Suggest '''/home/vm/{name}''' 1. The VM directory contains '''Vagrantfile''' that defines the VMs basic properties: 1. The base box upon which it is based 1. Network and hostname configuration 1. Directories shared with the host OS 1. Configuration options: memory limits, etc. 1. Note that Vagrant allows multiple VMs to be defined in a single Vagrant file. We will not use this feature 1. The host's '''vagrant''' user is used to access the VM from the command line 1. 1. Create and save an SSH key for accessing the machines? ==== Tips ==== 1. Vagrant uses Virtual``Box, by Oracle, to manage VMs 1. Use the '''VBox``Manage''' command to monitor VMs. Must be run as user ''vagrant'' (running displays an empty list!) 1. '''VBox``Manage list [-l] vms''' - Lists all VM instances that have been defined 1. '''VBox``Manage list [-l] runningvms''' - Lists all VM instances that are running 1. Always shutdown a VM before attempting to destroy it. Destroying a running VM will sometimes hang == VM Provisioning == * '''''To be completed - describe:''''' * DNS config (and naming convention) * Revery proxy config * Network config * Rails setup * Ruby version 1.9.x * Rails and associated Gems * Apache setup and config * Phusion Passenger apache module * Puppet = Application Deployment = 1. Includes MAPS and V2 deployment * '''''To be completed - describe:''''' * Application issues * Gemfile.lock * Apache config * Site/instance Git repository * Structure of repository * Deployment files * database.yml * Capistrano 1. System deployment 1. Application upgrade == V2 and MAPS Application Deployment == 1. See [[V2CapistranoDeployment]] == V2 and MAPS System Setup == See the following pages: 1. [[V2SetupDevelopment]] - Describes how to setup and bootstrap MAPS and V2 in a development or prototype environment 1. [[V2SetupProduction]] - Describes how to setup and bootstrap MAPS and V2 in a production environment == Deployment Types == 1. '''Predefined''' systems, based on an application, site and instance. 1. Use cases: personal development systems, prototype systems, production MAPS system, large-customer systems, white labelled systems 1. Requires manual configuration, some of which might be automatically generated 1. Deployment is automated, provided the configuration is completely defined 1. '''Generated''' systems, based on a product, which determines an application and a pre-defined configuration 1. Use cases: v2 instances, trial systems, demo systems 1. Requires a configuration template, which is created manually 1. The configuration template: 1. Makes use of other configuration elements (e.g. host machine for VMs, DNS entries, proxy server configuration, etc.), which must be prepared in advance 1. Allows multiple instances of pre-configured applications to be automatically deployed == Deployment Steps == The complete end-to-end list of deployment steps: 1. Pre-configuration: 1. Virtual Machine Host 1. DNS 1. Reverse Proxy 1. Virtual machine setup 1. Setup directories on virtual host for VM 1. Vagrant file creation 1. Bring up VM 1. Setup NHS shares to host machine 1. File system for documents 1. File system for backups 1. Provision VM 1. Application deployment 1. Setup installation directories on VM 1. Checkout from Git and deploy on VM 1. Migration: 1. Database 1. Documents and static file 1. Setup automated backups === Pre-Configuration === The following items must be prepared in advance. Typically the configured is either performed by hand, or defined by hand in Puppet configuration files. 1. '''Virtual Host''' (VH) preparation - A system that hosts virtual machines 1. Base server hardware and operating system prepared by hand 1. System provisioning performed by Puppet 1. '''DNS entry''' preparation - DNS entries for accessing virtual machines 1. Sets of entries for multiple instances prepared in advance by hand in Puppet configuration files 1. Deployment performed by puppet 1. '''Reverse Proxy''' configuration - Defines forwarding rules for external web access to application instances on virtual machines 1. ets of entries for multiple instances prepared in advance by hand in Puppet configuration files 1. Deployment performed by puppet == Application Deployment Configuration Variables == || '''Name''' || '''Example''' || '''Description''' || || application || v2p0 || Name of application to be deployed || || repository || git.softxs.ch:/home/git/gitroot/v2p0.git || Git repository || || scmuser || v2 || User name for Git checkout || || deployuser || v2 || Deployment user name, for file permissions || || deploygroup || v2 (www?) || Deployment group name, for file permissions || || deploypath || /v01/local/www/rails/v2p0-app || Path to deploy application to || || branch || release-x || Git branch to be installed || || vbox || lucid64_v2base2 || Name of Vagrant virtual machine base-box || || vhost || vh01.softxs.ch || Virtual hostname || || vhip || 192.168.4.1 || IP address of VH || || vmhost || vm0401vh01.softxs.ch || Hostname of VM, for external access || || vmiip || 172.16.4.1 || IP address of VM, for external access || || vmxip || 172.16.2.4 || IP address of VM, for deployment, only viewable from VH || || vmsshport || 2204 || Port number for direct access to VM || '''Notes''' 1. '''vmsshport''' - Must be a unique value from within a VM. Suggest using the '''x.y''' value = DMZ VM System = This section describes the configuration on the Rails deployment environment on the loki.softxs.ch server, located in the DMZ in the AH server room. == Current Services Provided in DMZ == All these services should be migrated to VM based servers. 1. idun.softxs.ch - email and DNS server 1. Incoming and outgoing SMPT mail transfer 1. IMAPS access to email 1. Storage of email 1. DNS server for DMZ, not used publicly 1. honir.softxs.ch - web server 1. Private web pages 1. Mackay family tree 1. Robert L. Mackay Diaries 1. Innovation web site 1. Salandra 1. Sho Takahashi 1. Alan 1. Venture internal pages 1. Business web pages 1. Demo systems 1. Hydro-2007 1. ITA-Demo 1. LTF-Demo 1. SpecMGT-Demo 1. Web based access to email, via [[http://squirrelmail.org|SquirrelMail]] 1. Wikis, via [[http://moinmo.in|MoinMoin]] 1. SoftXS public web pages 1. DrawMGT documentation in English and German 1. Internal Wiki 1. Customer Wikis 1. IDP 1. Cardenillo 1. MET 1. NDD 1. Sisimiut 1. HCMC2 1. AHYTRA - hydrodynamic modeling software and associated web-based plotting and display 1. Available via http://lu.softxs.ch/venture/ahytra 1. ymir.softxs.ch - Source control and blog server 1. CVS repositories 1. Git repositories 1. Git repository web access, via [[https://git.wiki.kernel.org/index.php/Gitweb|gitweb]] 1. Blogs, via [[http://wordpress.org/|WordPress]] 1. Salandra - AH private blog. http://blog.salandra.ch 1. SoftXS - Unused. http://blog.softxs.ch 1. DrawMGT test systems (can be discarded) == VM Server #1 == === Hardware === 1. CPU: Intel Core i5-3330 CPU @ 3.00GHz, Quad Core, SandyBridge 1. Main memory: 16 GB 1. Disk: 1. 1 x 128 GB SSD, KINGSTON SH103S3120G 1. 2 x 2 TB Western Digital Caviar Green, WDC WD20EARS-00MVWB0 1. Network: 1. Onboard 1 GB/Sec ethernet -- unusable 1. PCI card with 10/100 MB/sec ethernet === Basic Configuration === || Hostname, internal || loki.softxs.ch || || IP Address, external || 81.221.23.36 || || IP Address, DMZ || 192.168.2.4 || === Network Access Configuration === 1. ADSL router configuration 1. Hardware: Cisco 826 1. NAT Forwarding {{{ Pro Inside global Inside local Outside local Outside global tcp 81.221.23.36:22 192.168.1.36:22 --- --- # SSH tcp 81.221.23.36:25 192.168.1.36:25 --- --- # SMTP tcp 81.221.23.36:53 192.168.1.36:53 --- --- # DNS TCP udp 81.221.23.36:53 192.168.1.36:53 --- --- # DNS UDP tcp 81.221.23.36:80 192.168.1.36:80 --- --- # HTTP tcp 81.221.23.36:443 192.168.1.36:443 --- --- # HTTPS tcp 81.221.23.36:465 192.168.1.36:465 --- --- # SMTPS tcp 81.221.23.36:993 192.168.1.36:993 --- --- # IMAPS }}} 1. Firewall configuration 1. Hardware: [[http://soekris.com|Soekris]] net4501 1. Firewall software: [[http://m0n0.ch/wall|mon0wall]] 1. NAT forwarding configuration. Includes {{{ IF Proto Src Dest IP Source IP Dst Description WAN TCP 22 192.168.2.4 (ext. 192.128.1.36) 22 Incoming SSH to loki WAN TCP/UDP 53 192.168.2.4 (ext. 192.128.1.36) 53 Incoming DNS to loki WAN TCP 80 192.168.2.4 (ext. 192.128.1.36) 80 Incoming HTTP to loki WAN TCP 443 192.168.2.4 (ext. 192.128.1.36) 443 Incoming HTTPS to loki }}} === File Systems === '''TODO''' The 2 x TB WD disks have not been formatted as mirrored disks. {{{ Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_loki-lv_root 25G 8.2G 15G 36% / tmpfs 7.8G 0 7.8G 0% /dev/shm /dev/sdb1 485M 38M 422M 9% /boot /dev/mapper/vg_loki-lv_home 78G 14G 61G 19% /home }}} === User Configuration === 1. Vagrant 1. Puppet == VM Configuration == Summary of VMs: 1. Puppet Server VM 1. DNS and DHCP Server VM 1. Reverse Proxy VM 1. V2 Server VMs 1. Web Server VM -- '''DONE''' 1. Wiki Server VM -- '''DONE''' 1. Blog Server VM -- '''TODO''' === Puppet Server VM === '''TODO''' 1. Needs to setup additional port forwarding for ports 8140, 61613, and 443 1. See: http://docs.puppetlabs.com/pe/2.0/install_preparing.html Items to document for each VM: 1. Location on host files system 1. Guest BOX 1. Local port forwarding 1. File system shares === DNS and DHCP Server VM === '''TODO''' === Reverse Proxy VM === '''TODO''' === V2 Server VMs === '''TODO''' = ZG3 Beta Systems = This sections describes the configuration of the Rails environment on the zg-3.softxs.ch server, located in the Datawire datacenter in Cham. == Environment == The V2 server is a virtual host on the zg-3.softxs.ch server. The server has following software systems: 1. FreeBSD 9.0-RELEASE FreeBSD 9.0-RELEASE #0 1. Apache Server version: Apache/2.2.23 (FreeBSD) 1. Phusion Passenger apache module 1. Ruby: ruby 1.9.3p327 (2012-11-10 revision 37606) [amd64-freebsd9] 1. Rails 3.2.11 1. mysql Ver 14.14 Distrib 5.5.28, for FreeBSD9.0 (amd64) using 5.2 1. And many Gems The installation location for Rails applications: 1. /v01/local/www/rails For each application two items are required: 1. In the rails directory the following is required: 1. A directory with the path {app}-app where the git repository is cloned 1. A symbolic link {app} which points to the {app}-app/public directory 1. In /usr/local/etc/apache22/httpd.conf there must be a Rack``Base``Uri defined. See below Example directory structure for the V2pp and MAPS installations: 1. Note that the V2p0 and MAPS applications are currently configured for the development environment {{{ $ cd /v01/local/www/rails $ ls -l lrwxr-xr-x 1 alan www 16 Jan 24 13:00 maps -> maps-app/public drwxrwxr-x 13 root www 21 Jan 24 13:00 maps-app lrwxr-xr-x 1 root www 15 Jan 17 16:20 v2p0 -> v2p0-app/public drwxrwxrwx 15 alan www 23 Jan 24 17:14 v2p0-app }}} == Apache Configuration == The following is configured in usr/local/etc/apache22/httpd.conf {{{ ## ====== Rails ====== LoadModule passenger_module /usr/local/lib/ruby/gems/1.9/gems/passenger-3.0.17/ext/apache2/mod_passenger.so PassengerRoot /usr/local/lib/ruby/gems/1.9/gems/passenger-3.0.17 PassengerRuby /usr/local/bin/ruby19 DocumentRoot /v01/local/www/rails ServerName v2.softxs.ch Allow from all # -- v2p0 app RackBaseURI /v2p0 RailsEnv development Options -MultiViews # -- maps app RackBaseURI /maps RailsEnv development Options -MultiViews # -- test0 app RackBaseURI /test0 RailsEnv development Options -MultiViews # -- test1 app RackBaseURI /test1 RailsEnv test Options -MultiViews }}} == Procedure == 1. Log into zg-3.softxs.ch 1. You must be a member of group www 1. cd to the {app}-app dircetory 1. Git pull/fetch. Typically: {{{ git pull origin master }}} 1. Bundle install {{{ bundle install }}} Use 'sudu gem install' as necessary to install any missing Gems. Note that it appears that bundle install detects the presence of sudo and asks for a password for the bundle install if new gems must be installed. If any Gems are installed then you need to restart Apache: {{{ cd /usr/local/etc/rc.d sudo ./apache22 restart }}} Then check in /var/log/httpd-error.log to make sure there were no errors when Apache restarted. 1. Run rake tasks as necessary. The typical list of rake tasks is: {{{ rake db:drop rake db:create rake db:migrate rake db:seed_fu rake db:populate }}} 1. If you need to perform any tweaks in the database, use the following user/password to access the DB server: {{{ mysql -uroot -psqladmin }}} 1. Test the result. Links: 1. '''MAPS''' http://v2.softxs.ch/maps 1. '''V2p0''' http://v2.softxs.ch/v2p0