Migration External Data into V2

V2Master

• Contents

  1. Migration External Data into V2
  2. Introduction
  3. Import Tools
     1. V2Import Rails Modul
        1. Module Classes
        2. Module Usage
        3. DSL File Format
        4. CSV File Format
     2. V2Import Rake Task

Introduction

Import Tools

V2Import Rails Modul

Module Classes

• A new module V2Import implemented in lib/v2_import.rb. The module has the following main classes:

  • Helper - methods used in other classes of the module
  • Reader::DSLReader - class to read and check the import DSL configuration file
  • Reader::CSVReader - read the input CSV file into a hash
  • Import - loads data into the database based on the result of the two previous class

Module Usage

• The module can be used from the Rails console as described in this chapter or the Rake wrapper can be used as described in chapter "V2Import Rake Task" below.

  • Staring the Rails console:

    $ rails c production  # The development system will be used if "production" omitted
    > ActiveRecord::Base.logger.level = 1 # Switch off SQL logging to allow to see error messages easily
    > Authorization.current_user = User.find(2) # Set the appropriate user to avoid authorization errors

  The module should be initialized first passing the filenames (CSV file, import configuration DSL file) and CSV options:

  • import = V2Import::Import.new csv_file, config_file, csv_options

  csv_options are option of the standard Ruby CSV.new method. They can be used to change the default separators (col_sep: ',', row_sep: 'auto', quote_char: '"'). The module initialization exits with exception on the first configuration error found. The initialzed module can be queried to show the internal data formats:

  • y import.config_hash # show the internal format of the import configuration, read from the DSL file
    y import.csv_hash    # show the internal format of the input CSV file

  The load method of the initialized module should be called to load the data into the database:

  • import.load(load_options)

  load_options:

  • insert_only: default = true, all records are new.
  • rollback: default = false, rollback at the end, good for testing the input.

  The load method doesn't exits on errors, but rollbacks every changes if data error occurred. The 2 calls can be chained of course:

  • V2Import::Import.new(csv_file, config_file, csv_options).load(load_options)

DSL File Format

• The file format explained on two samples. Pre-defined DSL files can be found in script/v2_import/. The first format is a document/revision/notes/classification import:

  • # encoding: utf-8                                # needed only to allow syntax check with ruby -c
    v2_import do                                     # outline block always v2_import
      header_line 5                                  # line no of the header line (default 1). Previous lines ignored.
      model Document do                              # the main model block, can contain defaults, attributes,
                                                     # classifications blocks
        keys :code                                   # list of table fields of a possible composed unique key to find database records to be updated (use symbols)
        defaults do                                  # default values for the containing model
                                                     # the right side can contain evaluable rails code
          resource_sub_type_id ResourceSubType.get_default_by_resource(Document).id 
          folder :foreign_key do                     # association can be only foreign_key to load default value
            attributes do                            # attributes block for the folder record
              code "WBS"                             # lookup attributes, the right side can contain evaluable rails
                                                     # code
            end
          end
        end
    
        attributes do                               # attributes block for the main record
          code "Nº CARTA"                           # the right side must contain a reference to a CSV column
          revisions :required do                    # association is also an attribute, it can contain defaults,
                                                    # attributes, classifications blocks again
                                                    # required option means - that the record is created anyway
            keys :code                              # unique key within the revision records associated to the document
            defaults do
              code "1"
              number 1
              user :foreign_key,  do
                attributes do
                  initials "AH /SXS"
                end
              end
              resource_sub_type :foreign_key, :required do # this is a required association as foreign key
                attributes do
                  code "Revision Type"
                end
              end
            attributes do
              date "FECHA DE EMISION"               # the right side must contain a reference to a CSV column
              title1 "DESCRIPCION"
              notes do                              # not required -> created only if at least 1 non-default
                                                    # attribute filled (in this case the note attribute)
                defaults do
                  issuer_id 1
                end
                attributes do
                  note "OBSERVACIONES"              # the right side must contain a reference to a CSV column
                end
              end
            end
            classifications do                      # classification of the revision, see description below
              # "zero level" classification, the value in CSV column will go into one of the value fields of object_classifications depending on data_tape.
              REASON  "Reason for Issue", lookup: nil 
            end
          end
        end
    
        # all classifications defined as
        # - code of the classification class (list)
        # - CSV column reference for the lookup value
        # - lookup option for the lookup field to find the classification
        #   - "lookup: nil" means a "zero level" classification
        # Note, that a classification class can appear more than 1 time to add more
        # classification in the same class.
        classifications do                    # classification of the document      
          ORG "DESTINATARIO", lookup: :name # Destination
          LOC "ESTADO",       lookup: :name # Status
        end
      end
    end

  The second example to load references (object_links) in a second step if both sides of the references are already loaded:

  • v2_import do
      model ObjectLink do
        attributes do
          left :foreign_key, :required, model: Document do  # define model, because polymorphic assoc.
            attributes do
              code "Left Code"
            end
          end
          right :foreign_key, :required, model: Document do # define model, because polymorphic assoc.
            attributes do
              code "Right Code"
            end
          end
        end
      end
    end

CSV File Format

• The input file is a standard CSV file, with blank lines and line preceding the header ignored.

V2Import Rake Task

• There is wrapper rake task to use the import module in lib/tasks/v2_import.rake. Run for help:

  • rake -D db:import

  An example:

  • rake db:import csv_file='/tmp/REGISTRO DE CARTAS 2014.csv' config_file='config/sites/astaldi/site/import/import_cfg_registro_de_cartas_dsl.rb' rollback=true