Differences between revisions 3 and 4

Migration External Data into V2

V2Master

Contents

Introduction

Import Tools

V2Import Rails Modul

Module Classes

A new module V2Import implemented in lib/v2_import.rb. The module has the following main classes:
- Helper - methods used in other classes of the module
- Reader::DSLReader - class to read and check the import DSL configuration file
- Reader::CSVReader - read the input CSV file into a hash
- Import - loads data into the database based on the result of two previous class

Module Usage

The module should be initialized first passing the the filenames (CSV file, import configuration DSL file) and CSV options:
- ```
import = V2Import::Import.new csv_file, config_file, csv_options
```
csv_options are option of the standard Ruby CSV.new method. They can be used to change the default separators (col_sep: ',', row_sep: 'auto', quote_char: '"'). The module initialization exits with exception on the first configuration error found. The initialzed module can be queried to show the internal data formats:
- ```
y import.config_hash # show the internal format of the import configuration, read from the DSL file
y import.csv_hash    # show the internal format of the input CSV file
```
The load method of the initialized module should be called to load the data into the database:
- ```
import.load(load_options)
```
load_options:
- insert_only: default = true, all records are new. Currently not implemented, probably goes into the DSL.
- rollback: default = false, rollback at the end, good for testing the input.
The load method doesn't exits on errors, but rollbacks every changes if data error occurred. The 2 calls can be chained of course:
- ```
V2Import::Import.new(csv_file, config_file, csv_options).load(load_options)
```

DSL File Format

The file format explained on tho samples. The first format is a document/revision/notes/classification import:

# encoding: utf-8                                # needed only to allow syntax check with ruby -c
v2_import do                                     # outline block always v2_import
  header_line 5                                  # line no of the header line (default 1). Previous lines ignored.
  model Document do                              # the main model block, can contain defaults, attributes,
                                                 # classifications blocks
    defaults do                                  # default values for the containing model
                                                 # the right side can contain evaluable rails code
      resource_sub_type_id ResourceSubType.get_default_by_resource(Document).id 
      folder :foreign_key do                     # association can be only foreign_key to load default value
        attributes do                            # attributes block for the folder record
          code "WBS"                             # lookup attributes, the right side can contain evaluable rails
                                                 # code
        end
      end
    end

    attributes do                               # attributes block for the main record
      code "Nº CARTA"                           # the right side must contain a reference to a CSV column
      revisions :required do                    # association is also an attribute, it can contain defaults,
                                                # attributes, classifications blocks again
                                                # required option means - that the record is created anyway
        defaults do
          code "1"
          number 1
          user :foreign_key do
            attributes do
              initials "AH /SXS"
            end
          end
          resource_sub_type_id ResourceSubType.get_default_by_resource(Revision).id
        end
        attributes do
          date "FECHA DE EMISION"               # the right side must contain a reference to a CSV column
          title1 "DESCRIPCION"
          notes do                              # not required -> created only if at least 1 non-default
                                                # attribute filled (in this case the note attribute)
            defaults do
              issuer_id 1
            end
            attributes do
              note "OBSERVACIONES"              # the right side must contain a reference to a CSV column
            end
          end
        end
      end
    end

    # all classifications defined as
    # - code of the classification class (list)
    # - CSV column nreference for the lookup value
    # - lookup option for the lookup field to find the classification
    # Note, that a classification class can appear more than 1 time to add more
    # classification in the same class.
    classifications do
      ORG "DESTINATARIO", lookup: :name # Destination
      LOC "ESTADO",       lookup: :name # Status
    end
  end
end

The second example to load references (object_links) in a second step if both sides of the references are already loaded:

v2_import do
  model ObjectLink do
    attributes do
      left :foreign_key, :required, model: Document do  # define model, because polymorphic assoc.
        attributes do
          code "Left Code"
        end
      end
      right :foreign_key, :required, model: Document do # define model, because polymorphic assoc.
        attributes do
          code "Right Code"
        end
      end
    end
  end
end

CSV File Format

The input file is a standard CSV file, with blank lines and line preceding the header ignored,

V2Import Rake Task

There is wrapper rake task to use the import module in lib/tasks/v2_import.rake. Run for help:

```
rake -D db:import
```

An example:

rake db:import csv_file='/tmp/REGISTRO DE CARTAS 2014.csv' config_file='config/sites/astaldi/site/import/import_cfg_registro_de_cartas_dsl.rb' rollback=true