| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Migration of DrawMGT Systems into V2 = | = Migration External Data into V2 = |
| Line 9: | Line 9: |
| = Migration of DrawMGT 14.0 = | = Import Tools = |
| Line 11: | Line 11: |
| == Status == | == V2Import Rails Modul == |
| Line 13: | Line 13: |
| Currently a simple SQL script (''script/migrate_drawmgt.sql'' ) is created as a prototype. | === Module Classes === |
| Line 15: | Line 15: |
| == Issues == |
A new module !V2Import implemented in lib/v2_import.rb. The module has the following main classes: * Helper - methods used in other classes of the module * Reader::DSLReader - class to read and check the import DSL configuration file * Reader::CSVReader - read the input CSV file into a hash * Import - loads data into the database based on the result of two previous class |
| Line 17: | Line 21: |
|
* How to migrate passwords? * ''Revision code'' is currently a simple numeric value in V2 and can not be set by the user. * No place to migrate ''Comments.abstract''. * How to migrate users with the same email address? Currently duplicate email addresses perpended by 'dup_#{users.id}_' if the user is obsolete or test user,<<BR>> or the user id belonging to the same email address is not the highest one. * How to migrate roles? * Classification display parameters are specified on the record level in DrawMGT but not in V2. * How to migrate classification without code (in case of not selectable parent items in DrawMGT), e.g. !LocationRef. Currently code is set to sortOrder. * It is possible in DrawMGT, that alignment is filled but chainage not. Should we allow it also in V2? |
=== Module Usage === |
| Line 30: | Line 23: |
| == Currently Ignored Fields == |
The module should be initialized first passing the the filenames (CSV file, import configuration DSL file) and CSV options: {{{ import = V2Import::Import.new csv_file, config_file, csv_options }}} '''csv_options''' are option of the standard Ruby CSV.new method. They can be used to change the default separators (col_sep: ',', row_sep: 'auto', quote_char: '"'). The module initialization exits with exception on the '''first''' configuration error found. |
| Line 32: | Line 30: |
| List fields currently ignored, but probably will be used later: |
The initialzed module can be queried to show the internal data formats: {{{ y import.config_hash # show the internal format of the import configuration, read from the DSL file y import.csv_hash # show the internal format of the input CSV file }}} |
| Line 34: | Line 36: |
|
|| '''Table(s)''' || '''Fields''' || || !ContractRef || restrictedFlag || || Groups || restrictedFlag || || !DocumentTypeRef || displayWorkFlowTypeId, layoutTypeId, restrictedFlag, parentId || || !RevisionTypeRef || restrictedFlag, parentId || || !CommentTypeRef || workFlowTypeId, layoutTypeId, restrictedFlag, parentId || || Users || lockedFlag, testUserFlag, restrictedFlag, externalKey, languageId, companyId, address1-3, officePhone, mobilePhone, fax, skypeId, url, passwordDate, description || || Tasks || '''only''' commentCode, groupId, firstResponsibleId, commentTypeId, firstCompleteDate, summary '''are migrated''' || || !TaskNotes || commentNoteTypeId, lockedFlag, restrictedFlag, obsoleteFlag, attachment, thumbnail || |
The '''load''' method of the initialized module should be called to load the data into the database: {{{ import.load(load_options) }}} '''load_options''': * insert_only: default = true, all records are new. Currently not implemented, probably goes into the DSL. * rollback: default = false, rollback at the end, good for testing the input. The '''load''' method doesn't exits on errors, but rollbacks every changes if data error occurred. |
| Line 44: | Line 45: |
| For classification tables: |
The 2 calls can be chained of course: {{{ V2Import::Import.new(csv_file, config_file, csv_options).load(load_options) }}} |
| Line 46: | Line 50: |
|
|| '''Table(s)''' || '''Fields''' || || !ProjectPhaseRef, !CategoryRef, !LocationRef, !WorkTypeRef, !SiteClassification[1-4]Ref || searchOnlyFlag, displayClass, displayTextInput, displayTextDetail, displayTextList, displayHover || || !AlignmentRef || minChainage, maxChainage || |
=== DSL File Format === |
| Line 50: | Line 52: |
| == Migration notes == | === CSV File Format === |
| Line 52: | Line 54: |
|
* Only users with userID >= 100 migrated. * '''Obsolote''' users migrated as '''disabled'''. * Users with '''loginDisabled''' migrated as '''external'''. * Id of primary ''email_addresses'' are the same as ''users.id''. * Duplicate ''email2'' from Users are not migrated. |
== V2Import Rake Task == |
Migration External Data into V2
Introduction
Import Tools
V2Import Rails Modul
Module Classes
A new module V2Import implemented in lib/v2_import.rb. The module has the following main classes:
- Helper - methods used in other classes of the module
- Reader::DSLReader - class to read and check the import DSL configuration file
- Reader::CSVReader - read the input CSV file into a hash
- Import - loads data into the database based on the result of two previous class
Module Usage
- The module should be initialized first passing the the filenames (CSV file, import configuration DSL file) and CSV options:
import = V2Import::Import.new csv_file, config_file, csv_options
csv_options are option of the standard Ruby CSV.new method. They can be used to change the default separators (col_sep: ',', row_sep: 'auto', quote_char: '"'). The module initialization exits with exception on the first configuration error found. The initialzed module can be queried to show the internal data formats:
y import.config_hash # show the internal format of the import configuration, read from the DSL file y import.csv_hash # show the internal format of the input CSV file
The load method of the initialized module should be called to load the data into the database:
import.load(load_options)
load_options:
- insert_only: default = true, all records are new. Currently not implemented, probably goes into the DSL.
- rollback: default = false, rollback at the end, good for testing the input.
The load method doesn't exits on errors, but rollbacks every changes if data error occurred. The 2 calls can be chained of course:
V2Import::Import.new(csv_file, config_file, csv_options).load(load_options)
DSL File Format
CSV File Format