• V2Inspector2Synchronization

V2 Inspector2 Synchronization

V2Master | V2Inspector2Todo

Requirements

Goals

1. Project-specific input forms, allowing multiple input forms types for tag & scan records

2. File (photo) synchronization, noting that photos can never be updated, only created & deleted

   • Delete is only possible on the server
3. A future requirement is to allow for PDF files in addition JPEG images

Constraints

1. PouchDB and the sqlite-plugin-2 adapter are not reliable enough for managing attachments
2. PouchDB and the IndexedDB adapter do not allow us to exceed the browser storage quota
3. Therefore we must manage synchronization of files outside of PouchDB/CouchDB
   • This is also what the PouchDB developers recommend

Design

Overview

1. We will manage files (photos) in the existing RFID Rails app
2. A CouchDB server will be used to manage metadata records for the files
3. The Inspector2 app must store local status information about which files have been downloaded/uploaded:
   • This information can be stored in PouchDB and use filtered synchronization to not share it with the server
   • Is it possible store this information local storage instead?

File Metadata Example

• {
     "_id": "file-20151015T064937000Z-8d2d1c1997d3cfed",
     "_rev": "1-260876cc5829b2991e0b9353b45508b0",
     "scan_id": "scan-20151015T064936000Z-982885d34ee61c8a",
     "file_id": 23,
     "file_name": "DSC01166.JPG",
     "content_type": "image/jpeg",
     "path": "/system/photos/photos/000/000/023/original/DSC01166.JPG?1444891776"
     "created_at": "2015-10-15T06:49:37.000Z",
     "updated_at": "2015-10-15T06:49:37.000Z",
  }

1. This data is stored in the Inspector2 app's PouchDB and synchronized with the server's CouchDB

2. file_id is the Rail RFID app photos.id

3. path is the relative path on the Rails RFID app for uploading and downloading the file to/from the Rails RFID App

4. created_at and updated_at are managed by the Inspector2 app

5. The timestamps in _id and scan_id key fields have all non-alphanumeric characters removed

   • E.g. a JavaScript Date.now with dash, colon & period removed

   • Code: (new Date()).toISOString().replace(/\-|:|\./g,"")
   • Need to implement this in pouchdbService.newId

Synchronization

1. Must be able to recover from an aborted synchronization
   • Typically when PouchDB sync completes but the file synchronization fails to complete
2. We can assume that the CouchDB/PouchDB defines the correct state of the files
   • Including that some server files are missing?
   • How to recover from this on client? E.g. download files from an aborted upload from a different client
   • Probably can just display 'file not available'
3. Have to locally store individual positive acknowledgment records that indicate which files have been transferred

   • Two sets of records: uploaded files & downloaded files

File Key Cache

1. File Key Caches are needed to assist the synchronization process:
   • A File Key Caches Stores two caches of 'file keys'

   • File Key values are simply their _id key values

   • There must be a method that returns an arrays of keys from a File Key Cache
   • The existence of a File Key indicates that the file has been successfully transferred
2. There are two File Key Caches:
   • Downloaded files: File Keys for files that have been downloaded successfully
   • Upload files: File Keys for files that have not been uploaded successfully
3. The File Key Caches could be stored in:
   • PouchDB records that are not synchronized. E.g. using a synchronization filter

   • A separate PouchDB database, that is never synchronized. This may be possible. See: https://github.com/pouchdb/pouchdb/issues/952

   • Local storage: either normal Local Storage or IndexedDB records

File Download

1. Synchronize the PouchDB
2. Get the Downloaded File Key Cache array
3. Get the array of files on the server, using a view on the local PouchDB (which has just been synched)
4. Create an array of the differences between the two arrays. E.g. the files to be downloaded:
5. Loop over the differences array:
   • Download each file
   • On success: Add the File Key for the Downloaded File Key Cache
6. Create another array the the differences between the two arrays (in the other direction). E.g. the files that were deleted on the server:

   • Delete each file and its medium file

   • On success: Remove the File Key from the Downloaded File Key Cache

File Upload

New File Procedure

Each time a new file is created on the Inspector2 app:

1. Create a medium file
2. Create a new File Key entry in the Upload File Key Cache
3. Create a new PouchDB record for the file
4. On the Inspector2 app, keep the file existing processing code:

   • Change the photoService.idPath to use the use random hex part of the file's _id field for the sub-directory names

     • E.g. change paths like: "000/000/001"

     • To: "8d/2d/1c/file-20160605T061235000Z-8d2d1c1997d3cfed-original"

   • The file name should be the same as the File Key. E.g. the file records _id value

Upload Procedure

1. Synchronize the PouchDB
2. Get the Upload File Key Cache array
3. Loop over the array:
   • Upload each file
   • On success:
     • Delete the File Key for the uploaded file from the Upload File Key Cache

     • Update and add the file_id & path in the local PouchDB

4. Synchronize the PouchDB again

Notes and Restrictions

1. The Inspector2 app only allows file delete for files that have not yet been uploaded
2. How to know to purge files on Inspector2 when they have been deleted on the server
3. Reset application has to clear the local storage of uploaded/downloaded file keys
   • What happens if you reset but have aborted/incomplete uploads?
   • How to detect the difference between a record that has not been uploaded and a partially uploaded set of files?