| | 1 | |
| | 2 | == Setup of McGill servers == |
| | 3 | |
| | 4 | (Andrews email) |
| | 5 | |
| | 6 | The transfers scripts for moving files from data3 to data1 are all online at https://github.com/ahankinson/renamer. It's basically just a set of steps that the images go through to get transformed into a standardized directory structure and format. It's written as a Django web application. |
| | 7 | |
| | 8 | The heart of the project is here: https://github.com/ahankinson/renamer/tree/master/renamer/helpers |
| | 9 | |
| | 10 | 'to_archive' is what moves the files from data3 to data1. It does all sorts of data transformations, since the data is delivered to you in lots of different formats: |
| | 11 | - If it's a zip file it will uncompress it |
| | 12 | - If it's a PDF file it will extract the page images to lossless PNG |
| | 13 | |
| | 14 | It will also perform a basic data integrity check by checksumming the files, and will store the originals so that they can be recovered if the conversion process goes wrong. It follows the MPIWG directory structure having pageimg and info directories, but also adds 'backup' for the originals. |
| | 15 | |
| | 16 | 'to_diva' is what moves the files from data1 to data7 for serving with the IIP Image Server (JPEG 2000). It uses the kakadu 'kdu_compress' script, and also generates a Diva-specific JSON file for doing the layout and formatting of the images in the viewer. |
| | 17 | |
| | 18 | The rest of the web application is really just built around these scripts. It uses the 'celery' process manager to queue up the processing jobs, but other than that it's pretty straightforward. |
