XML Workflow Tool

​http://dm2e.eu

The XML Workflow tool is a Java web application for running text conversion and text manipulation scripts. It is expandable by writing new scripts in either Python, XSL, Perl or directly in Java. The ​source code is available and there are also ​detailed instructions for building and using the tool.

​XML workflow tools

Input for a typical workflow is a transcription that was created following the Data Entry Specifications. The work is divided into three phases: checking, creating a well-formed XML and creating valid XML, all of which are described below in detail.

Phase 1: Checking

The text document is uploaded via the webpage and the scripts are generally started by clicking on the "Run" buttoon.

If there are errors, these are displayed in the "error" tab, other information is shown in the "console" tab

Because of errors being present in the input file, the document has to be modified locally using a text editor.

After re-running the script, the text has passed the test and the next steps can be taken.

Below the output of the current script is a window which suggests the next meaningful script in the workflow. The output of the current script is used as input to the next one. Of course, other scripts can be selected, as well.

Checking the pagebreaks is important for synchronizing the digital facsimiles with the transcription.

The webservice displays the pages with the first few lines of the content. The user now checks manually if the text corresponds to what is seen on the digital facsimile. Links are provided for comfortable checking.

In this case, the script is divided into two parts, the first part creating an configuration file which can be altered by the user (shown below). This is then evaluated in the second step.

The configuration file for the pagebreak script. The last entry should be removed. There is no corresponding pagebreak for that image in the transcription.

A useful feature is to show the effects of one script by displaying the current and the previous version side by side: the Diff.

The Diff tool showing the beginning of the text document after application of the pagebreak script. Green lines have been altered: the filename has been written behind each pagebreak-pseudo-tag (pseudo, because this is not really XML yet).

Another preparational step is the treatment of unknown characters. Characters that were not recognized during data entry assigned a code and collected on a list together with a screenshot of that character.

A configuration file takes care of these replacements with its corresponding Unicode character. This file is evaluated in the next step and the replacements take place.

Phase 2: Creating well-formed XML

Following these important preparational steps is the conversion to a well-formed XML document. These are additional replacements and the resolving of shorthands that were used during data entry.

The built-in help describes the functionality of each script.

One of the final steps is the insertion of metadata that reside already in the system. For that reason, the identifier of the document has to be put in at this point.

The first lines of the XML, displayed in the browser.

A script checks the XML if it is well-formed.

Phase 3: Creating valid XML

Although the XML document being well-formed does not mean that it is also valid to an XML schema. There are still a few steps to be taken. This is done in the third phase.

Floating elements like notes and images are moved away from their original places, being replaced by an anchor. The diff shows the effect of this.

Lateron, a div structure is added which is also used for creating a table of contents.

As a final stage, the validity is tested against a schema. In this case, the document is valid. In some cases, the document has to be edited locally to stand this test.

Extras

Further extras can be applied to the XML document, available in the bottom window. For example, mathematical formulas written as LaTeX can be converted to MathML here.

Upload

A valid XML file can then be uploaded in the Sandbox for further checking.

In addition to that, the file containing the metadata of the work has to be expanded with the path and name of the XML document so that it is also ​displayed in the ECHO display environment.

Operation status shows that the text has successfully been uploaded. During that process, the text is also analysed morphologically and connected to various dictionaries available in the system.

Online representations

​The text being displayed in the sandbox, maroon-coloured words can be clicked on, showing appropriate dictionary entries.

​Word information

Using the ​DM2E Mapping Server, the metadata information is converted various RDF models, for example the ​DM2E Data Model. With the data present in this format, the various RDF-aware tools available in the ​DM2E toolchain can be used and the data finally ingested into the ​DM2E triple store

​After ingestion into the DM2E triple store, the results can be easily browsed

Of course, the source has also been given to ​Europeana