Annotation of texttool-concept/texttools.tex, revision 1.1.1.1
1.1 dwinter 1: \documentclass[a4paper]{article}
2:
3: \usepackage[latin1]{inputenc}
4: \usepackage[T1]{fontenc}
5: \usepackage{ae}
6: \usepackage{url}
7: \usepackage{graphicx}
8:
9: \graphicspath{{graphics/}}
10:
11: \title{Draft: Proposal for a text tool architecture for ECHO}
12: \author{Robert Casties}
13: \date{\today}
14:
15: \begin{document}
16:
17: \maketitle
18:
19: \section{Introduction}
20: \label{sec:introduction}
21: In the context of ECHO ``text'' represents scholarly metadata as well
22: as full texts of sources. As such, text forms the glue between the
23: different objects in the ECHO corpus. To fully exploit the potential
24: of text for semantic access and interlinking, tools have to support
25: the automatic or manual generation of links between different objects
26: within the ECHO corpus.
27:
28: A viewing environment should present configurable views on all texts
29: that allow to exploit relations to other texts and objects.
30:
31: Four different fields can be
32: identified, for which tools have to be developed:
33: \begin{itemize}
34: \item the generation of XML-structures
35: \item the analysis of the corpora
36: \item the meaningful linking of texts
37: \item the generation of scholarly metadata.
38: \end{itemize}
39:
40:
41: \section{Requirements}
42: \label{sec:requirements}
43: The handling of large corpora makes it necessary to define a minimal
44: standard XML structure for these documents. This implies the
45: development of tools to convert existing document formats into
46: these standard formats. In addition, tools for editing documents in
47: these formats will have to be made available.
48:
49: A prerequisite for generating links between documents is the
50: possibility to analyse texts and adding the results of this analysis
51: to the document. In general, two different types of this analysis can
52: be distinguished: automatically generated analysis following defined
53: rules and the manual analysis by marking words depending on the
54: context.
55:
56: The analysis of corpora is the basis for automatically generated
57: linking of documents. For example the wordlists generated by
58: morphological analysis can serve as starting point for linking to a
59: dictionary or a grammar. Another example would be the usage of a
60: wordlist consisting of technical terms serving as basis for linking to
61: an encyclopedia or glossary. Furthermore such wordlists can serve as
62: starting points for cross-linking within the text corpus using this
63: word lists as a common anchor.
64:
65: Beyond the automatically generated linking of documents, the linking
66: as result of scholarly work has to be supported by text tools,
67: e.g. showing connections between different texts in the corpus,
68: combining sources with translations or secondary texts, the linking
69: between images and describing texts or the
70: connection between full texts and images.
71:
72: In particular, an open environment for adding comments and notes to
73: sources can be a test bed for how collaborative work on sources could
74: be encouraged by the ECHO project in order to build a virtual European
75: research area on cultural heritage.
76:
77:
78: \section{Technical issues}
79: \label{sec:formats}
80:
81:
82: \subsection{Granularity of reference}
83: \label{sec:gran-refer}
84:
85: The basic layer of informational markup has to define the units of
86: reference for the higher layers. The granularity of these reference
87: units determines the amount of complexity needed for referencing in
88: the higher layers. The markup in the basic layer should also permit
89: changes in formatting and corrections in the source document to a
90: certain extent without loosing referential integrity.
91:
92: The proposed unit of reference in the basic layer is a \emph{word}.
93: Where \emph{word} means any sequence of characters between whitespace
94: or other special characters in the source document, excluding
95: formatting and markup. The word as a unit of reference is not meant to
96: be a semantical unit or even a morphological unit. It is only meant to
97: be the smallest easily recognizable unit used in the text.
98: Morphological, syntactical and semantical units can be assembled and
99: referenced on higher level layers as \emph{terms} comprised of one or
100: more, not necessarily adjacent \emph{words}.
101:
102:
103: \subsection{Information layers}
104: \label{sec:information-layers}
105:
106: The text tools operate according to the ``standoff
107: principle'' of XML markup. The basic text is marked up only to provide
108: the basis of raw data and reference. Additional syntactical and
109: semantical information -- be it automatically generated or scholarly
110: edited -- is provided in separated informational layers of
111: \emph{word lists} referencing other layers or the basic text.
112:
113: \begin{figure}[htbp]
114: \centering
115: \includegraphics[width=0.8\textwidth]{word-termlists}
116: \caption{Relation of basic text and term lists. }
117: \label{fig:word-termlists}
118: \end{figure}
119:
120: A \emph{word list} or \emph{term list}\footnote{\emph{word list} and
121: \emph{term list} will be used interchangeably in the following text
122: since both forms should be functionally identical.} is a list of
123: \emph{words} or \emph{terms} that are each linked to a list of
124: references to \emph{words} or \emph{terms} in other \emph{word lists}
125: or to \emph{words} in basic texts.
126:
127: An example for the informational layers in an English or Latin
128: text\footnote{English or Latin as examples for languages where
129: sufficient morphological analysis can be based on single words.}
130: would be:
131:
132: \begin{enumerate}
133: \item \emph{Basic text} layer, marked up with \emph{words}.\label{item:1}
134:
135: \item \emph{Basic word list} layer, an automatically generated list of all
136: unique words and references to their occurrence in the basic text
137: (\ref{item:1}).\label{item:2}
138:
139: \item \emph{Morphological term list} layer, an automatically generated list
140: of the morphologically normalized forms of all words and references
141: to their occurrence in the basic wordlist (\ref{item:2}).\label{item:4}
142:
143: \item Scholarly edited \emph{term list} layer, a manually edited list of
144: semantical units like technical terms used in the document,
145: referring to the basic text (\ref{item:1}).\label{item:5}
146: \end{enumerate}
147:
148: Additional annotation layers referencing the basic text or any other
149: layer could be produced and stored in the same text repository or on any
150: other server. Therefore it has to be possible to reference any layer in
151: a unique and stable way across the net.
152:
153: In languages with more complex morphological units the morphological
154: analysis layer can be based on an intermediate term layer that joins
155: basic words into morphological units.
156:
157:
158:
159: \subsection{Primary and secondary source texts}
160: \label{sec:backr-orig-source}
161:
162: The text tool system should be easily adaptable to different
163: workflows dealing with text in the ECHO domain. There are two
164: basic types of text sources with a different degree of integration an
165: the central ECHO text corpus.
166:
167: %% FIXME!!
168:
169: The \emph{primary source text} is maintained in the basic word tagged
170: form on a text corpus server. Updates and changes have to be worked
171: into the word tagged text without breaking the referential integrity.
172:
173: As \emph{secondary source text} the basic word tagged text is not
174: the primary source. A mapping file has to be provided,
175: that maps the words in the basic text to other referenceable units in
176: the primary source documents. Updates and changes in the primary
177: document may be followed by updates to the mapping file or the basic
178: text to maintain referential integrity.
179:
180: The distinction between these types of sources concerns mainly the
181: text cruncher producing the basic tagged text and eventually a mapping
182: file and the presentation tools producing views or references to the
183: original source texts.
184:
185:
186:
187: \subsection{Support of additional markup}
188: \label{sec:supp-addit-mark}
189:
190: The basic text tagging format should be transparent to additional
191: markup in the source text to enable the easy integration of the text
192: tools into existing formats and tools. The use of XML namespaces can
193: provide such transparency.
194:
195: The common viewing environment can not be completely
196: agnostic to additional markup. It must be able to interpret a common
197: set of minimal visual markup. Visual elements to be considered are:
198:
199: \begin{itemize}
200: \item paragraphs and/or line breaks
201:
202: \item page breaks
203:
204: \item page images (coupled to page breaks)
205:
206: \item inline images
207: \end{itemize}
208:
209: When presenting text parts to the user as results to a search request
210: it would be useful to have a general mechanism to select larger units
211: around the referenced word. Additional semantical units suitable for
212: this kind of reference would be sentences. The mechanism could try to
213: select the surrounding sentence and then fall back to larger units
214: like a paragraph, a page or the whole text.
215:
216: A translation scheme to map different existing visual markup tags into
217: the common set for the viewing environment should be implemented. The
218: translation could be done directly upon creation of second source
219: texts as these texts are decoupled from the original source text.
220: The translation would have to be done on-the-fly for primary source
221: texts where markup different from the common set is used.
222:
223:
224: \section{Tools}
225: \label{sec:tools}
226:
227:
228: \subsection{Text cruncher}
229: \label{sec:text-cruncher}
230:
231: The \emph{text cruncher} tool takes a text file and eventual
232: information about a primary source and produces a \emph{basic word
233: tagged text}, a \emph{basic word list}, and an eventual
234: \emph{mapping file} if the text is to be considered a secondary source
235: text.
236:
237:
238: \subsection{Morphological analyzer}
239: \label{sec:morph-analys}
240:
241: The \emph{morphological analyzer} tool for a given language takes a
242: word list or a term list of morphological units and
243: produces a \emph{morphological term list} of normalized forms, their
244: morphological description, and references to their occurrences in the
245: provided list.
246:
247: A sub function of the morphological analyzer should be a normalizer for
248: single words to be used in conjunction with the dictionary tool.
249:
250:
251: \subsection{Dictionary}
252: \label{sec:dictionary}
253:
254: The \emph{dictionary analyzer} tool takes a morphologically normalized
255: term list and produces a term list with known terms,
256: references to their definitions and references into the occurrences in
257: the provided list.
258:
259: A sub function of the dictionary analyzer should be a lookup tool for
260: single normalized words or terms.
261:
262:
263: \subsection{Cross referencer}
264: \label{sec:cross-referencer}
265:
266: The \emph{cross referencer} tool takes a word list from one text
267: and a set of word lists from other texts and
268: produces a word list with words from the first list and
269: references into all of the lists.
270:
271:
272: \subsection{Display environment}
273: \label{sec:display-environment}
274:
275: The \emph{display environment} should be able to display a text with
276: minimal visual markup and additional links defined by additional
277: wordlists.
278:
279: The set of necessary visual markup like page breaks, page images,
280: inline images or text formatting should follow an agreed standard.
281:
282: The functionality provided by the links could be direct linking into
283: other texts, morphological analyses, or dictionary entries if the word
284: is only referenced by one word list. In the case of multiple
285: references to a word a mechanism for the selection of one of the
286: possible sources must be provided.
287:
288:
289: \subsection{List inverter}
290: \label{sec:list-inverter}
291:
292: The \emph{list inverter} is a small auxiliary tool that takes a
293: normal word list that is ordered by unique words and produces an
294: \emph{inverted word list} that is ordered by word references.
295:
296:
297:
298:
299: \section{Use cases}
300: \label{sec:use-cases}
301:
302:
303: \subsection{Integration of Archimedes XML texts}
304: \label{sec:integr-arch-xml}
305:
306: The XML texts of the Archimedes project could be integrated in two
307: different ways: either as primary source texts, adding basic word
308: tagging to the Archimedes markup or as secondary source texts by
309: providing mapping files to the unchanged source files.
310:
311: In the first case basic word tagging would be added to the XML
312: document by the text cruncher. The resulting documents could then be
313: further processed and edited, provided that word references are not
314: broken. The text cruncher would produce a basic word list for use with
315: other text tools.
316:
317: In the second case only a secondary source text and a mapping file
318: would be produced by the text cruncher together with the basic word
319: list. The original source text would stay unchanged outside the text
320: repository.
321:
322: Additional mappings would have to be generated to adapt the visual
323: markup used in the Archimedes XML to the common markup for the display
324: environment.
325:
326:
327:
328: \subsection{Integration of existing webpages}
329: \label{sec:integr-exist-webp}
330:
331:
332:
333: \subsection{Integration of raw OCR text}
334: \label{sec:integration-raw-ocr}
335:
336: Raw OCR text as it is generated by automatic OCR on digitized document
337: pages could be considered original source material. The OCR produces
338: one plain text document per scanned image file. A suitable text
339: cruncher would produce a secondary source text for use in the
340: repository with a mapping file referencing the original text files.
341:
342:
343:
344: \subsection{Full text search}
345: \label{sec:full-text-search}
346:
347: (to be done)
348:
349:
350: \subsection{Cross linking of texts}
351: \label{sec:cross-linking-texts}
352:
353: (to be done)
354:
355:
356: \section{Proposed formats}
357: \label{sec:proposed-formats}
358:
359:
360: \subsection{Basic document}
361: \label{sec:basic-docum-form}
362:
363: The basic document format consists of word tags, and optionally language information
364: for morphological analysis and basic visual markup.
365:
366: An example in pseudo XML markup might look like this:
367:
368: \begin{verbatim}
369: <text lang="lat">
370: <word id="1">omnia</word>
371: <word id="2">gallia</word>
372: <word id="3">est</word>
373: <word id="4">divisa</word>
374: <word id="5">in</word>
375: <word id="6">partes</word>
376: <word id="7">tres</word>.
377: </text>
378: \end{verbatim}
379:
380:
381:
382: \subsection{Basic wordlist}
383: \label{sec:wordlist}
384:
385: The basic wordlist consists of all unique words and references to
386: their occurrences in the basic text.
387:
388: \begin{verbatim}
389: <list id="1">
390: <list-entry id="1">
391: <word>patria</word>
392: <word-ref>xlink:bello_gallico#36</word-ref>
393: <word-ref>xlink:bello_gallico#157</word-ref>
394: <word-ref>xlink:bello_gallico#336</word-ref>
395: </list-entry>
396: <list-entry id="2">
397: <word>bello</word>
398: <word-ref>xlink:bello_gallico#189</word-ref>
399: <word-ref>xlink:bello_gallico#236</word-ref>
400: <word-ref>xlink:bello_gallico#557</word-ref>
401: <word-ref>xlink:bello_gallico#1396</word-ref>
402: <word-ref>xlink:bello_gallico#1450</word-ref>
403: </list-entry>
404: </list>
405: \end{verbatim}
406:
407:
408: \subsection{Term list}
409: \label{sec:term-list}
410:
411: A term groups one or more words into a semantical unit. A term list
412: contains chosen terms and references to their occurrences.
413:
414: \begin{verbatim}
415: <list id="1">
416: <list-entry id="1">
417: <term>patria nostra</term>
418: <term-ref>
419: <word-ref>xlink:bello_gallico#36</word-ref>
420: <word-ref>xlink:bello_gallico#37</word-ref>
421: </term-ref>
422: <word-ref>xlink:bello_gallico#36</word-ref>
423: <term-ref>
424: <word-ref>xlink:bello_gallico#155</word-ref>
425: <word-ref>xlink:bello_gallico#157</word-ref>
426: </term-ref>
427: </list-entry>
428: <list-entry id="2">
429: <term>belllo gallico</term>
430: <term-ref>
431: <word-ref>xlink:bello_gallico#12</word-ref>
432: <word-ref>xlink:bello_gallico#13</word-ref>
433: </term-ref>
434: </list-entry>
435: </list>
436: \end{verbatim}
437:
438:
439: \subsection{Primary source mapping}
440: \label{sec:prim-source-mapp}
441:
442: A primary source mapping maps every word of a basic document to its
443: equivalent in the primary source document.
444:
445: \begin{verbatim}
446: <source-mapping>
447: <map id="1">
448: <word-ref>xlink:bello_gallico#1</word-ref>
449: <ref>xlink:bello.txt(1235)</ref>
450: </map>
451: <map id="2">
452: <word-ref>xlink:bello_gallico#2</word-ref>
453: <ref>xlink:bello.txt(1245)</ref>
454: </map>
455: <map id="3">
456: <word-ref>xlink:bello_gallico#3</word-ref>
457: <ref>xlink:bello.txt(1257)</ref>
458: </map>
459: </source-mapping>
460: \end{verbatim}
461:
462:
463:
464: \section{Development priorities and time plan}
465: \label{sec:devel-prior-time}
466:
467: (to be done)
468:
469: \section{Glossary}
470: \label{sec:glossary}
471:
472: \begin{description}
473: \item[word] In a basic text a word is any sequence of characters
474: between delimiters of whitespace or other delimiters. A word on this
475: level is not a semantical, not even a syntactical unit.
476:
477: \item[term] A term is a container for one or more not necessarily
478: adjacent words. Terms can be syntactical or semantical units. Terms
479: can be used and referenced like basic words.
480:
481: \item[word reference] A word reference is an xlink or similar
482: reference to a word or term in a word list or in a basic text.
483:
484: \item[term reference] A term reference is a reference to a term and
485: equivalent to a word reference.
486:
487: \item[word list] A word list is a list containing elements consisting
488: of a word and a list of word references.
489:
490: \item[term list] A term list is equivalent to a word list. Its
491: elements consist of a term and a list of word references.
492:
493: \item[word occurrence list] A word occurrence list is a list where
494: every element is treated like a type and a list of all its instances
495: -- occurrences -- in the text. The same word (type) can occur only
496: once in an occurrence list where it can reference many word instances.
497:
498: \item[word instance list] A word instance list is a word list where
499: every element is treated like a singular object (unlike a word
500: occurrence list). The same word (type) can occur multiple times in an
501: instance list where it can reference only one word or term instance.
502:
503: \end{description}
504:
505:
506: \end{document}
507:
508: %%% Local Variables:
509: %%% mode: latex
510: %%% TeX-master: t
511: %%% End:
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/cvs/cvsweb/icons/back.gif){kind=icon}
Return to texttools.tex CVS log ![[TXT]](/cvs/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvs/cvsweb/icons/dir.gif){kind=icon}
Up to [Repository] / texttool-concept