toolkit_polytonicgreek
Table of Contents
Toolkit_PolytonicGreek is a GATE plugin developed by the Liris DRIM team. Its purpose is to group various processing resources aimed at processing ancient polytonic greek (and especially Koine Greek). Processing tasks range from language processing (normalisation, lemmatisation) to quotation retrieval.
Text reuse retrieval
In order to perform a text reuse research, you can use the following resources in the following order:
- Biblindex Parser to read your input files into structured xml
- Then load the xml files into GATE as langage resources, and use the following processing chain on them:
- Biblindex Tokeniser
- Polytonic Greek Normaliser
- Polytonic Greek Lemmatiser (as many times as necessary if you happen to have several lemma files)
- Biblindex Structure Tagger
- Now you should have processed text for both the reused text and documents that reuse it
- Use Biblindex Processed Text Export on the Language Resource containing the reused text
- Then run Biblindex Reuse Finder using the xml export from the previous step on the Language Resources that contain text reuse
The result of this chain will be Language Resources with annotations indicating text reuse. There is no xml output for those yet (discussions for the exact output format are still being held).
Available resources
Overall, the available Processing Resources provided by this plugin are divided into two sets: the Polytonic Greek set, which focuses on processing general Greek text, and the Biblindex set, which adds concerns about the text structure (this part being specific to the formats having been used for the Biblindex ANR project -not that these formats are limited to Biblindex in any way, see below).
Polytonic Greek subset
| Resource | Polytonic Greek Word Break Handler |
|---|---|
| Resource type | Corrector - this PR modifies a Language Resource. Be careful not to run any annotating resource before a Corrector: annotations are characterised by their offset from the first character of the document, and Correctors may insert or delete characters. |
| Dependencies | none |
| Parameters | none |
| Description | This PR deals with word breaks in a text by reattaching the word and putting it on the line of its beginning. The code is somewhat specific to the format used during the Biblindex project to allow for text structure markup between both ends of the words (which means that any line beginning with numbers will be ignored in the process of searching for the end part of a word break). |
| Screenshot |  |
| Resource | BibleWorks Code Tokeniser |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | none |
| Parameters | none |
| Description | This PR recognises words and spaces in polytonic greek documents using the BibleWorks code. BibleWorks code uses many symbols to represent character alterations, so a conventional tokeniser does not yield good results. |
| Screenshot |  |
| Resource | Polytonic Greek Transcoder |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | GATE Unicode Tokeniser or BibleWorks Code Tokeniser |
| Parameters | input format: specifies the format of the input document, which defines which transcoders will be used; input format must be one of 'unicode', 'betacode' and 'bibleworks code'. |
| Description | This PR annotates each word with its transcoded forms: unicode, beta-code and BibleWorks code. Any of these formats is accepted as input (beta-code follows the Perseus lowercase implementation though). However, when using beta-code or BibleWorks code as input, you must also use a relevant tokeniser (we provide one for BibleWorks, and there is one for beta-code in the Perseus Hopper). |
| Screenshot |  |
| Resource | Polytonic Greek Normaliser |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | GATE Unicode Tokeniser or Polytonic Greek Transcoder |
| Description | This PR annotates each word token with its normalised form. |
| Screenshot |  |
| Resource | Polytonic Greek Lemmatiser |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | GATE Unicode Tokeniser or Polytonic Greek Transcoder, lemma table (see description) |
| Parameters | lemma table: the url of a lemma table (see below) |
| Description | This PR uses a lemma table to annotate each word with its lemma. It was built for statistical text analysis and not for critical edition purpose, and as such there is no disambiguation process (as hinted by the absence of grammatical analysis). You will need a lemma table containing lines in the format 'term==lemma' as input. The lemma table used in Biblindex is available here. Please keep in mind that lemmatisation in this PR is done with statistical processing in mind, so it will not try to guess which lemma is the right one when more than one lemma is available, rather using the first on the list (you can however specify several lemmata for a single term using the ';;' separator ('term==lemma1;;lemma2;;lemma3'). |
| Screenshot |  |
| Resource | Greek Lemma Export |
|---|---|
| Resource type | File Writer - this PR does not alter annotations and writes data into a file. |
| Dependencies | Polytonic Greek Lemmatiser |
| Parameters | destination file: the url of the file designed to store the output. |
| Description | This PR writes the content of the document with the lemma annotations to a file. It has no purpose other than providing some example code. |
| Screenshot |
Biblindex subset
| Resource | Biblindex Parser |
|---|---|
| Resource type | File Writer - this PR does not alter annotations and writes data into a file. |
| Dependencies | none |
| Parameters | source file: the url of the source file. destination file: the url of the file designed to store the output xml. data type: the dtat type of the input file; must be either 'rtf/tlg' for a tlg-styled rtf document or 'txt/bibleworks' for a BibleWorks text export. |
| Description | This PR converts a file written in one of the file formats used during the Biblindex ANR project (BibleWorks export file or TLG-styled RTF file) to a xml file. Word break handling and transcoding are taken care of when necessary. Please note that both input and output files must be given as parameters and that this PR does not make use of Language Resources. Files written by this PR can however be opened as xml Language Resources to be processed further. This PR acts as a Text Structure annotator (the GATE xml file reader keeping track of the xml annotations). It rally should process an already loaded LR ideally, but the code originally designed outside of the GATE platform has not been adapted yet outside of rendering it suitable for this PR. |
| Screenshot |  |
| Resource | Biblindex Tokeniser |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | xml file compliant with one of the Biblindex DTDs (output from the Parser for instance) read by GATE with markup awareness |
| Parameters | none |
| Description | This PR uses the xml markup to split the document into tokens. It should perform more quickly than the GATE Unicode Tokeniser in the specific case of a document compliant with a Biblindex DTD. |
| Screenshot |
| Resource | Biblindex Structure Tagger |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | xml file compliant with one of the Biblindex DTDs (output from the Parser for instance) read by GATE with markup awareness, Gate Unicode Tokeniser or Biblindex Tokeniser |
| Parameters | none |
| Description | This PR enriches each detected token in the processed Language Resource with structural information taken from the xml structure. This PR assumes that each processed LR is a xml file outputted by the Biblindex Parser PR. This PR will not transfer the structure annotation from the Original markups subset to the main subset. To this effect please use the Annotation Transfer PR from the Tools GATE plugin. |
| Screenshot |  |
| Resource | Biblindex Processed Text Export |
|---|---|
| Resource type | File Writer - this PR does not alter annotations and writes data into a file. |
| Dependencies | Biblindex Structure Tagger, Polytonic Greek Normaliser, Polytonic Greek Lemmatiser |
| Parameters | destination file: the url of the file designed to store the output xml. |
| Description | This PR stores the text and the annotations from the previous PRs in a xml form in a file. The xml format follows the structure of the text (see the Biblindex Parser PR for details). The output of this PR is basically the same as the output from the Biblindex Parser PR, except with normalised and lemmatised form for every word. If no data about these form are found in the annotations, the form that appears in the text will be used instead. |
| Screenshot |  |
| Resource | Biblindex Reuse Finder |
|---|---|
| Resource type | Annotator - this PR adds / modifies annotations / features in a Language Resource. |
| Dependencies | Biblindex Structure Tagger, Polytonic Greek Normaliser, Polytonic Greek Lemmatiser |
| Parameters | reused text file: an xml file containing the reused text, as processed by the Biblindex Processed Text Export PR. word order (runtime): whether the reuse retrieval algorithm will take into account the word order when searching for similar passages. processing (runtime): the level of processing at which the text must be considered (either none, normalised or lemmatised). filtering (runtime): the layers of filtering to use; simple filtering will discard any stopword in the text before even beginning, while double filtering will in addition discard any matches that rely on usual or recurring words only. n-gram size (runtime): the amount of common processed terms to provide a match between both texts. tolerance threshold (runtime): the amount of differences that are allowed in a match (think edit distance). tolerance (runtime): the way the threshold is applied (none is no threshold whatsoever, absolute is a fixed threshold, and relative is a threshold proportional to the match size -in this case, the above parameter is a percentage, e.g. 70 for 70%. |
| Description | This PR searches for hints of text reuse in the form of similar passages between the given reused text and each of the Language Resources in the corpus it is applied to. Once hint have been found, it attempts to locate a case of text reuse and creates an annotation for each discovered case. The annotation contains information about the reused passage, including which parts were found in common. The PR can be used several times with different runtime parameters, but will not take into account previously found text reuse annotations. |
| Screenshot |  |
XML DTDs
The DTDs used to build xml files by the Biblindex Parser, the Biblindex Tokeniser and the Biblindex Processed Text Export PRs are the following (the parser does not use the Word attributes norm and lem). You may format any (unicode greek) text with these DTDs and use them as Language Resources instead of using the Biblindex Parser PR.
- General format:
<!DOCTYPE DocWorks [
<!ELEMENT DocWorks (Work+)>
<!ELEMENT Work (Chapter+)>
<!ATTLIST Work name CDATA #REQUIRED>
<!ELEMENT Chapter (Paragraph+)>
<!ATTLIST Chapter num CDATA #REQUIRED>
<!ELEMENT Paragraph (Line+)>
<!ATTLIST Paragraph num CDATA #REQUIRED>
<!ELEMENT Line (Word*)>
<!ATTLIST Line num CDATA #REQUIRED>
<!ELEMENT Word (#PCDATA)>
<!ATTLIST Word num CDATA #REQUIRED>
<!ATTLIST Word norm CDATA #IMPLIED>
<!ATTLIST Word lem CDATA #IMPLIED>
]>
- Bible format:
<!DOCTYPE DocBible [
<!ELEMENT DocBible (Bible+)>
<!ELEMENT Bible (Book+)>
<!ATTLIST Bible name CDATA #REQUIRED>
<!ELEMENT Book (Chapter+)>
<!ATTLIST Book name CDATA #REQUIRED>
<!ELEMENT Chapter (Verse+)>
<!ATTLIST Chapter num CDATA #REQUIRED>
<!ELEMENT Verse (Word*)>
<!ATTLIST Verse num CDATA #REQUIRED>
<!ELEMENT Word (#PCDATA)>
<!ATTLIST Word num CDATA #REQUIRED>
<!ATTLIST Word norm CDATA #IMPLIED>
<!ATTLIST Word lem CDATA #IMPLIED>
]>
toolkit_polytonicgreek.txt · Last modified: 2014/06/18 15:09 by sgesche
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
