Evaluation strategy

What is evaluated

Annotation reliability and judgments by annotators of the correctness of each other's annotations.

What isn't evaluated

Quality of translation based on the interlingua

Language independence of the interlingua

Cost of tool and corpus development in additional languages

What the project proposes to evaluate in the future

Our current metrics, based on inter-annotator reliability, will be augmented to consider also the growth rate of the interlingual representation, the ability to handle translation divergences, and quality of the target language text that can be generated from the interlingua. IAMTC 2004, "Goals"