Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability

Title: Better Hypothesis Testing for Statistical Machine Translation:
Controlling for Optimizer Instability
Speaker: Jonathan Clark
When: Tuesday, 4/19 at Noon

Abstract:
In statistical machine translation, a researcher seeks to determine
whether some innovation (e.g., a new feature, model, or inference
algorithm) improves translation quality in comparison to a baseline
system. To answer this question, he runs an experiment to evaluate the
behavior of the two systems on held-out data. In this paper, we
consider how to make such experiments more statistically reliable. We
provide a systematic analysis of the effects of optimizer instability
(an extraneous variable that is seldom controlled for) on experimental
outcomes, and make recommendations for reporting results more
accurately.

This is joint work with Chris Dyer, Alon Lavie, and Noah Smith. It was
recently accepted for publication as an ACL short paper.

Qin Gao: Expanding parallel corpora for machine translation

Speaker: Qin Gao
When: at noon, March 8, 2011
Where: GHC 4405

We present an approach of expanding parallel corpora for machine translation. By utilizing Semantic role labeling (SRL) on one side of the language pair, we extract SRL substitution rules from existing parallel corpus. The rules are then used for generating new sentence pairs. An SVM classifier is built to filter the generated sentence pairs. The filtered corpus is used for training phrase-based translation models, which can be used directly in translation tasks or combined with baseline models. Experiment results on Chinese-English machine translation tasks show an average improvement of 0.45 BLEU and 1.22 TER points across 5 different NIST test sets.

Machine Translation and Computer-Assisted Translation

Title: Prospects for Integrating Machine Translation and Computer-Assisted Translation in the Translation Industry

Speaker: Gregory M. Shreve from the Department of Modern and Classical Language Studies at Kent State University and colleagues
Location: GHC 6115
Time: 12:30 pm, 14 Jan 2011


The speaker's CV can be found at http://www.kent.edu/mcls/faculty/mcls_shreve.cfm.

Efficient Language Model Inference - Kenneth Heafield

Title: Efficient Language Model Inference
Who? Kenneth Heafield
When? Tuesday, December 21 @ Noon
Where? GHC 4405


In GHC 4405 at noon on Tuesday Dec 21, I will give a speaking
requirement talk on Efficient Language Model Inference. As this is also
a MT Lunch, there will be free lunch.

If you're using SRILM, come to my talk to reduce your memory consumption
by 86% while reducing CPU time by 16%. Users of IRSTLM should come for
the same reason; the code uses 42% less memory and 19% less CPU.

Language models are an important feature in speech, translation,
generation, IR, and other technologies. More training data and less
pruning generally lead to higher quality, but RAM is a limiting factor.
Further, systems consult language models so frequently that lookups
dominate CPU time.

This talk presents language modeling code with several optimizations to
improve time and space performance. Storing backoff information in
feature state reduces redundant lookups. Constructing known
distributions and biasing binary search speeds search and reduces page
faults. Memory mapping reduces load time. Bit level packing increases
locality. Stronger filtering removes n-grams that cannot be assembled
during decoding due to phrase and sentence constraints. The code is
currently integrated into Moses and being integrated into cdec and
Joshua. I will cover how my code works and how to use it in other
decoders.

Choosing the Right Evaluation for Machine Translation

Time: Noon on Tuesday, October 12
Place: GHC 6501 (usual location)

Title: Choosing the Right Evaluation for Machine Translation: an Examination of Annotator and Automatic Metric Performance on Human Judgment Tasks
Authors: Michael Denkowski and Alon Lavie

Abstract:
This work examines the motivation, design, and practical results of several types of human evaluation tasks for machine translation. In addition to considering annotator performance and task informativeness over multiple evaluations, we explore the practicality of tuning automatic evaluation metrics to each judgment type in a comprehensive experiment using the METEOR metric. We present results showing clear advantages of tuning to certain types of judgments and discuss causes of inconsistency when tuning to various judgment data, as well as sources of difficulty in the human evaluation tasks themselves.

This work will be presented at AMTA 2010.

Models for Synchronous Grammar Induction for Statistical Machine Translation

Title: Models for Synchronous Grammar Induction for Statistical Machine Translation

Presenters: Chris Dyer, LTI& Desai Chen, CSD undergraduate

Tuesday, September 14, at Noon to 1:30pm in GHC 6501.

Abstract: The last decade of research in Statistical Machine Translation (SMT) has seen rapid progress. The most successful methods have been based on synchronous context free grammars (SCFGs), which encode translational equivalences and license reordering between tokens in the source and target languages. Yet, while closely related language pairs can be translated with a high degree of precision now, the result for distant pairs is far from acceptable. In theory, however, the "right"' SCFG is capable of handling most, if not all, structurally divergent language pairs. This talk will report on the results of the 2010 Language Engineering Workshop held at Johns Hopkins University that the goal to focus on the crucial practical aspects of acquiring such SCFGs from bilingual, but otherwise unannotated, text. We started with existing algorithms for inducing unlabeled SCFGs (e.g. the popular Hiero model) and then used unsupervised learning methods to refine the syntactic constituents used in the translation rules of the grammar.

Syntax-to-Morphology Mapping in Factored Phrase-Based Statistical Machine Translation from English to Turkish

Tuesday, June 29 at Noon, in GHC 6501

Title: Syntax-to-Morphology Mapping in Factored Phrase-Based Statistical
Machine Translation from English to Turkish

Authors: Reyyan Yeniterzi and Kemal Oflazer


Abstract:

We present a novel scheme to apply factored phrase-based SMT to a
language pair with very disparate morphological structures. Our
approach relies on syntactic analysis on the source side (English) and
then encodes a wide variety of local and non-local syntactic
structures as complex structural tags which appear as additional
factors in the training data. On the target side (Turkish), we only
perform morphological analysis and disambiguation but treat the
complete complex morphological tag as a factor, instead of separating
morphemes. We incrementally explore capturing various syntactic
substructures as complex tags on the English side, and evaluate how
our translations improve in BLEU scores. Our maximal set of source and
target side transformations, coupled with some additional techniques,
provide an 39\% relative improvement from a baseline 17.08 to 23.78
BLEU, all averaged over 10 training and test sets. Now that the
syntactic analysis on the English side is available, we also
experiment with more long distance constituent reordering to bring the
English constituent order close to Turkish, but find that these
transformations do not provide any additional consistent tangible
gains when averaged over the 10 sets.