Statistical Alignment and Machine Translation

Lecture 8: Statistical Alignment & Machine Translation (Chapter 13 of Manning & Schutze) Wen-Hsiang Lu (盧文祥) Department of Computer Science and Information Engineering, National Cheng Kung University 2014/05/12

(Slides from Berlin Chen, National Taiwan Normal University http://140.122.185.120/PastCourses/2004SNaturalLanguageProcessing/NLP_main_2004S.htm)

References: 1. 2. 3.

Brown, P., Pietra, S. A. D., Pietra, V. D. J., Mercer, R. L. The Mathematics of Machine Translation, Computational Linguistics, 1993. Knight, K. Automating Knowledge Acquisition for Machine Translation, AI Magazine,1997. W. A. Gale and K. W. Church. A Program for Aligning Sentences in Bilingual Corpora, Computational Linguistics 1993. 1

Machine Translation (MT) • Definition – Automatic translation of text or speech from one language to another

• Goal – Produce close to error-free output that reads fluently in the target language – Far from it ?

• Current Status – Existing systems seem not good in quality • Babel Fish Translation (before 2004) • Google Translation – A mix of probabilistic and non-probabilistic components 2

Issues • Build high-quality semantic-based MT systems in circumscribed domains • Abandon automatic MT, build software to assist human translators instead – Post-edit the output of a buggy translation

• Develop automatic knowledge acquisition techniques for improving general-purpose MT – Supervised or unsupervised learning

3

Different Strategies for MT Interlingua (knowledge representation) knowledge-based translation English (semantic representation)

English (syntactic parse)

English Text (word string)

semantic transfer

syntactic transfer

word-for-word

French (semantic representation)

French (syntactic parse)

French Text (word string) 4

Word for Word MT • Translate words one-by-one from one language to another – Problems 1. No one-to-one correspondence between words in different languages (lexical ambiguity) – Need to look at the context larger than individual word (→ phrase or clause) 2. Languages have different word orders English suit

French lawsuit (訴訟), set of garments (服裝) meanings 5

Syntactic Transfer MT • Parse the source text, then transfer the parse tree of the source text into a syntactic tree in the target language, and then generate the translation from this syntactic tree – Solve the problems of word ordering – Problems • Syntactic ambiguity • The target syntax will likely mirror that of the source text N

V

Adv

German: Ich esse gern ( I like to eat ) English: I eat readily/gladly

6

Text Alignment: Definition • Definition – Align paragraphs, sentences or words in one language to paragraphs, sentences or words in another languages • Thus can learn which words tend to be translated by which other words in another language

bilingual dictionaries, MT , parallel grammars … – Is not part of MT process per se • But the obligatory first step for making use of multilingual text corpora

• Applications – – – –

Bilingual lexicography Machine translation Multilingual information retrieval … 7

Text Alignment: Sources and Granularities • Sources of parallel texts or bitexts – Parliamentary proceedings (Hansards) – Newspapers and magazines – Religious and literary works

with less literal translation

• Two levels of alignment – Gross large scale alignment • Learn which paragraphs or sentences correspond to which paragraphs or sentences in another language – Word alignment • Learn which words tend to be translated by which words in another language • The necessary step for acquiring a bilingual dictionary Orders of word or sentence might not be preserved.

8

Text Alignment: Example 1

2:2 alignment

9

Text Alignment: Example 2

2:2 alignment

1:1 alignment 1:1 alignment

2:1 alignment

a bead/a sentence alignment Studies show that around 90% of alignments are 1:1 sentence alignment. 10

Sentence Alignment • Crossing dependencies are not allowed here – Word ordering is preserved !

• Related work

11

Sentence Alignment • Length-based

• Lexical-guided • Offset-based

12

Sentence Alignment Length-based method

• Rationale: the short sentences will be translated as short sentences and long sentences as long sentences – Length is defined as the number of words or the number of characters

• Approach 1 (Gale & Church 1993)

Union Bank of Switzerland (UBS) corpus : English, French, and German

– Assumptions • The paragraph structure was clearly marked in the corpus, confusions are checked by hand s1 s2 s3 s4 . . . sI

t1 t2 t3 t4 . . . . tJ

• Lengths of sentences measured in characters •

• Crossing dependences are not handled here – The order of sentences are not changed in the translation Ignore the rich information available in the text. 13

Sentence Alignment Length-based method source

Source

B1

S  s1s2  sI

T  t1t2 t J Target

B2 B3

Bk

s1 s2 s3 s4 . . . sI

target

t1 t2 t3 t4 . . . . tJ

possible alignments: {1:1, 1:0, 0:1, 2:1,1:2, 2:2,…}

a bead

probability independence between beads

arg max PA S , T   arg max P A, S , T    PBk  K

A

A

where A  B1 , B2 ,...,Bk 

k 1

14

Sentence Alignment Length-based method

– Dynamic Programming • The cost function (Distance Measure)





cost align l1 , l2    log P  align  l1 , l2 ,  , s 2







  log P align P  l1 , l2 ,  , s 2  align

 log PBk 

 l1 , l2 ,  , s 2   l2  l1  Ratio of texts in two languages





l1s 2   is a normal distribution square difference of two paragraphs

L2  L1

• Sentence is the unit of alignment • Statistically modeling of character lengths





Bayes’ Law



P  l1, l2 , , s 2  align  21  prob



The prob. distribution of standard normal distribution

15

Sentence Alignment Length-based method

• The priori probability Or P(α align)

Source



si si-1 si-2

tj-2 tj-1

tj



Di, j  1  cost 0 : 1 align , t j   Di  1, j   cost1 : 0 align si ,     Di  1, j  1  cost 1 : 1 alignsi , t j  Di, j   min   Di  1, j  2  cost 1 : 2 alignsi , t j 1 , t j  Di  2, j  1  cost 2 : 1 alignsi 1 , si , t j    Di  2, j  2   cost 2 : 2 alignsi 1 , si , t j 1 , t j Target



 





 



16

Sentence Alignment Length-based method

– A simple example

L1 alignment 1

L1 alignment 2

s1

t1

cost(align(s1, s2, t1)) t1 +

s2

t2

t2

s3

t3

s4

cost(align(s3, t2)) + cost(align(s4, t3))

t3

cost(align(s1, t1)) + cost(align(s2, t2)) + cost(align(s3,Ø)) + cost(align(s4, t3))

17

Sentence Alignment Length-based method

– 4% error rate was achieved – Problems: • Can not handle noisy and imperfect input – E.g., OCR output or file containing unknown markup conventions – Finding paragraph or sentence boundaries is difficult – Solution: just align text (position) offsets in two parallel texts (Church 1993) • Questionable for languages with few cognates (同源) or different writing systems – E.g., English ←→ Chinese

eastern European languages ←→ Asian languages

18

Sentence Alignment Lexical method

• Approach 1 (Kay and Röscheisen 1993) – First assume the first and last sentences of the text were align as the initial anchors – Form an envelope of possible alignments • Alignments excluded when sentences across anchors or their respective distance from an anchor differ greatly – Choose word pairs their distributions are similar in most of the sentences – Find pairs of source and target sentences which contain many possible lexical correspondences • The most reliable of pairs are used to induce a set of partial alignment (add to the list of anchors) Iterations 19

Word Alignment • The sentence/offset alignment can be extended to a word alignment • Some criteria are then used to select aligned word pairs to include them into the bilingual dictionary – Frequency of word correspondences – Association measures – ….

20

Statistical Machine Translation • The noisy channel model Language Model Pe 

e

f

Translation Model P f e

Decoder eˆ  arg maxe Pe f 

eˆ

e: English f: French

eak fk ea j

|e|=l

fj

|f|=m

– Assumptions: • An English word can be aligned with multiple French words while each French word is aligned with at most one English word • Independence of the individual word-to-word translations 21

Statistical Machine Translation • Three important components involved – Decoder eˆ  arg max Pe f   arg max e

pe  p  f e 

e

 arg max pe  p f e 

p f 

e

– Language model • Give the probability p(e) – Translation model



m l 1 l P f e   ...   P f Z a1  0 am  0 j 1

all possible normalization alignments constant (the English word that a French word fj is aligned with)

j

ea j



translation probability

22

An Introduction to Statistical Machine Translation Dept. of CSIE, NCKU Yao-Sheng Chang Date: 2011.04.12

23

Introduction • Translations involve many cultural respects – We only consider the translation of individual sentence, just acceptable sentences.

• Every sentence in one language is a possible translation of any sentence in the other – Assign (S,T) a probability, Pr(T|S), to be the probability that a translator will produce T in the target language when presented with S in the source language.

24

Statistical Machine Translation (SMT)

• Noise channel problem

25

Fundamental of SMT • Given a string of French f, the job of our translation system is to find the string e that the speaker had in mind when he produced f. (Baye’s theorem)

• Since denominator Pr(f) here is a constant, the best e is one which has the greatest probability.

26

Practical Challenges • Computation of translation model Pr(f|e) • Computation of language model Pr(e) • Decoding (i.e., search for e that maximize Pr(f|e)  Pr(e))

27

Alignment of case 1 (one-to-many)

28

Alignment of case 2 (many-to-one)

29

Alignment of case 3 (many-to-many)

30

Formulation of Alignment(1) • Let e = e1le1e2…el and f = f1m  f1f2…fm • An alignment between a pair of strings e and f use a mapping of every word ei to some word fj e = e1e2…ei…el

aj =i

f = f1 f2… fj… fm

fj = eaj = ei

• In other words, an alignment a between e and f tells that the word ei, 1 i  l is generated by the word fj, aj{1,…,l} • There are (l+1)m different alignments between e and f. (Including Null – no mapping ) 31

Translation Model

• The alignment, a, can be represented by a series, a1m = ala2... am, of m values, each between 0 and l such that if the word in position j of the French string is connected to the word in position i of the English string, then aj = i , and if it is not connected to any English word, then aj = 0 (null).

32

IBM Model I (1)



33

IBM Model I (2) • The alignment is determined by specifying the values of aj for j from 1 to m, each of which can take any value from 0 to l.

34

Constrained Maximization • We wish to adjust the translation probabilities so as to maximize Pr(f|e ) subject to the constraints that for each e

35

Lagrange Multipliers (1) • Method of Lagrange multipliers（拉格朗乘數法）: Lagrange multipliers with one constraint – If there is a maximum or minimum subject to the constraint g(x,y) = 0, then it will occur at one of the critical numbers of the function F defined by is called the

F ( x, y,  )  f ( x, y)  g ( x, y) – – – –

f(x,y) is called the objective function（目標函數）. g(x,y) is called the constrained equation（條件限制方程式）. F(x, y, ) is called the Lagrange function（拉格朗函數）.  is called the Lagrange multiplier（拉格朗乘數）.

36

Lagrange Multipliers (2) • Following standard practice for constrained maximization, we introduce Lagrange multipliers e, and seek an unconstrained extremum of the auxiliary function

37

Derivation (1) • The partial derivative of h with respect to t(f|e) is

• where  is the Kronecker delta function, equal to one when both of its arguments are the same and equal to zero otherwise

38

Derivation (2)

• We call the expected number of times that e connects to f in the translation (f|e) the count of f given e for (f|e) and denote it by c(f|e; f, e). By definition,

39

Derivation (3) • replacing e by ePr(f|e), then Equation (11) can be written very compactly as

• In practice, our training data consists of a set of translations, (f(1) le(1)), (f(2) le(2)), ..., (f(s) le(s)), , so this equation becomes

40

Derivation (4) • For an expression that can be evaluated efficiently.

41

Derivation (5) • Thus, the number of operations necessary to calculate a count is proportional to l + m rather than to (l + 1)m as Equation (12)

42

Other References about MT

44

Chinese-English Sentence Alignment • 林語君、高照明,結合統計與語言 訊息的混合式中英雙語句對應演算 法,ROCLING XVI,2004 • 問題:中文句子的界線並不清楚， 句點與逗點都有可能是句子的界線。 • 方法 – 群組句對應模組 • 2 Stages Iterative DP

– 句對應參數 • 採用廣泛的字典翻譯詞以及Stop list • 中文翻譯詞的部分比對（Partial Match） • 句中重要標點符號序列相似度 • 共同數字詞、時間詞、原文詞之對 應錨

– 群組句對應參數 • 句長相異度 • 句末標點符號要求相同

45

Bilingual Collocation Extraction Based on Syntactic and Statistical Analyses (Chien-Cheng Wu & Jason S. Chang, ROCLING’03)

46

Bilingual Collocation Extraction Based on Syntactic and Statistical Analyses (Chien-Cheng Wu & Jason S. Chang, ROCLING’03)

• Preprocessing steps to calculate the following information: – – – –

Lists of preferred POS patterns of collocation in both languages. Collocation candidates matching the preferred POS patterns. N-gram statistics for both languages, N = 1, 2. Log likelihood Ratio statistics for two consecutive words in both languages. – Log likelihood Ratio statistics for a pair of candidates of bilingual collocation across from one language to the other. – Content word alignment based on Competitive Linking Algorithm (Melamed 1997).

47

Bilingual Collocation Extraction Based on Syntactic and Statistical Analyses (Chien-Cheng Wu & Jason S. Chang, ROCLING’03)

48

Bilingual Collocation Extraction Based on Syntactic and Statistical Analyses (Chien-Cheng Wu & Jason S. Chang, ROCLING’03)

49

A Probability Model to Improve Word Alignment Colin Cherry & Dekang Lin, University of Alberta

Outline • Introduction • Probabilistic Word-Alignment Model • Word-Alignment Algorithm – Constraints – Features

Introduction • Word-aligned corpora are an excellent source of translation-related knowledge in statistical machine translation. – E.g., translation lexicons, transfer rules

• Word-alignment problem – Conventional approaches usually used co-occurrence models • E.g., Ø2 (Gale & Church 1991), log-likelihood ratio (Dunning 1993)

– Indirect association problem: Melamed (2000) proposed competitive linking along with an explicit noise model to solve

B(links(u, v) | cooc(u, v), ) scoreB(u, v)  log B(links(u, v) | cooc(u, v), )

CISCO System Inc.

思科 系統

• To propose a probabilistic word-alignment model which allows easy integration of context-specific features.

noise

Probabilistic Word-Alignment Model • Given E = e1, e2, …, em , F = f1, f2, …, fn – If ei and fj are translation pair, then link l(ei, fj) exists – If ei has no corresponding translation, then null link l(ei, f0) exists – If fj has no corresponding translation, then null link l(e0, fj) exists – An alignment A is a set of links such that every word in E and F participates in at least one link

• Alignment problem is to find alignment A to maximize P(A|E, F) • IBM’s translation model: maximize P(A, F|E)

Probabilistic Word-Alignment Model (Cont.) • Given A = {l1, l2, …, lt}, where lk = l(eik, fjk), then j consecutive subsets of A, li = {li, li+1, …, lj} P( A | E, F ) 

P(l1t

t

| E, F )   P(lk | E, F , l1k 1 ) k 1

k-1

• Let Ck= {E, F, l1 } represent the context of lk P(lk | Ck ) 

P(lk , Ck ) P(Ck | lk ) P(lk )  P(Ck ) P(Ck , eik , f jk )

P(lk , eik , f jk ) P(Ck | lk )   P(Ck | eik , f jk ) P(eik , f jk ) P(Ck | lk )  P(lk | eik , f jk )  P(Ck | eik , f jk )

 P(Ck , eik , f jk )  P(Ck )  P(eik , f jk | Ck )   P(eik , f jk | Ck )  1

 P(lk , eik , f jk )  P(lk )  P(eik , f jk | lk )   P(eik , f jk | lk )  1

Probabilistic Word-Alignment Model (Cont.) k-1

• Ck = {E, F, l1 } is too complex to estimate • FTk is a set of context-related features such that P(lk|Ck) can be approximated by P(lk|eik, fjk, FTk) • Let Ck’ = {eik, fjk} ∪ FTk P(lk | Ck' ) 

P(lk | eik , f jk ) 

 P(lk | eik , f jk ) 

P(Ck' | lk ) P(Ck' | eik , f jk ) P( FTk | lk ) P( FTk | eik , f jk )

 P( ft | lk )   P( A | E , F )   P(lk | eik , f jk )     k 1 ftFTk P ( ft | eik , f jk )  t

An Illustrative Example

Word-Alignment Algorithm • Input: E, F, TE – TE is E’s dependency tree which enable us to make use of features and constraints based on linguistic intuitions

• Constraints – One-to-one constraint: every word participates in exactly one link – Cohesion constraint: use TE to induce TF with no crossing dependencies

Word-Alignment Algorithm (Cont.) • Features – Adjacency features fta: for any word pair (ei, fj), if a link l(ei’, fj’) exists where -2  i’-i  2 and -2  j’-j  2, then fta(i-i’, j-j’, ei’) is active for this context.

– Dependency features ftd: for any word pair (ei, fj), let ei’ be the governor of ei ,and let rel be the grammatical relationship between them. If a link l(ei’, fj’) exists, then ftd(j-j’, rel) is active for this context.

 fta ( 1,1, host)  pair (the1 , l ' )    ftd ( 1, det)  pair (the1 , les)  ftd ( 3, det)

Experimental Results • Test bed: Hansard corpus – Training: 50K aligned pairs of sentences (Och & Ney 2000) – Testing: 500 pairs

Future Work • The alignment algorithm presented here is incapable of creating alignments that are not one-to-one, many-to-one alignment will be pursued. • The proposed model is capable of creating many-to-one alignments as the null probabilities of the words added on the “many” side.