Slides - CVIT - IIIT Hyderabad

Content Level Access to Digital Library of India Pages Praveen Krishnan, Ravi Shekhar, C.V. Jawahar CVIT, IIIT Hyderabad

IIIT Hyderabad

Digital Library of India (DLI)

http://www.dli.iiit.ac.in/

Vision : To enhance access to information and knowledge to masses.

• Partner to Million Book Universal Digital Library Programme. Information for people

Dataset for researchers

IIIT Hyderabad

Vamshi Ambati, N.Balakrishnan, Raj Reddy, Lakshmi Pratha, C V Jawahar: The Digital Library of India Project: Process, Policies and Architecture, ICDL , 2007.

Digital Library of India (DLI)

Vision : To enhance access to information and knowledge to masses. Content

Languages

Statistics

IIIT Hyderabad

• #Books 4 Lakhs • 41 different languages • #Pages 134 Million • Includes - Hindi, Telugu, Marathi.. • #Words 26 Billion - English, French, Greek.. Source: http://www.new1.dli.ernet.in/

Digital Library of India (DLI)

Meta data search • Supports Meta data based search. • No Content Level Access Indian freedom struggle and independence Search IIIT Hyderabad

Digital Library of India (DLI)

• Need Content Level Access • Content + Meta Data Indian freedom struggle and independence Search IIIT Hyderabad

Digital Library of India (DLI)

• Need Content Level Access • Content + Meta Data Indian freedom struggle and independence Search

Reliable Text Representation

?

IIIT Hyderabad

Goal Digital Library of India Search

• Build a search engine with support for Indian languages. • Word Spotting

IIIT Hyderabad

Goal Indian Language Document Search Engine

Text Query Support

खोज

Page 1 IIIT Hyderabad

Goal Indian Language Document Search Engine शिवाजी और मराठा साम्राज्य

खोज

Multi Keyword Support Page 1 IIIT Hyderabad

Goal Indian Language Document Search Engine शिवाजी और मराठा साम्राज्य

खोज

Ranks based on # Occurrences Page 1 IIIT Hyderabad

Goal Indian Language Document Search Engine शिवाजी और मराठा साम्राज्य

खोज

Semantically Related Words Page 1 IIIT Hyderabad

Goal Indian Language Document Search Engine शिवाजी और मराठा साम्राज्य

खोज

Seamless scaling to billions of word images.

Sub second retrieval Page 1 IIIT Hyderabad

Text from OCR

Hindi Page

Telugu Page

IIIT Hyderabad

- Hindi: Title - Praachin Bhaartiy Vichaar Aur Vibhutiyaan, Published in 1624 - Telugu: Title - Andhra Vagmayaramba Dasha, Published in 1960

Text from OCR

Hindi Page IIIT Hyderabad

Cuts

Telugu Page

Text from OCR

Hindi Page IIIT Hyderabad

Merges Cuts

Telugu Page

Text from OCR

Hindi Page

Telugu Page

IIIT Hyderabad

Variations in Script,Cuts Font and Typesetting.

Text from OCR

Char % Hindi

Telugu

IIIT Hyderabad

[1 ] D. Arya, T. Patnaik, S. Chaudhury, C. V. Jawahar, B. B. Chaudhury, A. G. Ramakrishnan, G. S. Lehal, and C. Bhagavati, “Experiences of Integration and Performance Testing of Multilingual OCR for Printed Indian Scripts,” in ICDAR MOCR Workshop, 2011.

Text from OCR

Word % Hindi

Telugu

IIIT Hyderabad

[1 ] D. Arya, T. Patnaik, S. Chaudhury, C. V. Jawahar, B. B. Chaudhury, A. G. Ramakrishnan, G. S. Lehal, and C. Bhagavati, “Experiences of Integration and Performance Testing of Multilingual OCR for Printed Indian Scripts,” in ICDAR MOCR Workshop, 2011.

Text from OCR

Search % Hindi

Telugu

IIIT Hyderabad

BoVW for Image Retrieval Text Retrieval

Image Recognition Query Image

Ranked Retrieved Results IIIT Hyderabad

Josef Sivic, Andrew Zisserman: Video Google: A Text Retrieval Approach to Object Matching in Videos. ICCV 2003

BoVW for Image Retrieval • Fixed Length Representation • Invariant to popular deformation Query Image

Ranked Retrieved Results IIIT Hyderabad

Josef Sivic, Andrew Zisserman: Video Google: A Text Retrieval Approach to Object Matching in Videos. ICCV 2003

BoVW for Document Image Retrieval

IIIT Hyderabad

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012.

BoVW for Document Image Retrieval

Histogram of Visual Words

IIIT Hyderabad

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012.

BoVW for Document Image Retrieval Cuts

IIIT Hyderabad

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012.

BoVW for Document Image Retrieval Cuts

Histogram of Visual Words

IIIT Hyderabad

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012.

BoVW for Document Image Retrieval Merges

IIIT Hyderabad

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012.

BoVW for Document Image Retrieval Merges

Histogram of Visual Words

IIIT Hyderabad

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012.

BoVW for Document Image Retrieval • Robust against degradation • Lost Geometry • Use Spatial Verification – SIFT based. – Longest Subsequence alignment. y 1 0.5

Clean

0

0.5

IIIT Hyderabad

V 1

1 V 2

V 6

1.5 Cuts 2 V 4

V 4

2.5 V 8

3 V 9

x

Merge

R. Shekhar and C. V. Jawahar. Word Image Retrieval Using Bag of Visual Words. In DAS, 2012. I. Z. Yalniz and R. Manmatha. An Efficient Framework for Searching Text in Noisy Document Images. In DAS, 2012

Query Expansion Querying Database

Query Image Rank 1

Rank 2

Histogram Rank 3

Rank 4

Rank 5

Rank 6

IIIT Hyderabad

Refined Histogram

Query Expansion Querying Database

Query Image Rank 1

Rank 2

Query Histogram Rank 3

Rank 4

Better Results

Rank 5

Rank 6

IIIT Hyderabad

Text Query Support • Originally formulated in a “query by example” setting.

Input Query Image

Histogram

IIIT Hyderabad

Text Query Support • Originally formulated in a “query by example” setting. • Need Text Queries

Input Text Query

IIIT Hyderabad

Text Query Histogram

Observations • Are the results of OCR and BoVW complementary?

IIIT Hyderabad

BoVW

OCR

OCR

BoVW

Observations

mAP

• mAP v/s Word Length

IIIT Hyderabad

No. of Characters

Observations • “OCR system has a high precision while BoVW approach has a high recall.” • Example: #GT = 5 OCR Out List; Precision = 1 ; Recall = 0.4

BoVW Out List; Precision = 0.8 ; Recall = 1 IIIT Hyderabad

Fusion • Fusion Techniques:• Naïve Fusion

mAP Chart

OCR

IIIT Hyderabad

Fusion • Fusion Techniques:• Naïve Fusion

mAP Chart

BoVW

IIIT Hyderabad

Fusion • Fusion Techniques:• Naïve Fusion Concatenating OCR Results with BoVW OCR

BoVW

mAP Chart

IIIT Hyderabad

Fusion • Fusion Techniques:• Edit Distance Based Fusion

OCR

BoVW

mAP Chart

IIIT Hyderabad

Fusion • Fusion Techniques:• Edit Distance Based Fusion

• Reordering BoVW • BoVW score • Modified Edit distance cost BoVW

mAP Chart

IIIT Hyderabad

Fusion • Fusion Techniques:• Edit Distance Based Fusion

• Reordering BoVW • BoVW score • Modified Edit distance cost BoVW

mAP Chart

IIIT Hyderabad

Fusion • Fusion Techniques:• Edit Distance Based Fusion

OCR

BoVW

mAP Chart

IIIT Hyderabad

Fusion • Fusion Techniques:• Hybrid Fusion

OCR

BoVW

mAP Chart

IIIT Hyderabad

Fusion • Fusion Techniques:• Hybrid Fusion

mAP Chart

• Re-querying BoVW using • OCR retrieved results. • Using rank aggregation techniques BoVW IIIT Hyderabad

Fusion • Fusion Techniques:• Hybrid Fusion

mAP Chart

• Re-querying BoVW using • OCR retrieved results. • Using rank aggregation techniques BoVW IIIT Hyderabad

Fusion • Fusion Techniques:• Hybrid Fusion

OCR

BoVW

mAP Chart

IIIT Hyderabad

Experimental Results

IIIT Hyderabad

Experimental Details • OCR [1] • Feature Detector – Harris Interest point detection. [2]

• Feature Descriptor – SIFT [2]

• Indexing – Lucene [3] IIIT Hyderabad

[1 ] D. Arya, T. Patnaik, S. Chaudhury, C. V. Jawahar, B. B. Chaudhury, A. G. Ramakrishnan, G. S. Lehal, and C. Bhagavati, “Experiences of Integration and Performance Testing of Multilingual OCR for Printed Indian Scripts,”in ICDAR MOCR Workshop, 2011. [2] http://www.vlfeat.org [3] http://lucene.apache.org/

Test Bed

Sample Word Images Language

#Books

#Pages

#Words

#Annotation

Hindi (HS1)

11

1000

362,593

Yes

Hindi (HS2)

52

10,196

4,290,864

No

Telugu (TS1)

11

1000

161,276

Yes

Telugu (TS2)

69

13,871

2,531,069

No

DLI Corpus IIIT Hyderabad

• In addition, we used HP1 & TP1 fully annotated dataset

Evaluation Measures • Precision

• Recall

•

TP = True Positive FP = False Positive FN = False Negative

mAP (Mean Average Precision) Mean of the area under the precision recall curve for all the queries.

• Precision @ 10 Shows how accurate top 10 retrieved IIIT Hyderabad

results are. Precision-Recall Curve

BoVW Search Language

#Query

BoVW + Query Expansion

mAP

Prec@10

mAP

Prec@10

Hindi (HP1)

100

62.54

81.30

66.09

83.86

Telugu (TP1)

100

71.13

78

73.08

79.89

Comparison of naïve BoVW with BoVW + Query Expansion IIIT Hyderabad

BoVW Search Language

#Query

BoVW using Text Queries

mAP

Prec@10

mAP

Prec@10

Hindi (HP1)

100

62.54

81.30

56.32

73.89

Telugu (TP1)

100

71.13

78

69.06

78.83

Comparison of naïve BoVW with BoVW + Text Query Support IIIT Hyderabad

Naïve Language #Query

Edit Distance

Hybrid

mAP

Prec@10

mAP

Prec@10

mAP

Prec@10

Hindi (HP1)

100

75.66

90.7

79.58

90.8

80.37

91.4

Telugu (TP1)

100

76.02

81.2

78.01

81.4

80.23

83.7

Comparative performance of different fusion techniques on HP1 & TP1 IIIT Hyderabad

OCR Language #Query

BoVW

Fusion

mAP

Prec@10

mAP

Prec@10

mAP

Prec@10

Hindi (HS1)

100

14.95

62.60

60.55

95.5

68.81

95.6

Telugu (TS1)

100

27.03

62.10

74.38

90.6

78.41

91.9

Performance statistics on DLI Annotated Corpus IIIT Hyderabad

Language Hindi (HS2)

Telugu (TS2)

#Query

50

50

Precision @ N

OCR

BoVW

Fusion

Prec@10

82.03

96.94

97.11

Prec@20

75.16

94.83

95.42

Prec@30

71.12

92.82

93.16

Prec@10

90.85

99.14

99.14

Prec@20

85.42

98.00

98.85

Prec@30

80.76

96.38

96.57

Performance statistics on DLI Un-Annotated Corpus IIIT Hyderabad

Retrieved Results

IIIT Hyderabad

Retrieved Results

IIIT Hyderabad

Failure Cases

• The word images shown in the figure fails in both OCR and BoVW. • Reason: – (a) Word Image smaller in length and containing a character not used these days. IIIT Hyderabad

– (b) A highly degraded word image.

Implementation Details • Search Engine Development – An elegant web based search and retrieval interface.

No of Images

Time in milliseconds

Lucene Scalability

IIIT Hyderabad

Sample Retrieved Page No of Visual Words

Search Architecture (Ongoing) Search Query

Ranked Results

Delegator

Partial Scores FUSION

Query Expansion

Ranking OCR

BoVW

IIIT Hyderabad

OCR Index Web Service

BoVW Index Web Service

Web Service

Ongoing Work • Learn to improve from annotated dataset – Use of visual confusion matrix to improve BoVW results from annotated datasets.

• Necessity of Costly Features for Re-ranking – The images shows in failure cases would require costly features to show up. – Use of machine learning algorithms. IIIT Hyderabad

• Exploration of features better than SIFT.

Thank You

IIIT Hyderabad