Federal Digitization Moving to Common Guidelines The U.S. Federal Agencies Digitization Guidelines Initiative (FADGI)
http://www.digitizationguidelines.gov/ PASIG, May 24, 2013 Carl Fleischhauer
[email protected]
Steve Puglia
[email protected]
Library of Congress Washington, DC
http://www.digitizationguidelines.gov/ 2
18 Participating Agencies http://www.digitizationguidelines.gov/participants/
Often participating, not “official”: NASA, NOAA, National Museum of Health and Medicine (U.S. Army), U.S. Supreme Court
32
http://www.digitizationguidelines.gov/stillimages/
http://www.digitizationguidelines.gov/audio-visual/
Guidelines • Conceptual framework documents – Content Categories & Digitization Objectives (still image reproduction; September 3, 2009) – Digitization Activities – Project Planning (November 4, 2009)
• Capture device performance – Digital Imaging Framework (high level about scanner performance metrics; April 2, 2009) – Audio Analog-to-Digital Converter Performance (August 20, 2012) – Audio Interstitial Errors (about unwanted dropouts or sample distortion; work in progress, 2012-13)
• Broad practices guidelines – Technical Guidelines for the Still Image Digitization of Cultural Heritage Materials (Many segments from 2004 NARA document; FADGI update, August 24, 2010) 6
Guidelines • Metadata including embedded data and file headers – TIFF Image Header Metadata (February 10, 2009) – Minimal Descriptive Embedded Metadata in Digital Still Images (Smithsonian document embraced by group; March 23, 2012) – Embedding Metadata in Broadcast WAVE Files, Version 2 (April 23, 2012) • Associated tool on SourceForge: BWF MetaEdit
– NARA reVTMD video technical metadata (February 2012; FADGI supporting role) • Associated tool on GitHub: AVI MetaEdit
• Format analysis and guidelines – File Format Comparisons (comparing still image and video formats; under development in 2013) – MXF Preservation Video Formatting Application Specification (under development during 2013 in cooperation with AMWA trade group; versions posted in 2010 and 2012) 7
Still Image Illustrative Example Odds and ends about still images
Still image specifications – this is what we all “used to do” • color/monochromatic • pixel density (good old “dpi”) • bit depth • . . . usually output-referred We want to move toward more, um, “scientific” specifications Tone Gamma
Resolution
Color
Spatial Frequency Response (SFR)
Luminance
Delta E2000
Resolution
Delta E(a*b*)2000
Sampling Efficiency
Channel Mis-registration
Sampling Frequency
White Balance
Uniformity % Lighting Non-uniformity
Noise Total rms deviation
From this document: http://www.digitizationguidelines.gov/guidelines/DIFfinal.pdf
Resolution rethink: new terms, scanner performance
• SAMPLING RATE • SPATIAL RESOLUTION – Spatial Frequency Response (SFR)
• SAMPLING EFFICIENCY Thanks to Barry Wheeler for his very helpful Signal blogs: http://blogs.loc.gov/digitalpreservation/2012/12/what-resolution-should-i-use-part-1/ http://blogs.loc.gov/digitalpreservation/2013/01/what-resolution-should-i-use-part-2/ http://blogs.loc.gov/digitalpreservation/2013/03/what-resolution-should-i-use-part-3/
Resolution rethink: new terms, scanner performance • SAMPLING RATE. Usually, the scanner’s ppi number is sampling rate – Sensors can only attempt to measure (sample) the brightness at each point. – Some light may scatter and miss the sensor, the scanner’s motor step may not be sufficiently precise, or the collected value may be inaccurate. Inside every scanner or camera, between the sensor and the screen is a small, highly specialized computer called a digital signal processor. This processor must work very hard to link a dot on the page to a dot on the screen.
• RESOLUTION. ISO standards (e.g., ISO 12233) define resolution in terms of Spatial Frequency Response (SFR) -- the actual result on the screen. • SAMPLING EFFICIENCY. . . . the difference between the pixel count and actually resolving each point, expressed as percentage.
From the revised guideline
http://www.digitizationguidelines.gov/guidelines/FADGI_Still_Image-Tech_Guidelines_2010-08-24.pdf
Tools to Support Image Performance Measurement • Digital Image Conformance Evaluation (DICE) System – Device Target – Imaging Device Performance – Object Target – Actual Image Quality – Software for Evaluation/Validation • Based in LabVIEW • Data export for use in SQC/SPC
Device and Object Targets
Object target as positioned for use
DICE Software – Main Panel
DICE – QC Summary Panel
Slide from old version of software
DICE – OECF detail page
DICE – SFR detail page
Audio-Visual Illustrative Example MXF format specification for reformatted video
Library of Congress Packard Campus, Culpeper
National Archives, College Park
Smithsonian Institution Archives
SAMMA from Front Porch Digital
Implementations • SAMMA at LC: Lossless compressed – Each frame is a JPEG 2000 image – Lossless (reversible) transform
• Emergent variants – NARA and other archives prefer uncompressed video – Other devices come on the market, e.g., from OpenCube (Belgium), Amberfin (UK), Cube-Tec (Germany), and others in process (e.g., Archimedia)
Standards-based format elements from SMPTE and ISO/IEC • MXF (SMPTE ST 377 and many more) • Standard definition uncompressed covered in ST 377 and also SMPTE ST 384 • JPEG 2000 encoding (ISO/IEC 15444-1) • JPEG 2000 mapped to MXF (SMPTE ST 422) • Other standards also play a role, most from SMPTE, some from EBU
Loose Ends • MXF, JPEG 2000, and even “uncompressed” video are complex standards • Entities that “conform” to the standards can be formatted in various ways – We have some elements that we want to include in order to produce an “authentic copy” – MXF “carriage” can be tricky to sort out
MXF Application Specification • An MXF AS is what some would call a profile
• Pin down preferred options, reduce the variables • Support greater interoperability • Increase the comfort level for users • Increase vendor competition • More adoption means better sustainability
Timecode • Source recordings may have multiple timecodes (VITC, LTC, etc.), some on purpose, some by accident, all may provide forensic help for future researchers. • Specify preferred practice for retaining and tagging multiple timecodes in the file
Audio tracks • Source may have multiple tracks • MXF audio track specifications cover “listing” or “allocation” (tagging) and other matters of terminology, need to pin these down
Metadata • Basic tech metadata is not an issue • Needed: specified options for embedding additional technical metadata: – – – –
process (like METS digiprov), about the source item about quality review outcomes preservation (like PREMIS),
• And some descriptive metadata – Schools of thought: some prefer minimal data (“just and identifier”), others would dump everything they have, specification should permit range of actions – “archivists choice”
Closed captioning, subtitles, ancillary data • US broadcast standards embed CC as binary data – “In the image raster” on line 21 – For digital TV, CC also in packets in MPEG stream – Awkward for future extraction, depends upon availability of decoding tools
• Desiderata – Put CC/subtitles in the file for easier access and extraction – XML rather than binary – Alas, MXF offers “too many” options for this, we seek to pin down the best ones
• By extension, this also applies to other ancillary data.
An MXF Application Specification is . . . • A formal industry statement – Not a “standard”
• Accompanied by a reference implementation and validation tools
MXF Application Specifications come from . . . • Advanced Media Workflow Association (AMWA) – Broadcast-industry group – AMWA Application Specifications include: • AS-10 for production – version for end-to-end digital production workflow (forthcoming) • AS-11 for contribution – the high end version contributed by a producer to a television network (published) • AS-03 for delivery – the reduced-data version “sent to the tower for broadcast” (published)
– AS-07 for archiving and preservation will be a sibling to those – http://www.amwa.tv/projects/AMWA_AS_overview 04-2013 web.pdf
Role of AMWA • Key roles played by Turner Broadcasting veterans and engineering staff • Members include AVID, BBC, Front Porch Digital (SAMMA), NARA, PBS, SONY, Discovery Communications, Fox, NBC Universal, and more • http://www.amwa.tv/ • Break into technical committees to push draft specifications
FADGI’s AMWA status • March 2012 – AMWA business committee approval to move ahead – Designate as AS-07
• September 2012 – Technical committee approval
• November 2012 – Team meetings began
• Early 2013 – Churning along
• End of 2013 – Dream of a first draft or better
http://www.digitizationguidelines.gov/
Carl Fleischhauer
[email protected]
