The Avalon Media System Yale University November 24, 2014
Jon Dunn Indiana University
Mark Notess Indiana University
Julie Rudder Northwestern University
Agenda Introduction & Overview – Jon Dunn, Indiana Demo – Julie Rudder, Northwestern Technical architecture, development process, release plans – Mark Notess, Indiana NU implementation and services - Julie IU implementations and services – Mark Future directions – Jon Q & A - All
Avalon Project Objectives Goal: Create an open source system to enable libraries and archives to provide online access to video and audio collections • Digital audio/video management and delivery system, focused on needs of libraries and archives • Follow an agile, open source development model • Leverage existing technologies, where feasible • Communicate and market the project broadly to increase awareness and grow the community of users and developers
Project Overview • Funded in part by a National Leadership Grant from the Institute of Museum and Library Services • Original name: Variations on Video • Planning grant: – August 2010 – July 2011
• Implementation grant: – October 2011 – September 2014 September 2015
• Partnership between libraries at Indiana University and Northwestern University
Project Partners and Advisors
Project Partners and Advisors
Avalon is a Hydra Solution Bundle
Why Avalon? • Existing repository systems don’t work well for timebased media – DSpace, Fedora, Digital Commons, ContentDM, … – Issues: integration with streaming servers; hard to support time-based navigation; difficulties in integrating access control
• Existing media systems don’t work well with repositories – Kaltura, Brightcove, Ensemble, ShareStream, … – Issues: focus on short-term access for teaching and learning; limited metadata and access control capabilities; integration with preservation workflows
• Strong desire for community-based solution
Demonstration
Avalon Architecture Users
Integrations
ILS LMS, websites Authentication Authorization Archival Storage
Avalon Media System Hydrant Rails App OmniAuth CanCan Fedora
Media Player (mediaelement.js)
Hydra Head
Solr
Blacklight
Matterhorn
Rubyhorn
Media Server
Avalon code – consists of Hydrant Rails App and the Rubyhorn gem. Other pieces are leveraged.
Search Browse View
Ingest Describe Manage
Desktop, Mobile All Users
Browser, Drop box
Collection Managers
Release History • • • • • • •
0.1 – July 2012 0.5 – October 2012 1.0 – May 2013 2.0 – October 2013 3.0 – May 2014 3.1 – July 2014 3.2 – December 2014 (expected)
Julie Hardesty
Jon Dunn
metadata specialist
project director
Chris Colvard
Mark Notess
lead developer
product owner
Cynthia Ramlo
Leah Lee
lead user experience designer
developer
Brian Keese
Ariadne Rehbein
developer
project assistant
Development Team Michael Klein
Karen Miller
lead developer
metadata specialist
Stefan Elnabli
Dan Zellner
audiovisual specialist
testing and production support
Claire Stewart co-project director
Julie Rudder product owner
Current development process • • • •
A single, blended team Agile Scrum process Code in public GitHub Face-to-face meetings at least 2x/year • Daily standups, IRC
0 – 3.1 Release functionality End-user functionality • Faceted discovery for search and browse • Video and audio playback in browsers or on mobile devices • Stream-level security • Ability to embed media player in other websites • Persistent URLs • RTMP and HTTP streaming, Flash and HTML5-based player Collection management • Interactive or batch uploading of media files and MODS metadata • Private collection dropboxes • A hierarchical model for permissions that supports a flexible approach to batch collections-based content management • Customized thumbnails by taking a “snapshot” or specifying a timepoint • Direct import of previously transcoded derivatives • Avalon transcoding for multiple quality derivatives
0 – 3.1 Release functionality Integration with other enterprise systems • Red5 and Adobe media servers • Local authentication services (CAS, LDAP and others, using OmniAuth) • Learning management systems, via the Learning Tools Interoperability (LTI) standard • Support for LDAP group access control • Support for persistent URL systems (PURL, Handle) • Master files: leave, delete, or rename/move after transcoding Installation • Easy installation and configuration via a virtual machine image and other methods • Fully-transparent integration testing using travis-ci.org - every commit and pull request triggers a build • Configuration approach for easier customization where possible
Upcoming releases (Fall/Winter 2014) Release 3.2 (December) • Upgrade to latest versions of Hydra, Blacklight, and Bootstrap • Metadata: adding language, original physical description, related item links, terms of use • Let staff select multiple items and publish, unpublish, delete, set access permissions, assign to collection Release 3.2.1 (January/February 2015) • Import descriptive metadata from an Integrated Library System • Import multiple quality levels of pre-transcoded derivatives
Release 4 (Spring 2015) • Structural metadata – Import and store structural metadata – Allow users to navigate using media sections
• Prioritize media in the transcoding queue • Bulk update of metadata • Fedora 4 (testing?)
Beyond Release 4 • Reporting and metrics • Time limits for access • IP restricted access • APIs for ingest, analytics, metadata, etc. • Fedora 4 migration • Accessibility (navigability, closed captioning) • Transcripts/Auxiliary items • Improved metadata • Playlists & clip making • Integration with Wowza and Amazon Cloud Front • Publish to YouTube or other systems • Integration with Spotlight exhibits tool • Internationalization
Follow our progress • For sprint summaries, read our blog • For more details and recordings, visit our wiki: wiki.dlib.indiana.edu/display/VarVideo/Avalon+Media+System • Avalon Current Sprint page wiki.dlib.indiana.edu/display/VarVideo/Avalon+Current+Sprint • Previous Sprints page wiki.dlib.indiana.edu/display/VarVideo/Previous+Sprints • Sprint Demos wiki.dlib.indiana.edu/display/VarVideo/Sprint+Demos
Community and future directions • Multiple implementations currently underway – Stanford, UVa, UToronto, UT Austin, …
• Working on new features, including: – Structural metadata – Transcripts and captioning
• Starting to deal with code contributions • Working with Hydra community – Leverage common features across “heads” – HydraDAM and Avalon, other preservation-specific integrations
• Sustainability/governance/business model • Exploring hosted options
Sustainability and Governance • • • •
Focus in 2015 with potential funder support Want to grow the number of implementers Ongoing costs of software support and development Potential revenue streams – Membership/sponsorship – Income from hosting – In-kind developer contributions
• Governance and management structure – Potential alignment with existing 501(c)(3) organization in higher ed open source community
Julie Rudder
[email protected] Digital Initiatives Project Manager DLF Forum 2014
Use Cases Course Reserves – restricted access Archival Collections MMLC – video management by units outside the library
Future Additional units outside the library? Integration with other campus media solutions?
Better integration with Ares Course Reserves?
Usage so far....
Existing Production Systems
Avalon Web App
Load Balancer
Streaming FMS 2
Streaming FMS 1
VM
Transcoding
Shared SOLR
Dedicated SOLR Core
4 TB dedicated Hardware Shared Production Fedora
Fedora
Production Environment
Background Integrations on Streaming at NUL • Canvas and Blackboard
Streaming Media since 2001
• Existing handle server
Total Audio-Video Assets Served: 34,146 • LDAP
• Adobe Media Server (secure streaming) • Related -- Ares Course Reserves (request mechanism)
Implementation Pilot – Summer 2013 • • •
A few classes support Two public collections Local VM infrastructure
Full production – Fall 2014 • • •
All new requests via Avalon (no migration) MMLC launches Unit NUIT (central data center) infrastructure
AVALON Retire current infrastructure by summer 2015 All new digitization
Summer 2014
Summer 2015
2004 – 30,000+ assets served Retire legacy systems
Staffing 1 DevOps (install, configure, support)
1 Developer (app customization, support)
1 Project Manager (app oversight, integrations oversight)
1 Service Owner (liaise with units & users, workflow, support)
Additional support –
st 1
tier
Business hours Library Digital Collection Staff 24 hour -Canvas helpdesk support
24 HOUR Central Campus IT helpdesk
24 HOUR Canvas helpdesk
• Avalon at Indiana University
IU Context • • • • •
History of audio/video digitization Legacy systems for delivery Eight campuses New demands Media Digitization and Preservation Initiative
Legacy Systems
IU Deployment Environment University IT Services
Library servers
Avalon Web App Solr, Fedora
Redundant Archive Storage (HPSS)
Master Files
Shared SAN
Derivatives
4 core, 1GB RAM VM 1GB Disk (VMware)
Transcoding 12 core, 120GB RAM,8TB dedicated hardware
IU Enterprise Streaming
Adobe Media Server
Integrations • • • • •
CAS authentication Microsoft Active Directory (via LDAP) PURL resolver Canvas and Sakai (via LTI) Scholarly Data Archive (HPSS master file storage) • Adobe Media Server (secure streaming)
IU Production Instance Pilots began Fall 2013 Moved to production, October 2014
Archival Film
Omeka Online Exhibit: World War II Propaganda Frilms
Media Within Archival Collections
Online Publishing: Ethnomusicology Multimedia
Specialized Avalon Instance: IJCCR
Specialized Avalon Instance: Kelley School of Business
Future Migrations • Video e-reserves (in pilot) • LTI integration with Sakai, Canvas
• Licensed educational video • LDAP for campus restriction
• Variations: music audio • Pedagogical, annotation tools
• Media Digitization and Preservation Initiative
Production Staffing • 10% of a systems administrator (server management; higher initially) • 20% of a developer (install, configure, support; higher initially) • 20% of project/service manager (app oversight, workflow planning, work with units and users, support) • Does not include collection management or central IT staffing of shared services
IU Media Digitization and Preservation Initiative
• mdpi.iu.edu
Avalon in the IU Media Digitization and Preservation Initiative (MDPI) • 8.4 petabytes of audio and video data over 5 years • Avalon serving as key access component – Integrations: • • • • •
Metadata: MARC/ILS, EAD/XTF Authentication: CAS Authorization: Active Directory Learning Management: Canvas, Sakai Media streaming: Adobe Media Server
• Side-by-side with trusted repository for preservation – Bit storage in existing hierarchical storage management system – Hope to leverage HydraDAM work – Exploring out-of-region options
Questions? For more about Avalon • Website: http://avalonmediasystem.org/ • Avalon e-mail list: E-mail
[email protected] to subscribe @avalonmediasys facebook.com/avalonmediasys
Backup Screenshots
Faceted browse via Blacklight
Desktop/Mobile Player
Embedding
LTI (in Canvas)
LTI (in Canvas)
Manage Content
Add item: Upload, Dropbox, or Batch
Dropbox view
Batch Folder
Descriptive Metadata: MODS
Access Control
Permanent URL minted on Publish
Bulk actions
