Update on April 2013 Activities

May 10, 2013 [1]

[Download PDF] [5]

You can follow HathiTrust on Twitter [6] or Facebook [7], or subscribe to receive email updates [3] (via Google Groups).

Top News

Print Disabilities Access

HathiTrust released a new service that allows designated proxies at partner institutions to provide access to in-copyright works in HathiTrust to users at their institutions who are certified as having a print disability. See http://www.hathitrust.org/accessibility [8] for more information.

Website Redesign

The HathiTrust website, including all web applications, was updated with a unified design and feature set, improving the overall look and functionality of the site. Details are available at http://www.hathitrust.org/hathitrust_new_look [9].

Board of Governors

The Board of Governors held an in-person meeting in Chapel Hill, North Carolina on April 28. The Board had a very full agenda, including a review of the Constitutional Convention ballot initiatives [10] and work related to the Program Steering Committee (including the appointment of members).

Ingest

Local Digitization

The survey [11]released last month to institutions that are depositing locally-digitized content will close May 15. The purpose of the survey is to gauge the readiness of institutions to deposit locally-digitized content, and help set a timeline for the development of a next installment of tools to assist in validating and packaging content prior to deposit. HathiTrust continued to provide support for the existing ingest tools and ingest procedures.

Working Groups and Committees

User Experience Advisory Group

The User Experience group continued to review elements of HathiTrust Web applications identified through user feedback and other means as being in need of improvement. The group also helped troubleshoot and make decisions in support of the new interface redesign.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

Staff from the California Digital Library (CDL) and the University of Michigan agreed on revised requirements and workflow changes to accommodate bibliographic rights determinations being made at the University of Michigan rather than in Zephir (the new system). CDL staff began development reflecting the revised requirements. Staff also updated the project timeline [12]. A parallel phase, to be completed prior to moving to Zephir, is now anticipated to run at the end of the summer.

Copyright Review

A summary of the determinations from HathiTrust copyright review activities in April is given below. See CRMS-US [13] and CRMS-World [14] for more information.

HathiTrust Research Center (HTRC)

HathiTrust Research Center project members assembled in Urbana, Illinois to launch Phase II of HTRC’s efforts. Following the successful release of HTRC software and services on March 30, 2013 (marking the completion of Phase I), HTRC is focusing its limited resources in Phase II along two lines: 1) community engagement and community-driven enhancements to HTRC software and services; and 2) the HTRC Sloan Cloud for non-consumptive research.  To the former, the HTRC is pleased to welcome Miao Chen as Assistant Director of Education and Outreach. Miao recently received her PhD in information sciences from Syracuse University. She will be helping to organize the outreach efforts. A significant upcoming event is the 2nd HTRC UnCamp, scheduled for early September. The event will be held at the University of Illinois in Urbana-Champaign, Illinois. Details will follow. The HTRC Sloan Cloud, a project in development that is funded by a grant from the Sloan Foundation, is the technical infrastructure for carrying out text analysis using general university compute resources securely in way that does not violate terms of copyright.

mPach

University of Michigan staff determined functional requirements for relating journal-level MARC records to article-level MARC records, and journal-level “aboutware”.

Development Updates

HathiTrust institutions performed the following work related to applications and Web interfaces:

Collection Builder

Staff investigated ways of improving the sorting of serial publications.

Data API

Staff implemented a mechanism for automatically deleting registered Data API keys that have not been activated.

Full-text Search

Staff explored ways to improve relevance ranking in full-text search results and fixed two bugs in searching materials with Chinese, Japanese, or Korean characters: searches did not always work when quotes were used around characters, and there was a problem in recognizing the Boolean “AND” operator.

PageTurner

Staff implemented a new code-debugging scheme for testing access by rights attribute.

Storage Hardware Replacement Cycle

HathiTrust installed new storage at the Indiana and Michigan sites that will both accommodate 2013 volume projections and replace storage scheduled for retirement. Storage due for retirement will be taken offline starting in late May or June.

Outages

Pages did not display in HathiTrust on Thursday, April 4 from 3:30 - 4:15pm due to a software problem that was identified and corrected.

Daily full-text indexing was suspended from Wednesday, April 24 to Tuesday, April 30 due to a complication with a storage system software upgrade. The problem was resolved and normal indexing processes resumed.

New Growth

As of May 1:

Public Domain (~31%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Most Accessed Volumes

May Forecast

• Continue to explore ways of improving relevance ranking  of full-text search results
• Begin automatic data migration from old to new storage at Indiana site

Presentations

• Jeremy York, “HathiTrust as a Model for Preservation and Access”, [25] Media Preservation Conference, Indiana University, April 17, 2013.
• Jeremy York, “HathiTrust: A Second Life for Library Collections” [26], Exploring Humanities Cyberinfrastructure, University of Michigan, April 30, 2013.

Update on March 2013 Activities

April 12, 2013 [1]

[Download PDF] [27]

Top News

Program Steering Committee Nominations

HathiTrust has requested nominations from partner institutions for the HathiTrust Program Steering Committee (PSC). The responsibilities of the Program Steering Committee are described in Article VII, Section 3 of the HathiTrust [28] Bylaws [28]. Among the first areas of work to be undertaken by the PSC are the ballot initiatives [10] passed at the 2011 Constitutional Convention, including expanding access to US government documents and creating infrastructure for shared monograph storage initiatives. Any member of a partner institution may submit nominations for the PSC until April 22 via the form at http://goo.gl/ [29]TV0CN [29].

HTRC Software Release

The HathiTrust Research Center reached a development benchmark in its release of production infrastructure to support data mining and textual analysis of volumes in HathiTrust.

The infrastructure includes an entrance portal, search and collection-building tools (using Blacklight [30]), and access to SEASR analysis algorithms that can be run against the HathiTrust public domain corpus (more than 3 million volumes).  In addition to the production services, the HTRC offers a development “sandbox”.  The sandbox runs against non-Google scanned content (about 260,000 volumes) and provides a test-bed for interested researchers to experiment with writing their own algorithms for use in the HTRC infrastructure.

The production release concludes the first six month period in Phase 2 of development of the HTRC (Oct 2012-March 2014). Phase 2 will also include the development of the HTRC-Sloan-Cloud – infrastructure that will include additional mechanisms to allow secure, non-consumptive access to the entire HathiTrust corpus – and systems to accommodate the full 10.6 million HathiTrust volumes in the HTRC. For more information on HTRC services and testing of the production infrastructure, please join our HTRC-usergroup-l listserv at https://list.indiana.edu/ [31]sympa/subscribe/htrc-usergroup-l [31].

Government Documents Registry Analyst 

HathiTrust is pleased to announce the hiring of Valerie Glenn to the Government Documents Registry Analyst position [32]. Valerie has served as a Federal Depository Librarian at both the University of Alabama and the University of North Texas, and has managed a variety of projects and activities related to government documents. Valerie brings deep expertise to a two-year initiative to begin to construct a comprehensive registry of U.S. federal government documents. This work is part of a larger HathiTrust effort to expand access to US government documents [33]. More information about the project is available at http://www.hathitrust.org/ [34]usgovdocs_registry [34].

HathiTrust Institution Survey

HathiTrust distributed a survey created by Syracuse University to gather information about institutions’ experiences with HathiTrust. The survey includes questions about print disabilities services, special collections, digital humanities, use of HathiTrust, and technical implementation issues. The survey is available at http://www.surveymonkey.com/s/ [35]9ZZ9KMW [35] until April 26. We encourage all partner institutions to participate. Results will be summarized and made available. 

HathiTrust Board of Governors

During a March meeting, the Board of Governors reviewed the HathiTrust budget and planned a longer agenda for an in-person meeting in April.

Ingest

Local Digitization

HathiTrust prepared a survey to send to institutions that have indicated they intend to deposit locally-digitized materials. The purpose of the survey is to gauge interest in, and aid in determining a development timeline for, enhanced tools to assist in validating and packaging materials prior to submission to HathiTrust. The survey will be sent out in mid April. HathiTrust also provided support to several institutions making preparations to deposit locally-digitized content. 

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Experience Advisory Group

The User Experience Advisory Group was pleased to welcome a new member: Matt Morgan, Director of the Website, NYPL Office of Strategic Planning.  The group continued to review elements of HathiTrust Web applications identified through user feedback and other means as being in need of improvement.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

Staff from the California Digital Library (CDL) and the University of Michigan discussed implications of a new requirement that automated bibliographic rights determinations must occur at the University of Michigan rather than at the University of California. The teams expect to have revised requirements finalized in April. CDL staff are determining the impact that the change will have on the development timeline.

Copyright Review

A summary of the determinations from HathiTrust copyright review activities in March is given below. See CRMS-US [13] and CRMS-World [14] for information.

mPach

Staff at the University of Michigan discussed modifications that are planned to be made to the Collection Builder application in order to use it as a means to navigate from articles in a single journal to the journal’s “aboutware” (information about editorial boards, submission policies, etc.).  Staff also discussed issues of discovering journal aboutware through the HathiTrust catalog, full-text search and Collection Builder interfaces, and user pathways for navigating between journal-level catalog records, article-level catalog records, and aboutware. More information about mPach is available at http://www.hathitrust.org/mpach [37].

Development Updates

HathiTrust institutions performed the following work related to applications and Web interfaces:

Collection Builder 

Staff corrected issues in the display of authors and titles, added an option to remove collection items to a batch Collection Builder tool, and discussed ways of supporting very large collections. Staff also worked on the development of new features to be implemented as part of the Website Redesign (see below).

Digitization Sources

Staff planned a new back-end strategy for recording content digitization sources and associated access parameters, which are expressed in HathiTrust interfaces.

Full-text Search

Staff continued research to improve relevance ranking.

PageTurner

Staff re-engineered a tool for testing and debugging volume access controls. 

Website Redesign

Staff continued work to implement a redesign of HathiTrust Web interfaces, using a unified framework for application code. Release of the new design is expected in April. Other improvements to be made in conjunction with the redesign include:

• Pagination of results in Collection Builder.
• The addition of book cover thumbnails to Collection Builder and full-text search results.
• Improved viewing interface in PageTurner and differential display of works depending on their reading order (right-to-left versus left-to-right).
• Ability to cancel full-book downloads.

Screenshots of some of the redesigned pages are given at the end of the update. 

Storage Replacement Cycle 

HathiTrust began to install new and replacement storage hardware at the Michigan repository instance as part of its regular purchase and replacement cycle. Installation of new storage and retirement of storage to be replaced will continue in April.

Server Replacement Cycle 

HathiTrust purchased and received new production web servers and new development web and index servers to replace servers scheduled to be retired. The new development servers will make use of virtualization to improve resource utilization and availability, and to reduce acquisition and operational costs. In concert with this upgrade, which is planned for the second quarter of 2013, the Linux distribution in use for the entire server infrastructure is being changed from Red Hat to Debian, to provide better and more manageable infrastructure for deploying Ruby-based applications.

Outages

No outages were reported in March.

New Growth

As of April 1:

Public Domain (~31%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Most Accessed Volumes

* Approximate due to a system configuration change.

April Forecast

• Complete the website redesign, including testing and deployment.
• Continue installation of new and replacement storage.

Presentations

• Cory Snavely and Jeremy York, “The HathiTrust Repository: Policy and Technical Issues in Building the Digital Archive [46]”, University of Michigan, March 1, 2013. 
• Stephen Downie, “Unlocking the Secrets of 3 Billion Pages: Introducing the HathiTrust Research Center [47]”, University of Hong Kong, March 18, 2013.
• Heather Christenson and John Wilkin, “Intellectual Property Rights & the HathiTrust Collection [48]”, UNESCO The Memory of the World in the Digital Age Conference Proceedings, September 26-28, 2012.
• Jeremy York, “A Preservation Infrastructure Built to Last: Preservation, Community, and [49]HathiTrust [49]”, UNESCO The Memory of the World in the Digital Age Conference Proceedings September 26-28, 2012.

Update on February 2013 Activities

March 8, 2013 [1]

[Download PDF] [50]

Top News

HathiTrust Bylaws

HathiTrust institutions voted unanimously to accept bylaws [28] put forward by the Board of Governors. The Board will have its first meeting following the acceptance of the bylaws in April.

HathiTrust Research Center Software Release

The HathiTrust Research Center (HTRC) is planning a release of its software infrastructure on March 31, 2013. The release is of two separate cyberinfrastructure stacks, the “HTRC sandbox” and “HTRC production stack”, each loaded with the latest and greatest service components and tools. The “HTRC sandbox” is an open test bed for community experimentation. It contains a subset of public domain volumes. The “HTRC production stack” hosts the full HathiTrust public domain corpus. Both the sandbox and production stack offer the suite of services and tools, with bug fixes and other improvements, that debuted during the September 2012 UnCamp. Looking ahead, a June 2013 release of the HTRC production stack will include the HTRC-Sloan-Cloud in early support of “non-consumptive research”. Non-consumptive research is research in which the content of digital works is read or operated on (“consumed”) in an automated way only; no portions of works are displayed or available to be read by researchers themselves during computation or in computational results.

We welcome anyone to join HTRC email lists for general announcements [51], technical discussion [31] about the HTRC, or announcements specific to the 2012 UnCamp [52] (see http://www.hathitrust.org/htrc [53] for details on each list).

Ingest

Local Digitization

HathiTrust provided support to several institutions on use of the HathiTrust ingest tools [54] and issues related to image conversion and metadata.

Internet Archive Digitization

HathiTrust ingested new content from Columbia University, the Library of Congress, Penn State, the University of California, the University of Illinois, and the University of North Carolina-Chapel Hill. HathiTrust also loaded bibliographic records from Boston College in preparation for content deposit.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Experience Advisory Group

The User Experience Advisory Group continued to review elements of HathiTrust Web applications identified through user feedback and other means as being in need of improvement.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

Staff from the California Digital Library (CDL) and the University of Michigan continued to plan for the transition to Zephir for HathiTrust bibliographic data management. CDL staff began loading records into the Zephir production database.

Copyright Review

A summary of the determinations from HathiTrust copyright review activities in February is given below.

mPach

Staff at the University of Michigan made architectural and procedural decisions related to the creation, storage, and revision of “aboutware” (information about editorial boards, submission policies, etc.) for journals deposited via mPach. Staff finalized journal article ingest procedures, and continued discussions about optimal handling of non-textual content that is embedded in articles or submitted as supplementary material. Staff also reviewed and updated the project timeline [37].

Development Updates

Full-text Search

HathiTrust received a revised estimated shipping date for high-performance storage that will be used to improve full-text search services. The systems are expected to arrive in March for installation and testing.

Michigan staff made changes to optimize synchronization of the full-text search index from the HathiTrust repository instance in Michigan to its mirror site in Indiana. At the same time, staff performed a full update of the index to be sure that all volumes in the repository were indexed appropriately. Staff developed a process to automatically update the full-text index when print holdings information submitted by partner institutions is updated or added. Staff also continued work to improve relevance ranking of search results.

PageTurner

HathiTrust updated the way URL parameters are sent to Google Analytics in order to improve usage reporting. The change will especially affect full-text searches performed within individual volumes.

Print on Demand

HathiTrust finalized and deployed processes for producing PDFs that are optimized for printing on Expresso Books Machines.

Storage Hardware Replacement Cycle

HathiTrust began processes to replace production web servers at the Indiana data center, and development index and web servers at the Michigan data center. The server upgrades are planned to begin in late March or April.

Website Redesign

Staff developing HathiTrust Web applications continued to implement a unified framework for application code. This work is being done as part of a larger effort to redesign the look and feel of HathiTrust’s Web interfaces. The redesign is scheduled for release in April.

Outages

HathiTrust was unavailable for some users from Tuesday, February 26 at 11:45pm to Monday, February 27 at 5:45am due to a network problem affecting the entire University of Michigan campus.

New Growth

As of March 1:

Public Domain (~31%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Most Accessed Volumes

March Forecast

• Continue work to improve relevance ranking for full-text search results.
• Continue to develop deployment process for full-text search spelling suggestion feature.
• Continue work on unified Web application framework and website redesign.

Presentations

• Jeremy York, “Your Library, Now Online! Putting HathiTrust in the Context of New (and Traditional) Library Services” [61], MCLS Webinar, February 6, 2013.
• Tom Burton-West, “Practical Relevance Ranking for 10 Million books” [62], Code4Lib, February 12, 2013.
• Jeremy York, “Putting It All Together: HathiTrust Vision, Practice, and Implementation” [63], SENYLRC Webinar, February 20, 2013

Update on January 2013 Activities

February 15, 2013 [1]

[Download PDF] [64]

Late Breaking News

HathiTrust Bylaws Accepted

HathiTrust institutions approved bylaws put forward by the Board of Governors for voting in January. The bylaws are available at http://www.hathitrust.org/documents/hathitrust-bylaws-201302.pdf [28].

Top News

HathiTrust Resources

HathiTrust hosts a page of resources [65] including handouts and informational sheets created by the Communications Working Group, and links to information about HathiTrust that institutions have posted for their own constituencies. If you have posted resources about HathiTrust (including videos, library guides, etc.) that are not listed, please let us know (feedback@issues.hathitrust.org [66]), and feel free to use and share those that are available.

Ingest

Local Digitization

HathiTrust hosted a conference call with members of several partner institutions that have been working with HathiTrust’s ingest tools [54], to discuss development options for the next iteration of the tools. A summary of the meeting, including next steps, is posted in a Google Group forum on HathiTrust Ingest [67]. Individuals interested in issues surrounding ingest of locally-digitized content into HathiTrust are welcome to join. HathiTrust continued to respond to inquiries about the ingest tools and ingest of locally-digitized materials.

Internet Archive Digitization

HathiTrust began ingest of new batches of volumes from the University of Florida, the University of Illinois, and the University of North Carolina at Chapel Hill. HathiTrust also loaded bibliographic records for new volumes from Columbia University and Penn State.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Experience Advisory Group

The User Experience Advisory Group began to review elements of HathiTrust Web applications that were previously identified as in need of improvement and to review feature requests that have been submitted to the User Support Working Group.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) continued to work with staff at the University of Michigan to test data exports from Zephir and to plan for the upcoming transition to Zephir for HathiTrust bibliographic metadata management. CDL staff made modifications to support rights determinations on bibliographic records that conform to the Resource Description and Access (RDA) standard. CDL is in the process of implementing a new backup strategy for Zephir to ensure the service will be accessible in the event of an outage. An updated timeline for the project is posted at http://www.hathitrust.org/htmms [12].

Copyright Review

A summary of the determinations from HathiTrust copyright review activities in January is given below.

mPach

Staff at the University of Michigan met to discuss possible changes to the HathiTrust PageTurner, item-level search, and item-level search results to accommodate materials submitted via mPach. Staff began working with sample analytic MARC records and METS objects to determine the scope of anticipated changes. Staff also reviewed the mPach ingest process and discussed workflows for processing issue- and journal-level metadata.

Development Updates

Full-text Search

Staff at Michigan continued to work on relevance ranking for full-text search results. Experiments performed in January found that some very short documents were being ranked too highly by Solr’s default ranking algorithm. Preliminary tests indicated that the issue may be resolved by using Solr’s new BM25, DFR and Information Based ranking settings. Experiments will continue in February.

Michigan staff received official relevance judgments for full-text search data submitted to the 2012 INEX Prove-IT Book Track [68] (see also the paper "Practical Relevance Ranking for 10 Million Books [69]"). After evaluating the results, staff submitted new data using updated relevance judgments.

Staff at CDL made significant progress on a spelling suggestion feature for full-text search. Staff improved the relevance of results through the creation of a new dictionary of suggestions that uses a bigram index for an entire shard’s worth of documents (about 800,000 volumes). The HathiTrust full-text index is broken into “shards” in Solr’s search architecture. Staff also tuned the algorithm that provides suggestions and made significant changes to suggestion scoring parameters, significantly increasing the quality of results. CDL will now look toward deployment of the new feature.

Michigan staff made adjustments to the full-text search indexing process to better support experimental indexing runs and to synchronize data between the full-text search index and information in the HathiTrust print holdings [70] database.

HathiTrust completed the purchase of high-performance storage for full-text search. The new systems are expected to be received in late February for installation and testing.

Lawful Uses

HathiTrust began to implement application-level changes to support access to materials by designated representatives at partner institutions on behalf of users at those institutions who have print disabilities. Designated representatives will need to register and to access HathiTrust using their Shibboleth login from a fixed IP address. Further details on the service will be forthcoming.

PageTurner

HathiTrust completed stylistic changes to messages in mobile PageTurner that appear when special access to materials is granted (e.g., access to volumes that fall under Section 108 conditions or to users who have print disabilities).

Storage Hardware Replacement Cycle

HathiTrust completed projections for new and replacement storage needed for HathiTrust in 2013.

Website Redesign

Staff at Michigan drafted implementation guidelines for a unified Web application framework for HathiTrust. The framework will simplify the execution of a redesign of the HathiTrust website, which is expected to be completed in April.

Outages

HathiTrust was unavailable from 5:00-5:40pm EST on Monday, January 21 due to an error in a software release.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org. [71]

New Growth

As of February 1:

Public Domain (~31%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Most Accessed Volumes

February Forecast

• Continue to work on authorization to support access for users who have print disabilities
• Continue relevance ranking tests in full-text search
• Begin steps to deploy spelling suggestion feature
• Continue work on common Web application framework

HathiTrust Year in Review 2012

January 29, 2012  [1]

Update on December 2012 Activities

January 11, 2013 [1]

[Download PDF] [118]

Top News

Access to Out-of-Print and Brittle or Missing Items

One of the lawful uses of in-copyright works HathiTrust has been pursuing is to provide access on an institutional basis to works that fall under United States Copyright Law Section 108 conditions: works in HathiTrust that are not available on the market at a fair price, and for which print copies owned by HathiTrust member institutions are damaged, deteriorating, lost or stolen. As a part of becoming a member, institutions are required to submit information about their print holdings for fee calculation purposes. We have also been requesting information about the holdings status and condition of works, to facilitate uses of works where permissible by law (specifications for HathiTrust holdings data are available at http://www.hathitrust.org/print_holdings [70]).

As of December 2012, we are using the holdings status and condition information submitted by United States member institutions, in combination with information about the market availability of works stored in the HathiTrust rights database, to determine whether or not access to applicable in-copyright works in HathiTrust is allowed. The specific terms of access are as follows:

• Access is only available to users affiliated with HathiTrust member institutions in the United States, and only from U.S. soil.
• In order to gain access, users from member institutions must be authenticated into HathiTrust via Shibboleth using their institutional login.
• Print copies of the works in HathiTrust must be owned currently or have been owned previously by the institution’s library system.
• The number of users who can access a given digital copy at a time is determined by the number of print copies held (or previously held) in the library system. If a library system only has one print copy, only one user at a time will be able to access the digital copy.

A general scenario for how out of print determinations are made and communicated to HathiTrust is available in the HathiTrust rights database documentation: http://www.hathitrust.org/rights_database#op [119].  Additional information on the service is available at http://www.hathitrust.org/out-of-print-brittle [91].

HathiTrust Bylaws

The Board of Governors completed a draft of HathiTrust bylaws, which was distributed to partner institutions in early December for comment. The Board is working on a final version with consideration for partner comments. The final version will be put forward to partners for voting in January.

Research Center Video

The Research Center released an informational video, following on the UnCamp that was held earlier in the fall of 2012. The video can be accessed at http://www.hathitrust.org/htrc [53].

Most-accessed Volumes in HathiTrust

This month we are including a new metric in our newsletter: the most accessed works in HathiTrust by pageview count. A table of volumes is included at the end of the update.

Ingest

Local Digitization

Staff at the University of Michigan met to discuss the next steps for HathiTrust’s ingest tools [54], created to aid institutions in validating and packaging locally-digitized content prior to deposit in HathiTrust. A conference call is planned in January, which will include members of several partner institutions that have been working with the existing tools, to discuss possibilities and options for the future. HathiTrust continued discussions about deposit of locally-digitized materials with the University of Illinois, and responded to questions from McGill University.

Internet Archive Digitization

HathiTrust ingested new content from Penn State University and loaded records for content from the University of Florida and University of North Carolina-Chapel Hill. Ingest of volumes from Florida and UNC, and additional volumes from Penn State, is expected to occur in January.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) continued to work with staff at the University of Michigan on preliminary testing of data exports from Zephir, the new HathiTrust bibliographic management system under development by CDL. CDL and Michigan staff continued to plan for the upcoming period when Zephir and the bibliographic management system at Michigan will be run in parallel, prior to the full transition to Zephir.

Copyright Review

A summary of the determinations from HathiTrust copyright review activities in December is given below. The numbers this month reflect a different methodology for aggregating statistics. In previous months, the number of Reviews was given, and the number of volumes reviewed that were Opened. In the majority of cases, volumes are reviewed more than once (by more than one person). This meant that the number of Reviews reported was larger than the number of actual volumes reviewed. Similarly, the number of volumes Opened represented volumes that may have been determined in more than one review to be in the public domain. The table below provides a more accurate representation of the number of volumes where a determination was made, and what the determination was. We will use this representation going forward.

IMLS Quality Grant

The project team will present a research poster at ALA Midwinter in Seattle, during the Preservation Administrators Interest Group Meeting on Saturday, January 26.  The poster will focus on digitization error related to material characteristics of a book. The project team continues to focus on more complex analyses of the data collected in the past year and also on presentation of the findings. Additional findings and results will be posted on the project website later this month: http://hathitrust-quality.projects.si.umich.edu [120].

mPach

Staff at the University of Michigan revised the list of modules [101] for mPach, to reflect recent changes in the planned system architecture.  An extensive conceptual workflow for ingest of an mPach Submission Information Package into HathiTrust has been devised and will be finalized soon. Michigan staff finalized plans for modifications to the HathiTrust Data API to support the retrieval via the API of JATS XML, derivative formats, and supplemental materials that may be associated with a JATS XML article.

Development Updates

Full-text Search

Staff at the University of Michigan released a bug fix for the Solr edismax query parser and a new index into production in late December (See the Update on November Activities [121] for details.).  These changes will significantly improve the precision of CJK (Chinese, Japanese, and Korean) search results.

Michigan staff began preliminary analysis of HathiTrust document length statistics. The results of the analysis will aid in designing tests of length normalization features for the new relevance ranking algorithms available in Solr 4.0 [122]. Staff built a test index using the new relevance ranking algorithms available in Solr 4.0 (DFR, BM25. IB).  Experiments using the test index will begin in January.

Staff at Michigan made a final selection of high-performance storage for full-text search and completed pricing negotiations (see the Update on November Activities [123] for background). Purchase of the storage is expected to be complete in January, with installation and testing to follow soon after in late January or early February.

Web Applications

Michigan staff completed the removal of sensitive information from source-controlled HathiTrust application code to designated system-level locations. Staff also completed the separation of privileges for accessing application databases. Different classes of applications now connect as different database users with different privileges.

Michigan staff began to implement improvements to the display of special access messages (e.g., for works that are out of print and brittle) in the mobile version of PageTurner.

The PageTurner scroll view now advances by full pages when the navigation controls are used (e.g., next page button), rather than advancing by half of a page at a time.

The HathiTrust feedback form now detects content and metadata-related feedback submissions by CRMS (Copyright Review Management System) reviewers, pre-filling problem tickets with CRMS-specific information to simplify the management of support requests.

Outages

No outages were reported in December.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org. [71]

New Growth

As of January 1:

Public Domain (~31%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Most Accessed Volumes

Papers and Presentations

• Jeremy York, “More, Better, Together: HathiTrust Aspirations and Accomplishments” [132], The European Library/Europeana Libraries Joint Conference, Universidad Complutense de Madrid, December 4, 2012.
• John Unsworth, Beth Sandore Namachchivaya, “Digital Humanities At Scale: HathiTrust Research Center” [133], Coalition for Networked Information Fall Meeting, December 11, 2012.
• John Wilkin, “HathiTrust: Strategies and Challenges in Consolidating the Published Record” [134], National Diet Library, Japan, December 18, 2012.
• Stephen Downie, “Introduction to the HathiTrust Research Center: A Briefing“ [135], University of Western Ontario, December 21, 2012.

See http://www.hathitrust.org/papers [110] for all papers, presentations, and reports.

December Forecast

• Hold meeting on next steps for HathiTrust ingest tools.
• Begin testing features of new  Solr relevance-ranking algorithms.
• Complete purchase of storage for full-text search.
• Continue work to consolidate CSS framework for Web applications.

Update on November 2012 Activities

December 14, 2012 [1]

[Download PDF] [136]

Top News

Bibliographic Corrections

HathiTrust has received a number of inquiries recently about corrections to bibliographic data. HathiTrust’s general policy on bibliographic data correction is available at http://www.hathitrust.org/bib_metadata_correction [90]. We consider the definitive records for volumes in HathiTrust (which are generally volumes digitized from print originals) to be those held by the depositing institutions. When institutions submit corrections to print records in HathiTrust, these corrections are not automatically propagated to WorldCat. Institutions must update the print records in WorldCat separately.

OCLC creates records in WorldCat for electronic versions of works as they become available in HathiTrust (OCLC uses the hathifiles [4]to identify when new volumes enter the repository and then derives digital master records from the print records identified by the OCLC numbers in the hathifiles). These electronic versions are solely OCLC’s responsibility and under its control. Institutions do not need to, and should not try to update records for electronic versions. We are working with OCLC to refine the process by which records for e-versions are updated to stay in sync with HathiTrust records, and records for print versions that institutions update. We will be providing more information on this in future updates. For the present, if you notice a problem with a record in WorldCat for a HathiTrust volume, please notify us at feedback@issues.hathitrust.org [66].

Infrastructure Changes for Out of Print and Brittle

HathiTrust completed changes that will incorporate the “in print” status of volumes (whether or not a volume is in print), as well as holding status and condition information provided by partners in their print holdings data [70], in volume access determinations.

Ingest

Local Digitization

Staff from Texas A&M University contacted HathiTrust to discuss deposit of locally-digitized volumes related to Texas agricultural history. HathiTrust provided ingest support to the University of Iowa, University of Illinois, and University of Utah, including elaboration of content specifications, help in running image validation tools, and assistance in diagnosing errors. The information page [54] about the tools HathiTrust provides for packaging and validating locally-digitized materials has been revised and includes a link to an updated HathiTrust Deposit Form [137], which in turn includes guidelines and specifications for deposit.

Internet Archive Digitization

The University of North Carolina submitted a sample of bibliographic metadata in anticipation of an upcoming deposit. The University of Florida began deposit of Internet Archive-digitized volumes and anticipates depositing 26,250 items over the next several months. HathiTrust ingested two additional batches of content (totaling nearly 400 volumes) from Penn State, with two more batches to be ingested in December. The University of Illinois deposited more than 800 volumes as part of an ongoing project.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Experience Advisory Group

The User Experience Advisory Group was pleased to welcome a new member, Nadaleen Tempelman-Kluit, to the group. Nadaleen is an Instructional Design Librarian at New York University.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) continued to work with staff at the University of Michigan to test processes for exporting bibliographic data from Zephir for use in HathiTrust services. CDL improved the speed at which data could be exported from Zephir. CDL and Michigan continued to plan for the time when the current bibliographic management system at Michigan and the new system (Zephir) will run in parallel. This will occur prior to HathiTrust moving to Zephir as the bibliographic management system for HathiTrust.

Copyright Review

A summary of copyright review activities in November is given below.

IMLS Quality Grant

The project team continued to plan user studies to evaluate and contextualize findings of the grant project. Grant principal investigator Paul Conway traveled to the University of Minnesota to launch the first user study, which will investigate thresholds for error tolerance in digitized volumes among library collections managers. Focus group meetings and other activities for this study will continue through the first quarter of 2013. The team submitted its second narrative report to IMLS, summarizing activities in the past year. The report will be posted soon on the project website [138].

mPach

Staff at the University of Michigan continued work on a mockup of changes needed to the PageTurner interface to support navigation of XML-based articles. Staff began to develop functionality to render JATS articles in PDF (for download purposes). Staff also engaged in discussions about the mPach article ingest workflow and proposed modifications to HathiTrust’s Collections feature to facilitate navigation among journal articles.

Development Updates

Full-text Search

This past June [139], staff at Michigan discovered a bug [140]in the Solr edismax processer that rendered search precision improvements for CJK (Chinese, Japanese, and Korean) materials smaller than expected. In November, conversations between staff at Michigan and Stanford about issues with CJK support lead Michigan to contact to Solr/Lucene developer and committer Robert Muir for advice. Muir (unaffiliated with Michigan or Stanford), an expert on multilingual issues, wrote and committed a code patch that fixed the bug. Staff at Michigan implemented the code patch and have seen orders of magnitude improvements (as an example the query [東京スカイツリー] (Tokyo Sky Tree) produced about 450,000 hits without the patch and 16 hits after the patch). HathiTrust is very grateful for this assistance. Michigan staff made further improvements to indexing, which will be used in a full re-indexing of the full-text index in December. Staff also produced a sample bigram index, which will be used in ongoing work at California Digital Library on a spelling suggestion feature.

Staff at Michigan reviewed proposals received in response to an RFP issued in October for high-performance storage for full-text search, and are in the process of selecting the final systems to negotiate pricing. Installation and testing of the high-performance storage is tentatively scheduled for January.

Web Applications

HathiTrust made a number of updates to Web applications, including:

1. Initiation of work to remove sensitive information from application code for increased security.
2. Modification of the PageTurner to retrieve bibliographic data from the HathiTrust catalog’s VuFind Solr index, rather than Michigan’s bibliographic database (in preparation for the move to Zephir for metadata management).
3. Correction of a problem in PageTurner that caused execution to fail when DNS servers were unavailable.
4. Migration of mapping information between HathiTrust namespaces and depositing institutions to a database table for easier maintenance.
5. Migration of an access control list for special uses of in-copyright materials (e.g., for copyright or quality review purposes) to a database table to streamline maintenance.

Website Redesign

Programmers for HathiTrust Web applications convened to develop a strategy for implementing a single Cascading Style Sheet (CSS) framework across all applications. A single framework will increase interface consistency and simplify future development, including a planned redesign of the HathiTrust home page and common portions of application interfaces.

Outages

On Saturday, November 3, search within a volume was unavailable to some users from 3:00-8:30am and full-text search was unavailable to some users from 6:00-8:00am due to a temporary disk space shortage on a search server at one HathiTrust site.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of December 1:

Public Domain (~31%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Papers and Presentations

• Kevin Hawkins and Jeremy Morse, “mPach: Publishing directly in HathiTrust [141]”, Digital Library Federation Fall Forum 2012.
• John Weise, Chris Powell, Kat Hagedorn, “HathiTrust: Sharing the Care and Feeding of the Elephant [142]”, Digital Library Federation Fall Forum 2012.
• Stacy Kowalczyk, "Digital Humanities at Scale: HathiTrust Research Center [143]", Digital Library Federation Fall Forum 2012.

See http://www.hathitrust.org/papers [110] for all papers, presentations, and reports.

December Forecast

• Continue work to consolidate CSS framework for Web applications.
• Continue work on indexing of CJK languages and relevance ranking for full-text search.
• Complete the separation of administrative data from code in HathiTrust Web applications.

Update on October 2012 Activities

November 9, 2012 [1]

[Download PDF] [144]

Top News

HathiTrust Board Officers

The HathiTrust Board of Governors has identified officers for the Executive Committee as follows:

Chair: Brian Schottlaender

Chair-elect/Treasurer: Sarah Michalak

Past Chair: Paul Courant

Chair of the Program Steering Committee: Bob Wolven

Executive Director (ex officio): John Wilkin

More information about the Board of Governors, including the charge and full membership is available at http://www.hathitrust.org/board_of_governors [82].

Funded PhD Opportunities with HathiTrust Research Center

The Graduate School of Library and Information Science (GSLIS) and the Illinois Informatics Institute (I3) at the University of Illinois are actively recruiting outstanding doctoral candidates interested in research assistantships with the HathiTrust Research Center (HTRC) to develop the HTRC infrastructure, create mechanisms for outreach and engagement with scholarly communities, and cross-pollinate ideas among HTRC stakeholders. View the full announcement [145] for more information.

Infrastructure Change for Out of Print and Brittle

HathiTrust altered the semantics of the “out-of-print and brittle” (“opb”) designation in the HathiTrust Rights Database to “out-of-print” (“op”) only, as outlined in last month’s update [146]. Volumes with the “op” designation began appearing in the tab-delimited Hathifiles on November 2. All “op” volumes will be updated in the Hathifiles on November 12. Rights Database documentation, including a sample scenario [147], has been updated to reflect the change.

Ingest

Local Digitization

HathiTrust answered questions from staff at the University of Missouri, University of Utah, and University of Washington about ingest of locally-digitized content, including questions about the new ingest tools [54] for packaging content prior to submission to HathiTrust.

Internet Archive Digitization

Penn State and Columbia University provided bibliographic records for new sets of Internet Archive-digitized volumes to be ingested. Content from Columbia University is from its Medical Heritage Library. The University of North Carolina contacted HathiTrust staff to begin deposit of a second batch of Internet Archive-digitized volumes. The Getty Research Institute resumed discussions regarding deposit of its IA-digitized materials.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Experience Advisory Group

The User Experience Advisory Group provided feedback on a new landing page for Limited (search-only) volumes in HathiTrust and a prototype of a new PageTurner design created by University of Michigan staff.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) staff worked with staff at the University of Michigan to test data exports from Zephir that will be used in HathiTrust services such as bibliographic and full-text search. The testing examined issues of performance in data transfer, as well as the structure of the exports.

CDL staff completed testing of the Zephir bibliographic record submission process with the majority of institutions that are contributing records to HathiTrust on an ongoing basis. CDL and HathiTrust staff met to discuss the process for communicating with institutions about submission of bibliographic data and content once the cutover to Zephir occurs.

Copyright Review

A summary of copyright review activities in October is given below.

IMLS Quality Grant

Members of the project team continued preparations to launch the first of two user studies related to content quality. The first study will use image review exercises and focus groups to examine thresholds of error tolerance in digital volumes for library collection managers. Staff from the University of Michigan and University of Minnesota will participate in the study. 

The project team analyzed outcomes of its meeting with imaging scientist Don Williams, which took place in September, and enhanced its catalog of commonly identified illustration errors based on information from the meeting.

The team worked to finalize a data curation profile and produce final datasets of the data collected during the grant project. More information on the project is available on the project website [138].

mPach

Staff at the University of Michigan completed a prototype of the Prepper module (see a list of all modules [101]), as well as enhancements to PageTurner to display  journal articles encoded in JATS XML, in time for a presentation and demo at the 2012 DLF Forum.

Development Updates

Authentication

HathiTrust fixed a bug that prevented authentication for users who had certain character entity references (e.g., “é”) in their Shibboleth displayName attribute. HathiTrust also implemented functionality to map users from multiple authentication Identity Providers (IdPs) to a single partner institution. This functionality comes into play when multiple campuses or organizations are members under the aegis of a single institutional.

Data API

HathiTrust completed final development work associated with supporting OAuth signatures on requests to the Data API. HathiTrust also began work on version 2 of the Data API, and tested new features that will support the delivery of PDFs for print-on-demand purposes, and include improved URI syntax to better support new formats such as JATS XML for mPach.

Full-text Search

Staff at the University of Michigan conducted a series of tests to gather technical requirements for an RFP for a new high-performance storage system to improve the response time of full-text search, increase the volume of searches the system can handle, and accommodate the extra load that new relevance ranking features would introduce. The tests resulted in specific numerical requirements that were incorporated as minimum specifications into the RFP, which was completed and released to ten suppliers in October, with proposals due back in early November. Evaluation and final pricing negotiation is expected to continue through November and December, with system installation to take place in early 2013.

Michigan staff made changes to full-text search, as well as the HathiTrust bibliographic catalog, to improve faceting on the Author field for works with multiple authors.

Staff continued research geared toward improving relevance ranking and indexing of works in Chinese, Japanese, and Korean.

Imgsrv

Imgsrv is the web application that serves derivatives of HathiTrust’s master images to Web applications such as the PageTurner. HathiTrust made changes to the way Imgsrv constructs PDFs for download to optimize for size. When possible, the original JP2 and TIFF images stored in the repository are included in the PDF. If there is a risk that the final PDF will be over 2GB, a lower resolution derivative is extracted from JP2 images and compressed as a JP2; TIFF images are scaled down and compressed as JPEGs.

PageTurner

In conjunction with recommendations from the UX Advisory Group, the default view in HathiTrust was changed to “scroll” view. HathiTrust also improved processes for caching images and made modifications to the landing page for the limited (search-only) works.

Website Redesign

Over the last several months, University of Michigan UX department staff have been working on new designs for the HathiTrust home page and application interfaces. In October, developers at Michigan began to explore options for a consolidated framework of Cascading Style Sheets (CSS) across HathiTrust applications.

Outages

No outages were reported in October.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of October 1:

Public Domain (~30%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Papers and Presentations

• Jeremy York “Update on Developments and Activities [148]”, University of Michigan Selectors Meeting, October 9, 2012.
• Jeremy York, “HathiTrust [149] Organization, Governance, and Costs [150]”, University of Michigan School of Information, October 9, 2012.
• Angelina Zaytsev, “Using HathiTrust in Research and Education [151]”, Open Access Week, University of Michigan, October 23, 2012.

See http://www.hathitrust.org/papers [110] for all papers, presentations, and reports.

November Forecast

• Continue work on indexing of CJK languages and relevance ranking for full-text search.
• Continue exploration of CSS frameworks for the website redesign.

Update on September 2012 Activities

October 12, 2012 [1]

[Download PDF] [152]

Late Breaking News

Case Closed - HathiTrust is Fair Use

In a decision that will have broad repercussions across libraries, on October 10, 2012 Judge Baer dismissed the lawsuit filed just over a year ago by the Authors Guild et al. against HathiTrust and several participating libraries. HathiTrust has released an official statement [86] on the ruling. Information [81] about the lawsuit, as well as relevant analysis and reactions [153] from around the Web are available on the HathiTrust website.

Top News

HathiTrust Research Center UnCamp

The HathiTrust Research Center held its first annual “UnCamp” in Bloomington, IN on September 10-11. 130 researchers, developers, and librarians from HathiTrust member and non-member institutions gathered in Indiana University’s new CyberInfrastructure Building for presentations, demos, and hands-on sessions with the emerging Research Center tools. These included tools both to perform research on the HathiTrust corpus and to create new or customized algorithms and processes for research. Responses to the UnCamp have been very enthusiastic, giving energy to efforts to enable computational access to the incredible body of works in HathiTrust. More information on the UnCamp, including presentations, resources, reactions and responses via tweets, and more can be found on the HathiTrust Research Center Wiki [93]. See the press release [154] also from the University of Illinois.

Government Documents Registry

HathiTrust has initiated a project to build a comprehensive registry of U.S. federal government documents. The Registry is an emerging effort in a broader undertaking by HathiTrust partners to improve access to U.S. federal government documents. Further information and background on the project is available on the Registry project page [34]. A two-year term Government Documents Registry Analyst position [155] for the project was posted in September.

Infrastructure Changes for Out of Print and Brittle

In the coming weeks, HathiTrust will begin making infrastructural changes to incorporate information about the holdings status and condition of volumes at partner institutions into access services. The changes will apply in particular to access on library premises to in-copyright works that fall under Section 108 provisions of the U.S. Copyright Act. One of the infrastructural changes will be altering the semantics of the “out-of-print and brittle” (“opb”) designation in HathiTrust’s rights database to “out-of-print” (“op”) only. This change will be made on November 1, 2012, and will be reflected in HathiTrust interfaces, and services such as the Hathifiles where rights information is made available.

Ingest

Internet Archive Digitization

HathiTrust coordinated with the University of Florida on upcoming deposit of volumes, and ingested a new batch of volumes from Penn State.

Local Digitization

HathiTrust ingested a new set of volumes from Utah State University Press and began conversations with the University of Delaware about processes and requirements for deposit of locally-digitized content. HathiTrust also corresponded with the University of Iowa about use of the new tools [54] for validating and packaging locally-digitized materials for deposit. Institutions with questions about the new tools should contact feedback@issues.hathitrust.org [66].

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

Communications Working Group

The Communications Working Group continued to follow developments in HathiTrust governance, and to evaluate how the communications function in HathiTrust might be improved once the new governance structure is in place. The survey [156] for HathiTrust training and information sessions has closed, and HathiTrust will use the results as a basis for upcoming informational events. If you did not have a chance to submit feedback and would like to, please email responses to the survey to feedback@issues.hathitrust.org [66].

User Experience Advisory Group

The User Experience Advisory Group continued discussions about a new home page design and provided feedback on mockups created by the University of Michigan.

User Support Working Group

A summary of issues received by the User Support Working Group is given in the table at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) is in the final phase of development to bring Zephir into parity with the existing bibliographic management system at the University of Michigan. Once Zephir is in operation, institutions will submit bibliographic records for volumes they plan to deposit to Zephir, and Zephir will produce exports of bibliographic data that will be used in HathiTrust Web services. In October, as part of preparations for integration testing with HathiTrust systems, CDL staff will begin preliminary testing of the Zephir outputs to evaluate system performance and confirm the structure of outputs (that they have the correct metadata fields, etc.). CDL has been contacting institutions that are contributing records to HathiTrust on an ongoing-basis to test the process for submitting bibliographic records to Zephir. If your institution is not contributing content to HathiTrust currently but you would like to test the new submission process, please contact feedback@issues.hathitrust.org [66].

Copyright Review

A summary of copyright review activities in September is given below.

IMLS Quality Grant

The project team finalized a catalog of commonly-seen illustration errors in HathiTrust volumes for a sub-study on illustrative error. Donald Williams, a renowned research imaging scientist, analyzed the errors and met with members of the project team to explain the sources of the errors and possibilities for correction.

The project team continued work on the design of user studies to evaluate project findings, collection of data to support the user studies, and administration of the user studies themselves. Team members also discussed ways that quality review interfaces developed during the grant might be modified to support the certification of individual volumes. For more information on the project, please visit the project website [138].

mPach

Staff at the University of Michigan created a mockup of PageTurner changes that will be needed to navigate the XML-based  journal articles that will be submitted via mPach. Work also continued on modifications to PageTurner to display JATS XML and embedded media and on refinements to the METS specification for mPach Submission Information Packages. Staff completed wireframes and began coding the Dashboard module (see the list of mPach modules [101] for more information). Michigan staff members will present on mPach at the 2012 DLF Forum.

Development Updates

Accessibility

HathiTrust has completed the first phase of improvements to enhance accessibility of HathiTrust Web applications. With a few minor exceptions that will be addressed in the second phase, HathiTrust interfaces are now compliant with Web Content Accessibility Guidelines (WCAG) 2.0 [89], Level A. The second phase will target compliance with WCAG 2.0 Level AA and include usability testing by users who have print disabilities.

Data API

As of October 1, all requests to the Data API must be signed with an access key provided by HathiTrust. Details are available at http://www.hathitrust.org/data_api [103].

The Data API is being configured to deliver watermarked image derivatives in JPEG and PNG formats at a range of resolutions. The API currently delivers un-watermarked master images from the repository in TIFF and JP2.  Enhancements to the Data API Web client were made to support image derivatives when they become available through Data API, and development-level debugging.

Full-text Search

University of Michigan staff modified the full-text search indexing process to prevent volumes from being indexed on more than one shard (section) of the full-text Solr index. Staff also began testing full-text search using Solr 4.0 Beta. Solr 4.0 offers new ranking algorithms that may provide better relevance ranking for long documents (e.g., books). A paper [69] by Michigan developer Tom Burton-West on full-text search relevance ranking in HathiTrust was published in the INEX 2012 pre-proceedings as part of the CLEF Labs Working Notes [157].

Following several months of informal research, Michigan staff began focused investigation into high-performance storage systems to improve full-text search response time and substantially increase search throughput capacity. An RFP for a new high-performance storage system will be issued in October.

Imgsrv

Imgsrv is the web application that serves derivatives of HathiTrust’s master images to Web applications such as the PageTurner. HathiTrust has enhanced Imgsrv to deliver HTML derivatives of born-digital content in support  of mPach and JATS XML.

PageTurner

HathiTrust implemented interface improvements designed by Michigan’s User Experience department for cases where special access to HathiTrust materials is available, such as access by users who have print disabilities. The improvements include dismissible notifications when special access is in effect, and updated explanatory text when special access that might be expected is not available (special access cases are described in HathiTrust’s Access and Use Policies [158]). Special access is currently only available as a pilot at the University of Michigan. Extension of special access to other member institutions is still planned. More information will be forthcoming.

HathiTrust’s embeddable Pageturner is now based on the mobile Pageturner interface, which offers improved presentation and greater functionality.

HathiTrust has updated the version information displayed in the PageTurner to include the time a volume a was removed from HathiTrust. Volumes may be removed from HathiTrust at the request of the rights holder, or in cases where the volume is wholly unusable or a superior copy is available.

Outages

From 1:00pm on Tuesday, September 25 to 8:30am on Friday, September 28, some bibliographic data failed to display in HathiTrust due to an outage of the system at Michigan that manages bibliographic data for HathiTrust.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of September 1:

Public Domain (~30%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Papers and Presentations

Tom Burton-West, "Practical Relevance Ranking for 10 Million Books [69]", INEX 2012 pre-proceedings, CLEF Labs Working Notes, September 2012.

HathiTrust UnCamp presentations and resources [93] (via HathiTrust Research Center Wiki), September 10-11, 2012.

Heather Christenson and John Wilkin, "Intellectual Property Rights and the HathiTrust Collection" (forthcoming), UNESCO - The Memory of the World in the Digital Age: Digitization and Preservation, September 26, 2012.

Jeremy York, "A Preservation Infrastructure Built to Last: Preservation, Community, and HathiTrust [159]", UNESCO - The Memory of the World in the Digital Age: Digitization and Preservation, September 26, 2012.

See http://www.hathitrust.org/papers [110] for all papers, presentations, and reports.

Update on August 2012 Activities

September 14, 2012 [1]

[Download PDF [160]]

Top News

HathiTrust Training and Information Sessions

In the Update on July Activities [161] we distributed a short survey to receive feedback on our next series of HathiTrust information and training sessions. We have received many responses. The deadline for completing the survey is September 21. If you have not already, please take a moment to provide input on the kinds of sessions you would like to attend or lead, and the form you would prefer these sessions to take (e.g., a webinar series, in-person meeting, or a combination of the two). The survey is available at http://tinyurl.com/8n3k9nr [162].

Data API Changes In Effect October 1

Beginning October 1, all requests to the Data API will need to be signed with an access key provided by HathiTrust. Access keys for programmatic uses of the Data API can be obtained at http://babel.hathitrust.org/cgi/kgs/request [163]. HathiTrust has also created a Web client [164] that employs a user’s login credentials as a proxy for an access key to facilitate non-programmatic uses. Complete documentation of the security enhancements, methods of obtaining keys, how to sign requests, and how to access the Web client is available at http://www.hathitrust.org/data_api [103].

Also effective October 1, the host “services.hathitrust.org” will no longer exist for the Data API. The new host will be “babel.hathitrust.org”, the same host as the PageTurner and other HathiTrust services. Calls to the Data API will therefore need to use URLs such as the following (note the additional “cgi” in the path):

http://babel.hathitrust.org/cgi/htd/meta/mdp.39015019203879 [165]

rather than

http://services.hathitrust.org/htd/meta/mdp.39015019203879 [166]

Shibboleth Library Walk-in

Later this year, HathiTrust will begin accepting the “library-walk-in” Shibboleth attribute from partner institutions to provide certain member privileges to guest users who do not have an institutional login. For instance, “Library-walk-in” users will have the ability to download full-PDFs of all public domain materials in HathiTrust. Partners who wish to use HathiTrust library-walk-in functionality must confirm in writing that they are asserting the library-walk-in affiliation only for users physically present in a library building at the time of session initiation. Please see Shibboleth Login [107] for more information about Shibboleth in HathiTrust.

Ingest

Internet Archive Digitization

HathiTrust ingested nearly all of a set of approximately 2,000 volumes from Boston College, and loaded bibliographic records for additional volumes that will be deposited by the University of Illinois. The University of Florida submitted sample bibliographic records to be analyzed in preparation for content ingest.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

Communications Working Group

The Communications Working Group did not meet in August, taking its first break since the group’s formation in May 2010. As the group awaits the solidification of the new HathiTrust governance, group members plan to address the results of their survey on training [162], and look ahead to fall activities and meetings.

User Experience Advisory Group

The User Experience Advisory Group continued discussions about a new home page design and provided feedback on mockups created by the University of Michigan.

User Support Working Group

A summary of the issues received by the User Support Working Group is provided at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) and University of Michigan staff agreed on a data workflow for updating rights information in the HathiTrust rights database when CDL takes responsibility for managing HathiTrust bibliographic data. The CDL team is refining and improving the performance of bibliographic data exports needed to support HathiTrust operations. Analysis continued to address issues with a small percentage of poor quality records.

Michigan staff successfully tested the bibliographic record submission process for Zephir (the new management system) and commented on corresponding submission guidelines. In the coming month CDL will be contacting institutions that are currently, or were in the past, contributors of content to HathiTrust to test the new process for submitting records. The test will be aimed primarily at current content contributors, but all contributors will be invited. Please contact feedback@issues.hathitrust.org [66] if your institution is not contributing content currently but you would like to test.

Copyright Review

A summary of copyright review activities in August is given below. For further information on these activities please see CRMS-US [167] and CRMS-World [14].

HathiTrust Research Center

The HTRC made preparations for its first “UnCamp”, held in Bloomington, Indiana on September 10-11. A full report on the gathering will be forthcoming.

IMLS Quality Grant

Project staff continued work to finalize the quality review datasets. This included reviewing datasets for completeness, accuracy, and missing data, and performing reliability and validation testing on data for volumes that were double-coded for quality assurance purposes.

The IMLS grant advisory board met for its second time in mid-August. The project team presented its findings to-date and advisory board members provided input on work to be completed in the final stages of the project, as well as on research directions in the future. Over the next several months the project team will focus on completing the design of user studies to further investigate quality in relation to the usefulness of digitized volumes, collecting data to support the user studies, and conducting the user studies themselves.

Efforts continue to develop a framework for certifying the quality of volumes in HathiTrust. This includes the development of a modified data collection Web interface based on the interfaces used in the grant thus far.

For more information on the project, please visit the project website [120].

mPach

The mPach team at the University of Michigan updated the project timeline on the HathiTrust project page [37]. Work continued on modifications to the HathiTrust PageTurner to display JATS XML, and on refinements to the METS specification for mPach Submission Information Packages. Michigan staff made progress on enhancements to the Norm tool (part of content preparation), specifically enhancements to normalize bulleted lists, figures with captions, and tables. Wireframes are nearly complete for the Dashboard module (see the list of mPach modules [101] for more information on mPach modules). Michigan staff will be presenting on mPach at the 2012 DLF Forum.

Development Updates

Accessibility

Staff at the University of Michigan continued work to improve general accessibility for HathiTrust Web applications.

Data API

Michigan staff extended functionality of the Data API to serve full PDFs of volumes for print-on-demand services on Espresso Book Machines (EBM) via the ExpressNet sales network. Staff also augmented Data API usage monitoring to explicitly track signed requests, and made enhancements that will enable the Data API to deliver dynamically-generated image derivatives (such as PNG images as opposed to TIFF or JP2 images).

First Full Repository Upgrade

Development and testing for the metadata upgrade reported in the Update on July Actvities [168] has been completed, and the upgrade will begin in October.

Full-text Search

Michigan staff continued to investigate the Solr edismax parser bug that is preventing CJK searching from working properly.  Staff confirmed that the bug also affects Solr 4.0 and submitted sample documents and queries demonstrating the problem to the Solr JIRA issue tracking system: see https://issues.apache.org/jira/browse/SOLR-3589 [140].  Staff investigated possible workarounds for this issue, and conducted e-mail discussions with several Blacklight developers who are working on CJK issues.

Staff also made changes to the automated full-text search indexing process so that failures caused by server errors are automatically re-queued.

The INEX (Initiative for the Evaluation of XML Retrieval) Book Track accepted a paper by Michigan developer Tom Burton-West on full-text search relevance ranking in HathiTrust. The paper will be published in the INEX 2012 Pre-proceedings as part of the CLEF Labs Working Notes.

PageTurner

Michigan staff made changes that will make it easier to support new formats in the PageTurner interface. The mPach project will make use of the changes to add support for JATS XML.

Outages

HathiTrust was unavailable on Monday, August 13 from 7:30-8am EDT for a security-related database reorganization.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of August 1:

Public Domain (~30%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Update on July 2012 Activities

August 10, 2012 [1]

[Download PDF [169]]

Top News

HathiTrust Information Sessions and Training Survey

HathiTrust has offered webinars in the past to orient new partners and provide updates on HathiTrust services and initiatives. For our next series, or program, we are considering offering information sessions led by staff throughout the partnership on topics of interest. To begin to plan for these sessions, we would like to receive feedback from members of partner institutions on 4 questions related to session topics, venue, and participation. A form with the questions is available at http://tinyurl.com/8n3k9nr [162]. Although feedback is especially sought from partner institutions, others may provide input. If there is sufficient interest we will consider offering sessions open to anyone, whether affiliated with HathiTrust partner institutions or not. Responses are requested by September 21, 2012.

HathiTrust Accessibility

University of Michigan developers, under the guidance of Michigan’s User Experience Department, are in the process of reviewing and making improvements to the accessibility of HathiTrust Web applications. The first phase of the work, which began in July, involves ensuring compliance of HathiTrust interfaces with the Web Content Accessibility Guidelines (WCAG) 2.0 [89], Level A. The second phase will target compliance with WCAG 2.0 Level AA, and include usability testing by users who have print disabilities. It is expected as part of this work that Michigan staff will begin to draft policies and guidelines to ensure that future coding for HathiTrust applications maintains these standards.

Staff from partner institutions with Web accessibility expertise who are interested in being involved with this initiative are encouraged to contact Suzanne Chapman (suzchap@umich.edu [170]).

Copyright Review Update

Over the last several of years, staff from HathiTrust partner institutions have been manually reviewing the copyright status of volumes published in the United States from 1923 to 1963, as part of CRMS-US, an IMLS funded grant project. The grant has come to an end but review by partner institutions continues, and has been expanded through a second IMLS grant, CRMS-World, that is targeting review of non-US-published works, beginning with English-language works published in the United Kingdom, Canada, Australia, and Spain. Through this process tens of thousands of works have been discovered to be in the public domain and opened in HathiTrust to viewers world-wide. Reports on the number of volumes reviewed and opened as of early August are shown below and will be included in future updates.

Ingest

Local Digitization

Michigan staff provided support as needed for the new ingest tools [54] that have been made available. Staff who have questions about using the tools, or who would like to initiate deposit of materials should contact feedback@issues.hathitrust.org [66].

Internet Archive

HathiTrust began ingest of a first set of Internet Archive-digitized volumes from Penn State and a second set of Internet Archive-digitized volumes from the University of Illinois.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_groups [36] for more information.

Operational

User Support Working Group

A summary of the issues received by the User Support Working Group in July is provided at the end of the update.

Projects

Bibliographic Data Management

California Digital Library (CDL) staff continued working with staff at the University of Michigan to develop processes to sync rights determination information between Zephir and the HathiTrust rights database. CDL also worked with Michigan to test new bibliographic submission guidelines and a new workflow for submitting bibliographic metadata to Zephir via FTPS. HathiTrust members currently depositing content will soon be asked to participate in a test of this new submission process, which will be put in place when the cutover from the bibliographic management system at the University of Michigan to Zephir takes place. Zephir development is in its final phase, and in the coming months Michigan and CDL will move to an integration phase that will involve extensive testing and operation of the two systems in parallel before a final cutover.

HathiTrust Research Center

The HTRC UnCamp is a little over a month away. The UnCamp is part informational, part community building, part boot-camp, and part unconference, designed to show the research capabilities that the HathiTrust Research Center can offer and garner feedback from a broad range of interested users. An exciting list of speakers includes John Wilkin, Executive Director of HathiTrust, Colin Allen, Professor of History and Director of the Cognitive Science Program at Indiana University, and Ted Underwood, Associate Professor of English at the University of Illinois. Details on the UnCamp can be found at http://d2i.indiana.edu/htrc/uncamp2012/ [171].

The UnCamp marks the 12-month point in the development of the HTRC, and contributes to a milestone set out in the MOU between the HTRC and HathiTrust of a demonstration of HTRC functionality 12 months into development. The HTRC is scheduled to transition into production in Spring 2013, at the 18-month mark from its inception, so the UnCamp is timely for gathering community input.

The HathiTrust Research Center is pleased to have recently received two allocation awards from XSEDE [172]for computational resources: one for exploratory research and the other in support of educational and outreach activities.

IMLS Quality Grant

Project staff focused work in July on finalizing the quality review datasets assembled over the course of the grant for analysis, and on developing a framework for reviewing and certifying the quality of volumes in HathiTrust. Staff also finished assembling a catalog of frequently-observed errors in illustrative content for in-depth analysis by an expert in digital conversion errors. The second meeting of the grant Advisory Board and project collaborators is scheduled for the end of August. The project team will present findings at the meeting and solicit feedback and input on direction from the attendees.

mPach

University of Michigan staff completed a diagram [100] of the mPach system architecture, which has been added to the mPach website [173]. Staff also refined the schema for mapping bibliographic data from JATS XML (the format to be used for encoded text) to MARC records, which are required for ingest into HathiTrust. Work continued on wireframes for the Dashboard module (see a description of all modules [101]), on the profile for the METS file that will accompany digital objects in Submission Information Packages, and on rendering XML articles in the HathiTrust PageTurner.

Development Updates

Collection Builder

University of Michigan staff made changes to the Collection Builder Web application to allow longer collection titles and descriptions, and notify users when entries exceed the allowed lengths.

Data API

Michigan staff worked to update documentation of the Data API to reflect changes reported in the Update on April Activities [174], as well as other recent changes. The documentation will be made available in August. Documentation of the interactions of existing clients with the Data API is also being developed. Michigan staff have begun planning to extend the Data API to support requests for derivative forms of content including scaled, individual, page images and PDFs. These features will initially serve quality review and print-on-demand applications but are expected to have other uses as well. Access will be subject to Data API authorization requirements.

Full-text Search

Programmers at the California Digital Library added language-aware relevance ranking to the search spelling suggestion feature under development for full-text search. Staff also built a regression framework for testing algorithm changes and began to make changes using heuristics to improve spelling suggestion quality for a test set of 100 HathiTrust queries. Over the coming weeks, additional changes to the scoring system for suggestions are expected to further improve the quality of suggestion results.

Michigan staff completed the second phase of planned improvements to indexing and searching of Chinese, Japanese, and Korean (CJK) languages in HathiTrust. The second phase involved re-indexing all volumes in the repository with a new schema to provide better searching over bibliographic data for CJK materials. Staff made corresponding changes in HathiTrust Web application code to take advantage of the new schema. Staff observed that the bug discovered in the first phase of work (reported in the Update on June 2012 Activities [139]) caused the level of improvement to be less than expected in the second phase as well. Work will continue to address this issue. In the meantime, the improvements made to full-text indexing in June reduced the time needed to index all 10.4 million volumes by nearly half – to approximately one week – despite the fact that there were interruptions.

First Full-Repository Metadata Upgrade

University of Michigan staff made preparations to begin the first repository-wide upgrade of metadata for HathiTrust objects. The upgrade applies primarily to PREMIS metadata, though metadata in other areas of the HathiTrust METS file will be affected as well. In conjunction with this upgrade, HathiTrust will begin moving toward a formalized model of publicly communicating planned full-repository changes.

Process for Item Deletion

Deletions from the HathiTrust repository are rare, occurring in instances where

• A volume is either wholly unusable due to quality problems, or a superior copy of the volume is available in HathiTrust (such deletions are authorized by the depositing institution) or
• Removal is requested by the rights holder.

Michigan staff are in the final stages of implementing an automated process (though the process must be initiated manually) to remove items from repository storage, as well as catalog and full-text indexes. In cases where volumes are deleted, a “tombstone” is created for provenance purposes and to maintain permanent links for references users may have created.

Outages

HathiTrust may have been unavailable for some users on Monday, July 16 from 8:15pm to 8:30pm due to a database locking problem at one repository instance, and from Monday, July 16 at 11:45pm to Tuesday, July 17 at 8:15am due to a problem with a web server at one instance.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

Presentations

Jeremy York: HathiTrust And TRAC [175]. Digital Preservation 2012 (annual meeting of the National Digital Information Infrastructure and Preservation Program and the National Digital Stewardship Alliance), Washington, D.C., July 25, 2012).

Jeremy York: HathiTrust: On TRAC [176]. ICPSR course on Applied Data Science, University of Michigan, July 26, 2012.

See http://www.hathitrust.org/papers [110] for all papers, presentations, and reports.

New Growth

As of July 1:

Public Domain (~30%)

* Includes volumes opened through copyright review and rights holder permissions

Summary of Issues Received by User Support

Update on June 2012 Activities

July 13, 2012 [1]

[Download PDF [177]]

Top News

Update to Bibliographic Ingest Specifications and Checklist

HathiTrust has updated its bibliographic metadata specifications and minimum bibliographic metadata requirements in preparation for moving to Zephir (under development by California Digital Library) as the bibliographic metadata management system for HathiTrust. The requirements are in effect immediately for institutions that have not previously deposited content in HathiTrust. Institutions that have already deposited content are requested to meet the minimum requirements, but it is not required (we will continue to accept bibliographic metadata as it has been submitted). The primary difference from previous requirements is that records from new depositors must at a minimum include a MARC Leader, 008 field, title field, and OCLC number in order to be loaded. Further details are provided at http://www.hathitrust.org/bib_specifications [92]. The HathiTrust Ingest Checklist [178] page has been revised in conjunction with these changes.

Tools to Support Ingest of Locally-digitized Materials

The University of Michigan made the first iteration of tools available to aid institutions in transforming, validating, and packaging digital content for deposit in HathiTrust. The tools can be downloaded at http://www.hathitrust.org/ingest_tools [54]. Notifications of updated versions of the tools will be sent to a Google Groups email list [67], and we recommend anyone who will be using the tools to subscribe.

New HathiTrust Project Librarian

by Jeremy York

I am pleased to announce the appointment of Angelina Zaytsev to the position of HathiTrust Project Librarian. Angelina has worked for HathiTrust part-time for the last year, assisting in the coordination of numerous HathiTrust activities including ingest of content from partner institutions, processing permissions to open access to materials, general user support, and multiple duties “as assigned”. Angelina will continue these duties while taking on a greater role in managing and coordinating projects for HathiTrust.

HathiTrust Research Center (HTRC)

The HathiTrust Research Center opened registration [179] for the HTRC UnCamp, to be held in Bloomington, Indiana on September 10-11, 2012. More information can be found at http://www.hathitrust.org/htrc_uncamp2012 [180].

New Copyright Status (IC-US)

Michigan staff completed the majority of development necessary to support a new rights status in HathiTrust Web applications. The status will apply to works that were restored to being in copyright in the United States by the General Agreement on Tariffs and Trade (GATT), but are now in the public domain in the rest of the world. An increasing number of these volumes are being identified as part of CRMS-World [14], the IMLS-funded continuation of the CRMS project [181].

Ingest

Internet Archive

HathiTrust continued working with Boston College and began working with Penn State and the University of Illinois on ingest of volumes digitized by the Internet Archive.

Working Groups and Committees

Working groups and committees in HathiTrust may have an operational or strategic focus. See http://www.hathitrust.org/working_ groups [36] for more information.

Operational

User Experience Advisory Group

The User Experience Advisory Group continued discussions about a new home page design and provided feedback on mockups created by the University of Michigan.

User Support Working Group

The table below shows a summary of the issues received by the User Support Working Group in June.

Projects

Bibliographic Data Management

California Digital Library (CDL) staff completed a portion of development needed to sync Zephir with rights information in the HathiTrust rights database. Staff also reloaded records for HathiTrust items into Zephir as part of an iterative process to be sure rights and other necessary administrative metadata are being properly loaded. CDL staff completed documentation of Zephir metadata ingest and workflow guidelines and will be working with HathiTrust project staff to add the information to the HathiTrust website as the launch of Zephir gets nearer.

mPach

University of Michigan staff rewrote the list of modules [101]to be included in mPach, a package of tools Michigan is developing for publishing open access journal content in HathiTrust. Staff divided modules into three categories: Editorial Workflow and Peer Review, Content Preparation, and HathiTrust. Work continues to adapt PageTurner to handle full-text XML content and to develop wireframes for the Dashboard module. Staff began developing code to validate the Submission Information Package.

IMLS Quality Grant

Project staff completed review of the 4th and final 1,000-volume sample, consisting of non-Roman language materials. This concluded the data collection phase of the project. The project team finalized plans for a specialized study of errors in digitized illustrations and began to assemble and review select illustrations from each of the Library of Congress classifications. The study is designed to be an in-depth investigation into errors in digitized illustrations; it is not meant to describe or characterize the extent of errors in HathiTrust as a whole.

Jackie Bronicki presented current findings of the project at the ALA Annual Meeting in June. The new interface for the project website, which has been undergoing a redesign in the last couple of months, was released at the same time.

The project team also made progress on a framework for certifying the quality of volumes in HathiTrust.

Development Updates

Full-text Search

Staff at Michigan completed the first phase of work to improve indexing and searching of CJK (Chinese, Japanese, and Korean) languages. The first phase involved re-indexing all 10.4 million volumes in the repository using the new CJKBigramFilter available in Solr 3.6, and a custom unigram filter. The new index was put into production in mid-June. The improvements in search precision for CJK queries turned out to be smaller than anticipated. Investigation revealed the cause to be a bug in Solr’s edismax query processor, and a bug report [140] was filed in the Solr JIRA bug-tracking system. Michigan staff are investigating both temporary workarounds and a long-term fix to the bug.

Michigan staff indexed the INEX Book Track corpus [182] and conducted a series of relevance ranking experiments. From the experiments, 6 “runs” were chosen and submitted to the INEX Book Track “Prove It” task. Three of the runs were designed to simulate users searching the HathiTrust full-text index and three were baseline runs to measure the impact of the default HathiTrust search configuration on queries of different types and lengths. The results from the INEX Book Track will be used to tune relevance ranking in HathiTrust’s repository-wide and single-volume, full-text search.

California Digital Library refined the algorithm used to score spelling suggestions based on queries extracted from HathiTrust log files and improved the way suggestions are made when stop words and words that are inappropriately combined are present in the query. The next step will be to experiment with making suggestions in different languages.

Michigan removed a long-standing bottleneck in the full-text indexing process, effectively doubling throughput. Under ideal conditions, staff believe it should be possible now to index approximately 100,000 documents per hour.

Outages

No outages were reported in June. HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of July 1:

Public Domain (~29%)

* Includes volumes opened through copyright review and rights holder permissions

Presentations

• Jon Rothman: HathiTrust Supplemental Metadata [183]. ALCTS Holdings Information Forum, ALA 2012 Annual Convention, June 23, 2012.

Partner-specific:

• Jackie Bronicki: Validating the Quality of Digitized Books for Long-Term Preservation: Preliminary Findings on Digitization Error. Preservation Administrators Interest Group, ALA 2012 Annual Convention, June 23, 2012 (slides pending).

See http://www.hathitrust.org/papers [110] for all papers, presentations, and reports.

2012 Mid-Year Review

June 25, 2012  [1]

http://babel.hathitrust.org/cgi/htd/meta/mdp.39015019203879

http://services.hathitrust.org/htd/meta/mdp.39015019203879

http://services.hathitrust.org/api/htd/pathinfo-arguments

http://services.hathitrust.org/htd/pathinfo-arguments

Update on May 2012 Activities

June 8, 2012 [1]

[Download PDF [199]]

Late Breaking News

What's In Your Collection?

Read our new blog post [88] about building HathiTrust collections.

Top News

Report on Board of Governors Meeting

The new Board of Governors met in Chicago in conjunction with the ARL membership meeting in May. The group spent some time before the meeting identifying priorities, focusing primarily on the organizational work of the Board. The Board quickly formed an Executive Committee, as stipulated in the Constitutional Convention ballot proposal [200]. The new Executive Committee members include Paul Courant, Carol Diedrichs, Laine Farley, Sarah Michalak and Bob Wolven. Another group chaired by Pat Steele was charged with initiating the process to assemble by-laws. This group will also attend to issues such as the duration of the appointment of the Executive Committee, and expects to conclude its work by the end of November. A third group will be formed to focus on the development of a Charter.

The Board of Governors will meet by teleconference for the next several months, targeting one meeting per month, as the process of developing by-laws moves forward. In these meetings the Board plans to review HathiTrust’s past work, which will include a review of the HathiTrust budget as well as HathiTrust’s committees and working groups.  Although it was not able to discuss HathiTrust’s existing committees and working groups in detail in Chicago, the Board expressed a deep appreciation for the work the Strategic Advisory Board, the Collections Committee, and the current operational working groups and committees have done. The Board asked that the existing groups continue their work (with the Board’s enthusiastic support) until and while a review of committees can take place.

New Resources and Guides

We are pleased to announce the new HathiTrust Resources and Guides [65] page, where we bring together overviews, instructional materials, and guides created by HathiTrust partner libraries, the Communications Working Group, and beyond. Materials posted on the page include reusable handouts, a detailed guide to using HathiTrust, lively blogs and dynamic videos. Please use, repurpose, and enjoy!

Have you created HathiTrust user guides or instructional materials? We encourage you to submit them to feedback@issues.hathitrust.org [66].

Embed Your Favorite Work

It is now possible to embed HathiTrust volumes in web pages. Code snippets to do this can be found at http://www.hathitrust.org/embed [106].

Ingest

Local Digitization

Staff at the University of Michigan completed development of the first iteration of tools to help depositors create and validate content packages prior to submission to HathiTrust. The tools will be made available in early June to several partner institutions that are working on ingest of locally-digitized materials.

Google

HathiTrust ingested approximately 150,000 additional public domain volumes from Harvard University Library.

Working Groups and Committees

Communications

As noted in Top News, the Communications Working Group released a new Web page [65] featuring HathiTrust instructional materials from across the partnership, including guides developed for public services use.  In addition, the group submitted a briefing to the new Board of Governors with recommendations for carrying out communications activities in the future. The working group also launched a Pinterest account [87] for HathiTrust.

User Experience Advisory Group

The User Experience Advisory Group focused its attention on a project being undertaken by University of Michigan staff to redesign the HathiTrust home page (www.hathitrust.org). The group will begin consulting regularly on this project in June.

User Support Working Group

The table below contains a summary of the issues received by the User Support Working Group in April.

*See User Support Working Group Issue Types [201] for a description of the types of issues included in each category.

Projects

Bibliographic Data Management

Staff at California Digital Library (CDL) refined the code for loading bibliographic records into Zephir (the new bibliographic management system) and reloaded all HathiTrust records in the test environment. Work continued to code a process to sync rights information in Zephir with the HathiTrust rights database. The CDL team is developing documentation and guidelines for submitting bibliographic records to Zephir, and documentation of reports to be provided to institutions when records are loaded.

mPach (formerly jPach)

University of Michigan staff continued work on modifications to the HathiTrust PageTurner to display JATS XML. Staff began development of wireframes for the Dashboard module and are close to the completion of a specification for mapping JATS metadata elements to MARC fields to create analytic records for journal articles.

HathiTrust Research Center (HTRC)

Who should attend? The HTRC UnCamp is targeted to the digital humanities tool developers, researchers and librarians of HathiTrust member institutions, and graduate students. Attendance will be capped at 60 participants, so plan to register early!

Travel funds and Registration. HTRC anticipates funding a small number of travel grants that can be used by an attendee to bring along a graduate student or for a HathiTrust member librarian/technologist to bring along a researcher from their organization who is interested in engaging with our research center. The Uncamp will have a minimal registration fee so as to make the Uncamp as affordable as possible for you to attend.  

IMLS Quality Grant

All of the data collection for English language volumes was completed in May, including double-review of subsets of volumes for quality assurance. Review of volumes in the grant’s final 1,000-volume sample, which includes volumes from 6 major non-Roman languages (Chinese, Japanese, Korean, Arabic, Cyrillic and Hebrew), is still in progress. At the end of May, staff had reviewed 77,115 of the total 95,086 pages sampled for review. Data collection is expected to be complete in mid-June.

Work in June will focus on analysis of the collected data, as well as research, development, and data collection for use case studies, which will comprise the final portion of the grant. Staff will also undertake a specialized study of errors in digitized illustrations to try to more accurately describe the types of errors that are observed and their impact on use.

Current findings of the project will be presented at the ALA Annual Meeting in June 2012. The project website [120] is being updated with a new graphic design; further initial findings [138] will be forthcoming. Please see the website for details on the volumes samples, error models, and other grant activities.

Development Updates

Data API

Staff at the University of Michigan made minor changes to the Data API and the Data API’s key service and Web client to better manage user privileges. A Data API security monitoring and reporting script was also deployed that runs on a daily basis.

Full-text Search

Michigan staff undertook work to improve indexing and searching of CJK languages (a discussion of the issues is available on the large-scale search blog [202]). All 10+ million volumes are being re-indexed using the new CJKBigramFilter [203]available in Solr 3.6, and a custom filter that will create a separate unigram index of Han characters (to support queries consisting of a single Han character). Staff revised the Solr indexing schema to eliminate unused fields and filters and to take advantage of upgraded Solr 3.6 filters. Staff also made changes in development to the full-text search and “search within a book” Web applications in preparation for the improved CJK indexing. Testing and production release of the application enhancements and newly-created index are anticipated in early June.

Staff at Michigan downloaded and began to index the INEX Book Track [182] “Prove It” task corpus to use as a testbed to investigate various relevance ranking issues in HathiTrust full-text search.

Staff at California Digital Library (CDL) completed development of fast lookup data structures in the language-sensitive dictionary that will support a spelling-suggestion feature in full-text search (last reported on in the Update on February Activities [204]). Staff used probabilistic techniques to fit the massive dictionary into RAM, allowing very fast lookup of bigram and unigram data. Staff also ported code from the CDL-developed XTF system that ranks spelling suggestions to the new structure, though the code is not yet fully functional. Next steps include modifying the ranking algorithm to take advantage of data from the language-sensitive dictionary, and evaluating and revising the algorithm to produce quality suggestions.

PageTurner

Staff at Michigan deployed fixes to the code that allows users to embed PageTurner views in Web pages using an iframe. Staff also added improved wording and an explanatory link to the PageTurner interface, recommended by the UX Advisory group in April, to clarify when full-PDF download of volumes in HathiTrust is or is not available.

Storage Hardware Replacement Cycle

Michigan staff completed the steps necessary to retire all storage that was scheduled for replacement in 2012. Staff had completed the installation of replacement and additional storage at the Michigan and Indiana sites in March.

Web Hosting Infrastructure Changes

HathiTrust’s VuFind-based bibliographic catalog was successfully moved from University of Michigan Library Web hosting infrastructure to HathiTrust’s Web hosting infrastructure. This completes a migration project that also involved HathiTrust’s Drupal-based informational website and will greatly simplify future Web development.

Outages

Full-text search in HathiTrust was unavailable on Wednesday, May 9 from 6:00-8:30am EDT due to a problem with an index server. Shibboleth authentication to HathiTrust was unavailable on Monday, May 21 from 9:23-9:28am EDT due to a problem with a helper service required by Shibboleth.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of June 1:

Public Domain (~28%)

* Includes volumes opened through copyright review and rights holder permissions

Presentations

• Jeremy York: HathiTrust Overview [205], Michigan Association of Law Libraries Annual Meeting, May 18, 2012.
• Jeremy York: We’re Preserving the Past, What About the Present? [206], NISO Webinar, May 23, 2012.

Partner-specific:

• Brian Vetruba: HathiTrust-- A Govdocs Repository? [207], Regional Government Documents Conference, May 4, 2012.
• Heather Christenson: How HathiTrust Serves the UC Community [208], User's Council, May 21, 2012.

See http://www.hathitrust.org/papers for all papers, presentations, and reports.

June Forecast

• Rebuild the Large Scale Search Solr/Lucene index with CJK (Chinese, Japanese, Korean) indexing improvements.
• Distribute first iteration of tools to aid in preparing content for ingest into HathiTrust.

You can follow HathiTrust on Twitter [198] or subscribe to receive the monthly update by email [3] (via Google Groups).

Update on April 2012 Activities

May 11, 2012 [1]

[Download PDF [209]]

Top News

Automatic Partner Login

Staff at the University of Michigan have developed functionality that allows users from partner institutions to be “automatically” logged into HathiTrust when following links from local institutional catalogs or other resources. Permanent links to HathiTrust volumes can now be wrapped with a single sign-on URL that automatically passes users through their own institution’s authentication service. Users who are not already authenticated are prompted to do so. Documentation of the new functionality is available at http://www.hathitrust.org/automatic_login [102]. Thanks to Johns Hopkins University for suggesting this enhancement.

Ingest

Local Digitization

Staff at Michigan continued to work on tools that content depositors can use to create and validate locally-created content packages prior to submission to HathiTrust. The tools will available to partner institutions in May.

Google

HathiTrust began ingest of Google-digitized content from the University of Illinois in April, bringing in more than  80,000 volumes.

Working Groups and Committees

Communications

The Communications Working Group continued regular activities and development of a briefing for the new Board of Governors.  New communication initiatives are awaiting the transition to the new Board.

User Experience Advisory Group

The UX Advisory Group revisited issues related to the labeling of PDF download options in PageTurner. The group’s recommended changes aim to clarify when full PDF downloads are or are not available. The changes are under development and will be implemented in May.

User Support Working Group

The table below contains a summary of the issues received by the User Support Working Group in April.

*See User Support Working Group Issue Types [201] for a description of the types of issues included in each category.

Projects

Bibliographic Data Management

California Digital Library created prototype exports of the metadata that will be used to populate HathiTrust’s tab-delimited inventory files (“hathifiles [4]”) and bibliographic catalog. Timing tests for these exports were also conducted. The CDL team continued to reconcile bibliographic records in Zephir with records in the current system at the University of Michigan to ensure all the data is accounted for, addressing record discrepancies and ingest errors as encountered. The team has also begun development of a process to sync rights information in Zephir (the new management system) with the HathiTrust rights database.

jPach

University of Michigan staff continued work on modifications to the HathiTrust PageTurner to display JATS XML. jPach’s Norm module (see descriptions of all jPach modules [210]) can now extract 15 common components of a journal article, plus embedded media, from a DOCX file and create valid JATS with references to associated media files. A specification for a Submission Information Package for jPach content is nearly complete and will be posted to the jPach website [192] soon. Work has begun on developing wireframes for the Dashboard module. A timeline for the project is available on the HathiTrust jPach project page [211].

HathiTrust Research Center (HTRC)

The HTRC completed the agreements necessary with Google to receive Google-digitized public domain volumes from the HathiTrust repository and make them available for computational purposes. With the Google agreements and a Memo of Understanding with HathiTrust in place, the HTRC is actively working with staff at Michigan to bring in the complete set of more than 2.9 million public domain volumes in HathiTrust. Preparation for the transfer includes setup of disk storage and compute nodes at Indiana University (IU), which is  being done in collaboration with IU Research Technologies. All computation on HathiTrust volumes will becarried out on HTRC machines; the HTRC itself will not make content available for download. Users interested in receiving texts should follow the directions at http://www.hathitrust.org/datasets [212].

HTRC was represented at the recent Committee on Institutional Cooperation Digital Humanities Summit in Nebraska. Many attendees were already aware that the HTRC was a digital scholarship initiative of HathiTrust; brochures were on hand to provide a deeper level of detail. 

The HTRC has created Meandre workflow components (Meandre is part of the SEASR [213]infrastructure) that retrieve texts from the HTRC using the HTRC data API, spell-check the texts, correct OCR errors, and then perform topic modeling on the texts. The HTRC has demonstrated this functionality, creating topic models of all pages returned from the data API from single-word queries on a full-text index of volumes. For example, a search for “dickens” in the non-Google digitized public domain corpus returns more than 100 topics with associated keywords. The diagrams below show tag clouds of keywords for the topics “lady” and “men”.

IMLS Quality Grant

Project staff completed whole-volume review of the first 1,000-volume sample (1,000 English language, pre-1923 volumes digitized by Google), and over 70% of the second 1,000-volume sample (1,000 English language, post-1923 volumes digitized by Google). Approximately 150 volumes (15%) from each of the two samples were coded by two reviewers for quality assurance. The project team decided to perform whole-volume review on the same volumes sampled earlier in the project for page-level review in order to allow for comparison and more in-depth analysis of the data, and yield a better understanding of error within the volumes.

As of the end of April, staff had completed page-level review of approximately 50% of the fourth 1,000-volume sample (1,000 non-Roman language volumes including Korean, Chinese, Japanese, Arabic, Hebrew and Cyrillic).

In the months to come, the focus of the project team will shift away from data collection to data analysis and reporting, and use-case studies research. More information about this research is forthcoming. In May, the team will focus on developing a sub-study to better identify and describe errors in illustrative content. The project website has been updated to report initial findings. See the links under “Quality Review” at http://hathitrust-quality.projects.si.umich.edu/results.htm [138].

Development Updates

Data API

University of Michigan staff deployed the security enhancements described in the Update on March 2012 Activities [214], and the Data API now supports the use of 0Auth [215]1.0-signed requests. As outlined in the March update, there will be a transition period, ending October 1, 2012, during which signed access to the Data API will be possible but not required. After October 1, all requests to the Data API will need to be properly signed with an access key provided by HathiTrust. HathiTrust has created a Web client [164] that employs a user’s login credentials as a proxy for these keys to facilitate non-programmatic uses. Complete documentation of the security enhancements, methods of obtaining keys, signing requests, and accessing the Web client is forthcoming.

Also effective October 1, the host “services.hathitrust.org” will no longer exist for the Data API. The new host will be “babel.hathitrust.org”, the same host as the PageTurner and other HathiTrust services. Calls to the Data API will therefore need to use URLs such as the following (note the additional “cgi” in the path):

http://babel.hathitrust.org/cgi/htd/meta/mdp.39015019203879

rather than

http://services.hathitrust.org/htd/meta/mdp.39015019203879

Full-text Search

HathiTrust released the second phase of advanced full-text search functionality in April. Users can now combine up to four different fields connected by the “AND” or “OR” operators. Search parameters are retained when users click on the “Revise this advanced search” on the search results page.  The advanced search interface also allows complex Boolean expressions in the query box, for example:

(dog OR cat) AND (food OR drink) [216]

If a user enters unbalanced parenthesis, quotes or operators, for example

dog OR OR cat [217]

the application strips out the operators and does a default Boolean AND search and provides a message informing the user.

Several bugs in advanced search were also fixed:

• The reset button now actually clears the form.
• Queries with the characters “<,>", or "&” are now handled correctly.
• The words “and” and “or” are now only interpreted as Boolean operators if the query is in lower case or mixed case and the operators (AND|OR) are all upper case.

PageTurner

The HathiTrust PageTurner now displays a version for items in the repository (at the bottom of the left column when viewing an item). The version is the date the item was last updated. Items are updated when improvements such as higher quality or more complete scans have been made.

Web Hosting Infrastructure Changes

HathiTrust’s Drupal-based informational website was successfully moved from Michigan library web hosting infrastructure to the existing dedicated HathiTrust web hosting infrastructure. Work continued on the move of HathiTrust’s VuFind-based bibliographic catalog, which is expected to be completed in early May.

Outages

No outages were reported in April 2012.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of May 1:

Public Domain (~28%)

* Includes volumes opened through copyright review and rights holder permissions

Presentations

Stacy Kowalczyk, [218]DLP [218]: The HathiTrust [218] Research Center: An Overview, April 4, 2012.

Jeremy York, Access Services in the Age of Mass Digitization [219]. IviesPlus Conference, University of Chicago, April 20, 2012.

May Forecast

Rebuild the Large Scale Search Solr/Lucene index with CJK (Chinese, Japanese, Korean) indexing improvements; to be completed in May or June

You can follow HathiTrust on Twitter [198] or subscribe to receive the monthly update by email [3] (via Google Groups).

Update on March 2012 Activities

April 13, 2012 [1]

[Download PDF [220]]

Top News

Board of Governors Elections

HathiTrust has announced the members of its new Board of Governors. The full announcement [85], as well as information about the elections process [186], are available on the HathiTrust website. The composition of the Board, which officially begins work April 16, is as follows:

Representatives appointed from the founding partner institutions:

Representatives elected at-large:

Serving 5-year terms from 2012-2016

Serving 4-year terms from 2012-2015

Serving 3-year terms from 2012-2014

Call for Nominations: User Support Working Group

The User Support Working Group is seeking nominations from partner institutions for up to 4 new members. Nominations should be sent to Jeremy York (jjyork@umich.edu [221]) and include the name, title, and a short description of current job duties. Additional information that might be relevant to participation in the group may be included as well. User Support members are on call at least one day per week and follow up on inquiries throughout the week, requiring between 2-4 hours of work. Staff that participate on the group will

• Gain knowledge about HathiTrust’s user base, typical problems and questions that are raised and how they are resolved.
• Become aware of new ways HathiTrust is being used, and features and functionality that users desire.
• Gain knowledge of HathiTrust organizational and technical infrastructure, and policies and procedures relating to copyright, access, collection development, deposit of materials, and preservation.

The charge for the working group is available at http://www.hathitrust.org/wg_user-support_charge [190].

Data API Modifications

Effective May 1, support for legacy Data API URLs in the following form will be removed:

http://services.hathitrust.org/api/htd/pathinfo-arguments

After May 1, URLs should be submitted according to the current Data API schema [103] without the “api” path element:

http://services.hathitrust.org/htd/pathinfo-arguments

Data API Security 

Over the next several months HathiTrust will be implemeting security enhancments to the Data API. The enhancements will require developers using the API to acquire an OAuth 1.0 [215] access key that identifies them, and a secret key that must be used to “sign” URLs to retrieve HathiTrust resources via the Data API. HathiTrust will also provide a Web client that employ’s a user’s login credentials as a proxy for these keys to facilitate non-programmatic uses. In March, staff at the University of Michigan integrated 2-legged [222] OAuth into the Data API and began to develop  the Data API client. Once OAuth is released, there will be an approximately 6-month transition period, ending October 1, 2012, during which signed access to the Data API will be possible but not required. After October 1, all requests to the Data API will need to be properly signed with an access key retrieved from HathiTrust. Complete documentation of the security enhancements and methods of obtaining keys and accessing the Web client is forthcoming. OAuth is planned for release in April 2012.

Ingest

Local Digitization

University of Michigan staff are preparing tools that will allow partners to build complete ingest packages for materials they wish to deposit in HathiTrust. The tools will include functionality to remediate images and build METS files to HathiTrust specifications, and validate files prior to submission to HathiTrust. Several institutions have agreed to test the tools in the coming months. It is hoped that over time all partners and other entities that contribute content to HathiTrust will use the tools to create their submission packages, thereby distributing the effort needed to ingest materials produced from different sources.

Working Groups and Committees

Collections

The Collections Committee’s report on duplicate volumes [83] in HathiTrust is now available. As described in last month’s update, the report recommends that HathiTrust retain all duplicate copies ingested into the repository for the time being, with periodic reassessment. The Strategic Advisory Board has requested that the Committee make further recommendations about the criteria that should be applied in future assessments and identify the future costs and risks of retaining duplicates in the corpus. The Committee also hopes to finalize its recommendations concerning a process for responding to requests and offers within the next several months.

User Experience Advisory Group

The UX Advisory Group conducted informal usability testing to evaluate the impact of changes proposed to the PageTurner interface to incorporate a volume version (date of last ingest). The group plans to discuss the results and make recommendations on the changes in April, with implementation to follow shortly thereafter.  

User Support Working Group

The table below contains a summary of the issues received by the User Support Working Group in March.

*See User Support Working Group Issue Types [201] for a description of the types of issues included in each category.

Projects

Bibliographic Data Management

California Digital Library achieved a milestone in March, loading all bibliographic records submitted by HathiTrust contributing institutions into the Zephir production environment. The goals of this dry run load were to test the functionality of the new metadata management system (Zephir), to test the production infrastructure, and to compare the production loading time with a previous load on a development server. The metadata management team continued to reconcile bibliographic records in Zephir with those in the current system at the University of Michigan to assure all data was accounted for, addressing record discrepancies and ingest errors as they were encountered. The team also began to verify that bibliographic record collation processes in Zephir resulted in the same records clustering as collation processes at Michigan.

jPach (formerly HathiTrust Publishing)

Staff of the University of Michigan formally named the journal publishing platform Michigan will use in conjunction with HathiTrust: jPach. Design principles and requirements for jPach, plus a description of the platform’s modules, are posted on the University of Michigan Library website [192]. The project page [223] on the HathiTrust website now includes a full project timeline.

Michigan staff continued work to generate valid JATS XML from DOCX files, render JATS XML files in PageTurner, and create a METS profile for the jPach Submission Information Package.

HathiTrust Research Center (HTRC)

The HathiTrust Research Center released a report of its activities [191] over the last 6 months. More information about the Research Center can be found on the HTRC web page [53].

IMLS Quality Grant

Project staff continued whole-volume review of digital volumes in the first production sample (pre-1923 English-language Google-digitized volumes), looking for errors such as missing, duplicate, and out-of-order pages, as well as generally “bad” pages, defined in relation to the severity scale established for page-level review. Staff also continued page-level review of the project’s 4th 1,000-volume sample, consisting of non-Roman language volumes. Physical review of Michigan volumes sampled in the second production run (post-1923 Google-digitized English-language volumes) continued in March. Students have completed review of 543 of the 600 Michigan volumes present in the 1,000-volume sample. Further information about the grant project is available from the project website [120].

Development Updates

Full-text Search

Staff at the University of Michigan completed work on the next iteration of advanced full-text search, which will allow users to build queries with greater Boolean complexity and enhance the ability to revise advanced searches. The new features will be released in early April. Staff made significant progress on plans to improve search results relevance ranking.

Storage Hardware Replacement Cycle

Michigan staff installed new storage at the Indiana and Michigan sites that will both accommodate 2012 volume projections and replace storage scheduled for retirement. Storage due for retirement will be taken offline starting in April.

Web Hosting Infrastructure Changes

Developers and system administrators at Michigan began preparations to move HathiTrust’s Drupal-based informational website and VuFind-based catalog from their initial hosting environments, currently on Michigan library infrastructure, to dedicated HathiTrust hardware, where they will run alongside other HathiTrust applications. This move will simplify application integration.

Outages

No outages were reported in March 2012.

HathiTrust sends notice upon discovery and resolution of unscheduled outages and in advance of scheduled outages and maintenance work that may result in an outage. We welcome and encourage additional recipients for these notices. If your institution is not receiving outage notifications and would like to, please contact feedback@issues.hathitrust.org [66].

New Growth

As of April 1:

Public Domain (~28%)

* Includes volumes opened through copyright review and rights holder permissions

** Corrected 5/11/2012. Previous number included 1,389 images from the Minnesota Digital Library

Papers and Presentations

Presentations

Jeremy York, "HathiTrust: Aspiring to Build the Universal Library [224]". UKSG Annual Conference, March 26, 2012.

Jeremy York, "HathiTrust and the Research Library of the Future [225]". American Antiquarian Society Conference on Needs and Opportunities, March 31, 2012.

April Forecast

• Release user interface enhancements for advanced full-text search
• Continue work on relevance ranking of full-text search results
• Complete work on Data API security

You can follow HathiTrust on Twitter. [226]

HathiTrust Research Center Semi-annual Report (October 1, 2011 - March 31, 2012)

Updates are provided in relation to the milestones listed at HathiTrust Research Center Timeline and Deliverables [227]

Executive Summary

The HathiTrust Research Center (HTRC) had a productive 6 months as it works out core issues in Phase I of its development effort.    Milestone wise, we are looking forward to and planning for a public demonstration of functionality that is tentatively scheduled for June 2012 as is in accordance with the MOU between HathiTrust and HTRC.   Phase II in which HTRC Is operational is scheduled to begin date 01 Jan 2013.   

In a striking accomplishment, HTRC is delighted to report that three legal agreements guiding the Center have been completed at the University level.  The MOU between Hathi Trust and the HathiTrust Research Center has gotten signatures at IU and UIUC and is with University of Michigan. The MOU between IU and UIUC has been fully executed. With the Google Agreement, UIUC and IU have each entered into an agreement with Google separately but the same terms.  The agreements have been signed at the University level and are with Google. 

The philosophy behind the technical infrastructure is to use existing services and cyberinfrastructure as much as possible to reduce development and maintenance costs.  That philosophy manifests itself in a recent evaluation of tools like Blacklight for instance. 

The HTRC cyberinfrastructure is up and running on 4 4-core virtual machines hosted at IU.  We are working out access to more disk space in anticipation of the next steps with execution of the Google agreement.  We have a sandbox set up at UIUC to permit broader internal testing. The sandbox consists of a Cassandra noSQL server v1.0 (for volume store), a Solr index, and v0.1 of the HTRC Data API.  The volumes that are available in the sandboxes are 68,724 volumes of non-Google scanned content. 

1.  Technical Accomplishments 

Data API: The HathiTrust Research Center released a beta version 0.1 of the HTRC Data API. The API is a RESTful API through which the HTRC Solr index and volume store are accessed. It cannot be used to download volumes, but can be used to move data to a location where computation takes place. It can also be used to search the Solr index for a set of volume IDs and pass the volume IDs to a service for access and computation. Access to the API will require OpenID authentication and appropriate authorization. The Data API is installed on two sandbox machines, one at UIUC and another at IU, for internal testing. Both sandbox installations work against a small subset of non-Google scanned volumes. 

Sandbox:  HTRC has set up a sandbox at UIUC that consists a volume store Cassandra repository, corresponding Solr index, and a collection of 68,724 volumes of non-Google scanned content. It supports the Data API v0.1 but without security enabled to encourage testing and exploration. Data Management components: 

• Solr Indexes: HTRC has one 4-core, 16 GB RAM machine at IU to run HTRC Solr indexes. We currently run 1 index each for our test collections, and this will exist while we are pre-Google agreement execution.  The Solr index is accessed through the Data API layer.  The Data API layer limits some access, and does auditing, but otherwise is a pass through to the Solr API. 
• Volume Store: HTRC uses a noSQL data store cluster to hold the volumes of digitized text.   HTRC currently supports a 250,000 volume collection of non-Google digitized content and a 50,000 volume collection of content that IU libraries digitized.  These collections reside in a cluster of 3 4-core, 16 GB RAM machines running v1.0 of Cassandra.  These nodes give volume and page level access to HTRC data through the HTRC Data API.  Each machine has 500 GB of disk, and the volumes are partitioned and replicated across the 3 Cassandra instances.   
• Registry:  IU is running a version of WSO2 Governance Registry [228], where applications are registered prior to running in the non-consumptive framework. The registry is also used as a temporary storage for return results. 

Non-Consumptive Research:  HTRC received funding from the Alfred P. Sloan Foundation for development of secure infrastructure on which to carry out execution of large-scale parallel tasks on copyrighted data using public compute resources such as FutureGrid or resources at NCSA.  The high level design uses a pool of VM images that run in a secure-capsule mode and are deployed onto compute resources.  The team is working on a proof of concept deployment process onto an OpenStack platform using Sigiri [229]. 

Blacklight proof of concept. Searching for data is an important function for digital humanists – finding all of the works with a specific set of concepts, a certain genre, by one or more known authors, and other such criteria is generally the first step in the research process. Rather than developing a new search interface, a time consuming activity, the HTRC technical team determined that Blacklight [30], an open source library catalog search and retrieval system, would be a reasonable choice for the HTRC; Blacklight is designed to support data that is both full text and bibliographic, the exact type of data that the HTRC has; it is built on sorl, the same technology that we already use to index the HTRC data; and Blacklight supports faceted searches, a known need of researchers. We have a test implementation deployed on a shared server at UIUC that we were able to deploy with very few problems. In the next quarter, we expect to use the customization options to configure the look and feel of the interface and perhaps to extend the functionality to show snippets of the text to help researchers refine their results. Any new functionality that we develop will be shared with the larger Blacklight community. We expect Blacklight to be a significant component of the public face of the HTRC. 

OCR error detection study: Members of the HTRC undertook a study recently on quantifying OCR errors in the HathiTrust corpus. Scholars are interested in doing quality text analysis, but results can be confounded by OCR errors. Information on which books (or pages) in the collection have significant rates of OCR errors could help. The HTRC explored a couple of approaches to OCR error detection and have results for one approach that uses machine-generated and expert-evaluated rules. Starting with a large dictionary of correctly spelled words, HTRC members identified outlier words that were in the HathiTrust corpus but not in the dictionary. 

As a check on identified words, the rules by which outliers were detected were verified by a human expert. Using this approach, HTRC formulated 48,308 rules that identified outlier words and provided corrections. HTRC members applied the rules to 256,000 non-Google digitized volumes from HathiTrust, which took 4 hours using the National Center for Supercomputing Applications (NCSA) Ember supercomputer. The results showed that the probability of a word having an OCR error (detected by the rule set) was 0.20%. The average number of errors per page was 0.57. The average number of errors per volume was 156. The probability that a page had one or more errors on it was 11%. The probability that any volume had one or more errors was 84.9%. Overall, 217,754 of the 256,416 volumes had one or more OCR errors and 7,745,034 of the 69,297,000 pages had one or more errors. 

2. Outreach 

Talk “Digital Humanities At Scale: Hathi Trust Research Center”, by Beth Plale at UMaryland February 29, 2012 hosted by Maryland Institute for Technology in the Humanities and the University of Maryland Libraries. 

Google Digital Humanities Awards Recipient Interview Report.  John Unsworth commissioned a study of award recipients of the Google Digital Humanities Award over the period 2010 – 2011.  The study, Google Digital Humanities Awards Recipient Interview Report, interviewed recipients of the Google awards to determine what difficulties the recipients encountered when working with the Google corpus. A recurring theme was weak metadata and poor OCR. 

OCR Summit, Oct 17-18, 2011, Texas A&M University. Loretta Auvil of UIUC attended the OCR Summit whose purpose was to bring together experts to work on the problem of Optical Character Recognition for early modern texts, when printing techniques make it difficult for machines to type text by “reading” page images. HTRC proposed involvement would be to provide post processing capabilities based on work we have done in the SEASR Services project to correct the Google Ngrams data as well as a corpus of 18^th and 19^th century novels. http://idhmc.tamu.edu/ocr-summit-meeting/ 

Digging Into Data: The HTRC is working with one of the recently awarded Digging into Data projects. The Principal Investigators of the “Digging by Debating” project, have approached the HTRC for help in extending the infrastructure and developing functionality that would benefit humanities researchers in general and the Digging by Debating project in particular. Colin Allen and Katy Börner, Indiana University, Bloomington, NEH; Andrew Ravenscroft, University of East London, Chris Reed, University of Dundee, and David Bourget, University of London, AHRC/ESRC/JISC are interested in creating interfaces and systems that harvest concepts that cross domains; for example, they want to investigate the process by which concepts in one domain, such as philosophy, are used in another domain, such as physics. 

HTRC Web Site: A new HTRC web presence was launched early December 2011 as part of the HathiTrust website and can be seen ahttp://www.hathitrust.org/htrc. In addition to the direct access provided via the URL, the HTRC webpages are also available from the HT’s site navigation; under the About option, the HTRC is referred to as “Our Research Center.” The HTRC website has entirely new content that covers the governance of the HTRC, technical architecture and organization, policies for access and use, project information with timelines and deliverables, information about collaboration opportunities, and demonstrations of future functionality. 

3. Initiatives 

Community Engagement through “Challenges”. HTRC is creating a set of ongoing "challenges" as a mechanism by which community interest and engagement in the research opportunities represented by the HTRC can be piqued and cultivated. These challenges are inspired by such successful challenges in other domains as, TREC (Text Retrieval Conference), Netflix, and the Music Information Retrieval Evaluation eXchange (MIREX). Downie is the founder of MIREX (2004), so he has had considerable experience in organizing and running evaluation challenges. As it stands now, four challenges are being sketched out as possible candidates: