Tag Archives: digital preservation

A preservation workflow using our toolkit

Kit Good will describe a potential records management workflow using these open source tools. This post describes a potential preservation workflow. Since UoL does not yet have a dedicated digital preservation service, this workflow will have to remain hypothetical, but the gist of it is whether these tools can succeed in creating preservable objects with sufficient technical metadata to ensure long-term continuity.

Ingest

The records to be preserved would arrive as Submission Information Packages (SIP). As a first step we would want to ingest selected records into the digital archive repository we haven’t got yet. In our hypothetical scenario, this would probably be done by the records manager (much the same way he transfers records of permanent value to the University archivist). If we had a digital archivist, all we would need is an agreed methodology between that person and the records manager, for example a web-based front end like Dropbox that allows us to move the files from one place to another, with some metadata and other documentation.

Validation

At a bare minimum, the archive service would want to run the following QA steps on the submitted records:

  1. Fixity
  2. Virus check
  3. Confirm submission formats can be supported in the repository

DROID can perform some of these steps for us with its built-in checksum, and the way it identifies formats by comparing them with its online registry. (This would need to align itself with a written preservation policy of some sort). At the moment we lack a virus checker, but it would not be unfeasible to use an open source virus checker such as Avast. This is one step where the DPSP works out nicely, with its built-in virus check and quarantine stage.

Next, we might want to validate the formats in more detail, since DROID doesn’t look at the signature of the file formats. This is where JHOVE comes in, although the limitations of that tool with regard to Office documents have already been noted.

You’ll notice by this point we are not advocating use of the NZ Metadata Extraction Tool, since its output has been found a bit lacking, but this would be the stage to deploy it.

Transformation

Next step is to normalise the records using Xena. This action is as close as we get to migration in our repository and the actions of Xena have already been described.

These steps create a little “bundle” of objects:

  • The DROID report in CSV format
  • The JHOVE output in XML format
  • The normalised Xena object in .xena format

For our purposes, this bundle represents the archival copy which in OAIS terms is an Archival Information Package (AIP). The DROID and JHOVE files are our technical metadata, while the actual data we want to keep is in the Xena file.

Move to archive

Step four would be to move this AIP bundle into archival storage, keeping the “original” submissions elsewhere in the managed store.

Access

A fifth step would be to re-render the Xena objects as dissemination copies, as needed when requested by a member of staff who wanted to have access to a preserved record. For this, we would open the Xena object using the Xena viewer and create a Dissemination Information Package (DIP) version by clicking the OpenOffice button to produce a readable document. (As noted previously, we can also make this OO version into a PDF).

This access step creates yet more digital objects. In fact if we look at the test results for the 12 spreadsheets in our test corpus, we have created the following items in our bundle:

  • One DROID file which analysed all 12 spreadsheets – in CSV
  • One JHOVE file which analysed all 12 spreadsheets – in XML
  • 12 normalised Xena objects – in XENA
  • A test Open Office rendition of one the spreadsheets in ODS
  • A PDF rendering of that OO version

In a separate folder, there are 12 MET files in XML – one for each spreadsheet

In real life we certainly wouldn’t want to keep all of these objects in the same place; the archival files must be stored separately from the dissemination versions.

Checking

Many of the above stages present opportunities for a checksum script to be run, to ensure that the objects have not been corrupted in the processes of transformation and moving them in and out of the archival store. If we wanted to go further with checking, we would re-validate the files using JHOVE.

Sounds simple, doesn’t it? But there are quite a few preservation gaps with this bare-bones toolkit.

Gap #1

The technical metadata outputs from DROID and JHOVE are all “loose”, not tightly fused with the object they actually relate to. To put it another way, the processes described above have involved running several separate actions on the object that is the target for preservation. It has also created several separate outputs, which have landed in several different destinations on my PC.

We could manually move everything into a “bundle” as suggested above, but this is extra work and feels a bit insecure and unreliable. For real success, we need a method (a database, perhaps) that manages all the loose bits and pieces under a single UID, or other reliable retrieval method. Xena does create such a UID for each of its objects – it’s visible there in the dc:identifier tag. So we may stand a chance of using that element in future iterations of this work.

Another thing which may seem trivial, but JHOVE, DROID and Xena can do batch processing, and MET cannot. This results in a mismatch of outputs, and the outputs are created in different formats.

There is also some duplication among the detail of the technical metadata that has been extracted from the objects.

Gap #2

Ideally we’d like the technical metadata to be embedded within a single wrapper, along with the object itself. The Xena wrapper seems the most obvious place for this. I lack the technical ability to understand how to do it, though. What I would like is some form of XML authoring tool that enables me to write the DROID and JHOVE output directly into the Xena wrapper.

Gap #3

All the steps described above are manual. Obviously if we were going to do this on a larger scale we would want to automate the actions. This is not surprising and we knew this would be the case before we embarked on the project, but it’s good to see the extent to which our workflow remains non joined-up.

Gap #4

Likewise, our audit trail for preservation actions is a bit distributed. Both DROID and JHOVE give us a ‘Last Modified’ date for each object processed, and Xena embeds naa:last-modified within the XML output for each object, but ideally these dates ought to be retrievable by the preservation metadata database and presented as a kind of linear chain of events. We’d also like to have a field identifying what the process was that triggered the ‘Last Modified’ date.

Gap #5

How we manage the descriptive metadata in this process? In order to deliver a DIP to our consumer, they would have to know the record exists and want to know something useful about it to enable them to retrieve it. We have confirmed that descriptive metadata survives the Xena process, but how can we expose it for our users?

What we’re talking about here is a searchable front end to the archival store which we haven’t yet built. Kit is proposing a structured file store for his records, so maybe we need to expand on this approach and think about ways of delivering a souped-up archival catalogue for these assets.

Significant Properties in Office documents

Significant properties are properties of digital objects or files which are considered to be essential for their interpretation, rendering or meaningful access. One of the aims of digital preservation is to make sure we don’t lose or corrupt these properties through our preservation actions, especially migration. For this particular project, we want to be assured that a Xena normalisation has not affected the significant properties of our electronic records.

Much has been written on this complex subject. Significant properties are to do with preserving the “performance” and continued behaviour of an object; they aren’t the same thing as the technical metadata we’ve already assessed, nor the same as descriptive metadata. For more information, see The InSPECT project, which is where we obtained advice about significant property elements for various file formats and digital object types.

However, the guidance we used for significant properties of document types was Document Metadata: Document Technical Metadata for Digital Preservation, Florida Digital Archive and Harvard University Library, 2009.

To summarise it, Florida and Harvard suggest we should be interested in system counts of pages, words, paragraphs, lines and characters in the text; use of embedded tables and graphics; the language of the document; the use of embedded fonts; and any special features in the document.

When normalising MS Office documents to Open Office, the Xena process produces a block of code within the AIP wrapper. I assume the significant properties of the documents are somewhere in that block of code. But we can’t actually view them using the Xena viewer.

We can however view the properties if we look at the end of the chain of digital preservation, and go to the DIP. For documents, The Open Office rendering of a Xena AIP has already been shown to have retained many of the significant properties of the original MS Word file. (See previous post). The properties are visible in Open Office, via File, Properties, Statistics:

The language property of the document is visible in Tools, Options, Language Settings (see image below). I’m less certain about the meaning or authenticity of this, and wonder if Open Office is simply restating the default language settings of my own installation of Open Office; whereas what we want is the embedded language property of the original document.

The other area Open Office fails to satisfy us is with the embedded fonts. However, these features are more normally associated with PDF files, which depend for their portability success on embedding fonts as part of their conversion process.

For our project, what we could do is take the Open Office normalisation and use the handy Open Office feature, a magic button which can turn any of its products into a PDF.

We could use the taskbar button to do this, but I’d rather use File, Export as PDF. (See screenshot below.) This gives me the option to select PDF/A-1a, an archival PDF format; and lossless compression for any images in the document. Both those options will give me more confidence in producing an object that is more authentic and can be preserved long-term.

This experiment yields a PDF with the following significant properties about the embedded fonts:

These are the Document Property fields which Adobe supports.

The above demonstrates how, in the chain of transformation from MS Word > Xena AIP > Open Office equivalent > Exported PDF, these significant properties may not be viewable all in one place, but they do survive the normalisation process.

DIPs from a Xena object – descriptive metadata

In OAIS terms, the Xena digital object is our Archival Information Package (AIP). This is the object that would be stored and preserved in UoL’s archival storage system (if we had one).

For the purposes of this project, the records manager needs to be assured we can render and deliver a readable, authentic version of the record from the Xena object. In OAIS terms, this could be seen as a Dissemination Information Package (DIP) derived from an AIP. Among other things, the DIP package ought to include sufficient descriptive metadata.

We’ve defined a minimal set of what we would need for records management purposes (names, dates for authenticity) and for information retrieval, and criteria for assessing overall success of the transformations – such as legibility, presentation, look and feel, and basic functionality. See our previous post for the documentation of how we arrived at our criteria.

For most of the examples below we are looking at standard Xena normalisations, which can produce a DIP by the following methods:

  • Export the AIP to nearest Open Office equivalent
  • Export the AIP to native format
  • Export the AIP to a format of our choice (an option we haven’t tried as yet)

Emails are slightly more complex – see below for more detail, and previous post on emails.

Documents, spreadsheets and powerpoints

For these MS Office documents, Xena normalises by converting them to their nearest Open Office equivalent. We can view this Open Office version via the Xena Viewer simply by clicking on Show in OpenOffice.org.

In the case of a sample docx, this shows us the document with its original formatting intact. In Open Office, we can now look at File, Properties, and confirm the following descriptive metadata are intact (some of these we will revisit when we look at significant properties):

  • Title
  • Subject
  • Keywords
  • Comments
  • Company Name
  • Number of pages
  • Number of tables
  • Number of graphics
  • Number of OLE objects
  • Number of paragraphs
  • Number of words
  • Number of characters
  • Number of lines
  • Size
  • Date of creation / created by who
  • Date of modification / modified by who

One missing property is a discrete ‘Author’ field in the OO version.

PDFs

Xena Viewer has its own Adobe-like viewer for PDFs. When we look at the Document Properties for a normalised PDF, we get the following descriptive metadata:

  • File size
  • Page count
  • Title
  • Author
  • Creator (i.e. software that created the file)
  • Producer (i.e. software that created the file)
  • Date of creation
  • Date of modification

Images

We may not need much descriptive metadata for individual images. We looked at the converted images using a free EXIF viewer tool and the date values (creation and modification) are intact in the normalised object.

Emails

For emails, the AIP will either be a standard Xena normalised object or a binary Xena object. Broadly, the former can export to an XML version and the latter can export back to its native format (.msg file). We’re always guaranteed to get a minimum of descriptive metadata whichever transformation we enact. This would be:

  • Title
  • Author
  • Recipient
  • Date

Results – progress so far

Below is a PDF of my table of analysis of descriptive metadata (and significant properties).

We also have Kit’s assessment of the transformations, including his comments on whether a reasonable amount of metadata is intact.

Meeting up

It is always great to share ideas and experiences with others in your field, and so the meeting of the Digital infrastructure programme – of which our project is part – was enjoyable and very useful.

The first thing, after arriving at the impressive LSE Library building, was to introduce ourselves and provide some 5 min presentations on our projects. A full list is here: 
http://www.jisc.ac.uk/whatwedo/programmes/preservation/12-11projectlist.aspx

Through our project and my growing general interest, digital archiving techniques are helping me to shape some new approaches to more conventional electronic records management problems.

The outcome of Manchester’s Caracanet Case Study, which deals with the acquisition and preservation of email accounts, will be invaluable to records managers trying to define approaches to managing emails as records across their organisations.

The Institute of Education’s Digital Directorate project was interesting to me as archivists approaching essentially a records management challenge. The inter-disciplinary approach can surely only benefit both fields.

I am a keen advocate of training and awareness and several projects – DICE, PrePARE, SHARD – looked at ways of engaging the producers of research data with preservation issues. I’ll also be very interested in the outcomes of Bristol’s DataSafe project. There are so many methods now of delivering training; these projects should give some useful pointers to what works best and what our target audiences want.

The discussions that followed covered the value and benefits of digital preservation and how best to establish a business case for DP projects. I’m still convinced that it will be the hard numbers around reduced storage costs that will give a bedrock to the other DP benefits. The implied processes of appraisal and selection around what groups of records to keep will surely give us the opportunity to get rid of the huge amounts of unstructured information many organisations are storing.

There was a debate about the best approaches to ‘community engagement’ around digital preservation, including an account of a previous three day ‘hackathon’ as part of the AQuA project. Whilst I am a big fan of listservs and online forums, this event showed that getting a few people in a room to share their experience is still the best way to build and maintain a ‘community’.

Thanks to JISC and the SPRUCE project for organising and to the attendees for a really enjoyable and useful meeting.

Adapting OAIS for records management

In planning this project we’re obviously referencing the OAIS standard for a digital archive.

To quote from Wikipedia, ‘Open Archival Information System (or OAIS) is an archive, consisting of an organization of people and systems, that has accepted the responsibility to preserve information and make it available for a Designated Community’.

The OAIS approach reminds me of the old style paper records store method for managing records. We (the records manager) receive the closed files and store them safely and securely. When a record is required we are asked by the department for a specific box or file. We then make it available.

This is being replicated in several ways in current records management approaches. At one conference I heard of one organisation moving its paper files to offsite storage, then providing a ‘scan on demand’ approach when a file was requested. The offsite store would provide a scanned copy and email it to the requesting department. I thought this was a nice way of sidestepping a massive digitisation project – a digital ‘copy’ is provided when required, the mass of original paper retained in a low-cost storage environment.

In-house electronic document management solutions, from a simple intranet to a full blown EDRMS, have often relied on the information being available instantly via a search.  That obviously relies on the quality of the stored records and speed / accuracy of the search engine.

Could the OAIS approach work in an internal business context? Can the electronic records be submitted or ‘declared’ to the records management team and stored. The records with long term retention converted into preservable formats. On demand, the records can be made available through a print to PDF facility. The audit trail of access is inherent in the very record produced.

There are several visual depictions of the standard on produced on page 4-1 on http://public.ccsds.org/publications/archive/650x0b1.pdf, to which I add my ‘scrawled on the back of a napkin’ version to explain how this approach might work:

The benefit, you’d hope, would be ‘future proofed’ electronic documents managed and made available at a reasonable cost over decades. The drawbacks would, perhaps be in the user experience where people are used to the instant search return of a Google query. Now the user would effectively be searching a catalogue of record references and requesting access to the one they want.

The challenge is: would the approach actually work? That is one of the things we are trying to explore with this project.

What and why…

Most of the information now created in the University is ‘born digital’. Staff create documents in file formats that have much shorter life spans than paper. We need to mitigate the risk that these files become unreadable.

Buying an EDRMS is a huge cost for an organisation in terms of licences, implementation and training. Is there a way that electronic records can be effectively managed and preserved without this cost?

We thought: “Can we use our existing infrastructure, with some open source tools, to build a practical, cost-effective solution to the long-term management of our key electronic records?”

This project will build a test environment for converting files into preservable formats. Will they be fit-for-purpose as university records?

The aim of the project is to identify the opportunities and challenges to this approach to electronic records management. Is it a viable alternative? Is it practical for records managers?

Kicking off…

My name is Kit Good, University Records Manager and Freedom of Information Officer at the University of London. I’m lucky enough to work in an institution where the technology department, the University of London Computer Centre (ULCC), has a dedicated Digital Archiving team with a lot of experience in delivering preservation projects around electronic records.

 I’d met with Ed Pinsent, Digital Archivist, several times since I started last year to discuss the challenges around electronic records management. It was back at the start of September when Ed approached me about submitting a bid to the JISC 12/11 Digital Infrastructure Programme.

Ed and Kit

Our bid proposed testing the concept of ‘a simple toolkit of services and software that can plug into a network drive and create preservation copies of core business documents that require permanent preservation’. We were delighted to find out earlier this month that our bid was successful. 

We are hoping that the outcomes of this project will be useful to records managers, the digital archivist community and the Higher Education Sector as a whole. This blog will track the development of the project from its November kick-off to its February close.

More posts to follow…