Blue Ribbon Task Force on Digital Preservation - 1

The NSF Blue Ribbon Task Force on Digital Preservation has released its Interim report, which sets the conceptual framework for a second report in about a year.  That report will complete the NSF brief, with recommendations and strategies for an overall approach to preserving public digital information.  This initial report is a fine piece of work that will inform the national agenda on digital preservation for the foreseeable future.  It synthesizes a broad spectrum of testimony by practitioners and embeds that testimony within principles of sustainability and business models that are the central core of the problem.  And there is a lot of data, as well... tables summarizing notable preservation activities in many countries that are navigating uncharted waters.

In 2007 we passed a major inflection point.  According to IDC analysts, quoted in the report (page 10),

the quantity of digital information created, captured, or replicated exceeded available storage capacity.

And the discrepancy is projected to grow, leaving half of all digital information without a permanent home by 2011 [seen through the perspective of someone who has jettisoned a third of his life ‘cargo’ in the past year, this is not an unalloyed tragedy]. The question becomes one of selection and election… How do we choose what to save, and how do we pay for it?

The features of the vulnerable datasets give texture to what otherwise might easily be a dull abstraction (however important).   The task force entertained testimony from managers of data with wildly different characteristics and value propositions, including:
•    Scientific data sets, including the Protein Data Bank and the ICSPR data repository
•    Commercial endeavors  (Boeing, RealNetworks, Microsoft)
•    Libraries
•    Think tanks and educational organizations (IDC, the Wharton School of Business)
•    Public Television and the Academy of Motion Picture Arts and Sciences
•    The Internet Archive , Portico,  and LOCKSS

To bring the issue home (literally)… do you know who is charged with the responsibility for looking after all those great public television shows you grew up with?  No one… not stations, not the creators of the work, not the Corporation for Public Broadcasting, not PBS… no one.   In fact, the pervading climate is one of resistance to preservation, or at least to the expenditure of scarce resources on then when the now is hard enough to pay for.  At least for some of these materials, there are resale markets (be part of the solution… go buy some DVDs from your local PBS station!).  The rest is teetering on the edge of digital oblivion… or already gone.  And this is the stuff that has presence in the public mind.

The problem gets harder for more obscure data (that is, almost everything else). The then versus now issue is core.  Though a portion of this report speaks of the tension between 'dividing the pie' to include preservation (and doing less of something else), versus 'growing the pie', this seems a hollow distinction.  Make the pie bigger, and you still have an allocation problem.  The question will always be what part of the budget must we allocate for preservation, and, more to the point, what portion of current production must be foregone in order to do so?  

Beyond the cost issues there lie systemic organizational and social issues which will be difficult to solve, paraphrased here from the executive summary:
•    Funding models which do not match long term needs
•    Unclear preservation responsibilities among various stakeholders
•    Inadequate incentives for collaboration among the stakeholders
•    Complacency – we shouldn’t feel desperate yet
•    Fear – we feel too desperate – there is no solution!

This report is a lucid characterization of the problem, and documents the current state of affairs well.  It deserves careful study (my own efforts in that direction will be noted in additional posts here).  The next part – actionable strategies and recommendations -- is going to be far harder still.

Congratulations to the task force on their achievements to date.
---------------

I've lost the view that afforded images of The Mountain like this for the paltry cost of turning my head as I awoke in the morning.  I wonder if I'll ever have it again?  Seattle is a great city... even (especially?) kneeling in the snow.

Comments

[Chris Rusbridge]

I think that "inflection point" quote is very odd. Storage has always, in my experience, been an expensive resource. I started in computing before there were any disk drives; the data for those structures we were analysing way back then (many of them now reified as Motorway bridges in the UK) were on punched cards, or temporarily on tape. But I doubt they were kept. Until recently, every personal computer hard drive I've had has tended towards full. Currently I've stacks of room for all the emails and files I want to keep from the last 15 years or so, but that's not true of my children: their hard drives are full to bursting of, well, you can guess what.

I think the amount of digital data created has always been way ahead of our capacity to keep it!