Keeping electronic records:  issues and strategies 1

Author:  Greg O'Shea - Australian Archives National Office Canberra 2

ABSTRACT:  The increasing shift to recordkeeping and information systems which require hardware and software for their interpretation have forced a fundamental re-evaluation of the techniques required to capture and preserve records as evidence of organisational transactions in this environment. This situation presents a major challenge to organisations wanting to balance their drive for profit and/or efficiency against their need to meet their legal and information requirements. It also presents a major challenge to our society to maintain the significant electronic evidence of our times for future generations.

The beginning of this document looks at the traditional approaches taken to records and archives management and then provides reasons why many of these traditional approaches will not work in the electronic age. The middle section of the document addresses three issues which have emerged as central to an understanding of the theory which underpins electronic records management:

• - the Records Continuum
• - Appraisal of records by function not format
• - Records as evidence

The document concludes by outlining the Australian Archives current strategy for the management and preservation of electronic records and how we are establishing recordkeeping requirements in our own systems integration project as a practical test of the methodology.

Table of Contents

Traditional approaches to the management of records and archives Problems presented by these approaches for the management of electronic records Records continuum not life cycle The medium is not the message: appraisal of electronic records Records as evidence: not documents or data Australian Archives strategies for the management of the electronic records of the Federal Government Systems integration and the functional requirements for recordkeeping preliminary investigation analysis of business activity identification of recordkeeping requirements assessment of existing systems identification of strategies for recordkeeping design of recordkeeping system(s) implementation of recordkeeping systems(s) post-implementation review. Conclusion

Traditional approaches to the management of records and archives

The traditional approach to records and archival management has had the following general characteristics:

• - archivists at 'arms length' from the records management process and the consequent view of archives as records dumping grounds
• - archives at the end of the records life cycle, records management at the beginning and middle
• - records and archives physically held by archival institutions
• - appraisal of physical accumulations of records (either in the agency or upon receipt by the archives)
• - preservation of the original media
• - luxury of time
• - records managers and archivists as separate professions
• - descriptive practices based on records with the same numbering system or physical characteristics
• - comprehensible without the need for machinery (except for microform and film)
• - require large physical storage space for records
• - the provision of passive access to electronic records (ie access to the media not the contents)

Problems presented by these approaches for the management of electronic records

The rapidity of technological change and the instability of the media have meant that traditional approaches to archives and records management, where they have been tried, have not been satisfactory for the long term preservation of electronic records.

Technological change impacts on electronic environments as hardware and software undergo evolutionary change at an exponential rate. At the hardware level changes to both machinery and capability as well as to storage media are in some cases an almost annual event. On the software side development is even more rapid to the point where companies 'market' software which doesn't even exist yet; i.e. vaporware.

Examples of these changes include:

• upgrades to newer versions of operating systems or application programs;
• upgrades of hardware platforms and changes in storage device density and compression algorithms;
• the development of compound multimedia documents; and
• migration issues arising from the first three.

In the process of upgrading systems it is usually the practice for records and data which are 'current' to be migrated to the new platform to meet current business needs. The non-current records / data will often not be migrated. In addition to this the functionality of the system and the format of the data might change as part of the upgrade process. Why is this of concern ?

The longer the non-current records / data await conversion the more difficult the process becomes as the time increases after the system upgrade. In certain cases the records / data may actually be written off because it has no immediate value or as a cost cutting measure. This can cause problems down the line if the process is not conducted systematically and valuable records / data are lost or becomes inaccessible.

If the functionality of the system changes then the new or 'improved' system is different from its previous manifestation. Even if the function which is being managed is the same, improvements or alterations to any given system can affect user views via changes to the data, applications and access privileges. This process has implications for the production of evidence. For example if an appeal is lodged over the rejection of an application for benefits and part of the evidence is contained in the Benefits Case Management System it may be very difficult to view the evidence if there have been modifications or upgrades to the system between the original application and the consideration of an appeal.

In the last ten years of computing the power has literally gone to the people. With a computer on 'every desk' now and the development of inter and intra agency networks large amounts of computer power and much control now rests with individuals. The negative side this shift in power has occurred without an accompanying management or regulatory structure. A very similar result, in fact, to that which has occurred with the decentralisation of traditional paper document registry systems. Individuals now have a great deal of control of the documentation process, whether it be in paper or electronic environments, without procedures and structures in place to enable them to exercise an informed choice. Many organisations have only recently awoken to the fact that their greatest asset (after people and dollars), information, is not adequately or systematically managed. As a consequence resources which could be more effectively utilised in the pursuit of organisational priorities are often wasted because of poor access to and control over this information.

There are a number of risks associated with this changing electronic environment, whether it be unregulated end users or poor systems design. They are:

• uncontrolled accumulation of records, documents and data;
• inadvertent destruction of records, documents and data;
• unauthorised tampering with records and documents;
• lack of or absence of systems documentation and associated metadata; and
• poor uncoordinated recordkeeping and duplication caused by the maintenance of parallel paper and electronic systems

If these characteristics are present in your system they can cause:

• system paralysis or at the very least hindrance in accessing information;
• additional costs associated with the purchase of additional storage;
• increased risk of wholesale, unsystematic and possibly illegal destruction;
• loss of valuable business and archival records;
• increased risk of security breaches;
• unauthorised alteration or deletion of records (loss of evidence);
• public embarrassment ;
• unnecessary delays in or breakdowns in the business process; and
• lack of public accountability.

Records continuum not life cycle

Because electronic records are software and hardware dependent and these change with time, the notion of the life-cycle management of electronic records is difficult to sustain. The life-cycle approach, based on the 'movement' of self-contained paper based records from creation through administrative use to ultimate destruction or retention, is unsatisfactory in a changing technological environment.

In the traditional life-cycle approach, decisions are made about the value of records as archives at the end of their active life. Only archives are then transferred to the archival repository. Following this approach these decisions could only be made about those electronic records that survived to the end of the active life. Given electronic records' dependency on software and hardware and their susceptibility to inadvertent loss through technological obsolescence this would be a huge risk. As a consequence attention must be given to records and archival issues from the outset for electronic records. The ideal time is for this occur is as part of the systems development process at the point systems are being established or upgraded ie even before the records are created. (see Systems Integration and the Functional Requirements for Recordkeeping)

Another consequence of the continuum approach is the breaking of the connection between the status of records and their physical custody. This has important implications for the control of and access to electronic records as physical location becomes somewhat redundant, both for the organisations which create and manage electronic records for business purposes and for archival institutions who are required to ensure the preservation of those electronic records which have archival value. For example some organisations within the Commonwealth sphere have already outsourced the provision of their IT (information technology) services to other Government agencies or to private companies. We live in a virtual world.

The medium is not the message: appraisal of electronic records

In the electronic environment, because records are not self evident and need appropriate software, hardware and associated metadata to be understood we have concluded that it is imperative to specify not only what values records have but which records are to be captured. As a consequence, to enable the records to be captured, we need to provide more specific details about what data might be needed to make the record linked to good descriptions of the functions to which they relate. This can be done as part of the business process modelling phase

This process is part of the system development life-cycle. IT managers are clearly key to this process as it is they who will be responsible for:

(a) the capture and maintenance of electronic records; and

(b) the development or up-grade of systems

Records managers and archivists and others associated with the process of records creation and recordkeeping, therefore have to be involved in the systems development process in order to provide sufficient information about what records need to be kept and for how long.

Appraisal, of electronic records, should be undertaken at the logical level i.e. the high-level diagrammatic representation of the system where it is relatively easy to see what functions the system manages, what business processes are involved and where records of transactions need to be kept. Appraisal done at this level is independent of how the records are physically stored. The next step is then to build a link between the logical level of a system and the physical level (the level of databases, files, data etc). This approach, i.e. functional / logical level appraisal, is advocated, as it produces a simple, integrated and non-redundant definition of the records that is independent of frequent system and software changes. This detail is then enshrined in the business rules for the new system.

Regardless of the way in which records are kept, be it electronic or otherwise, they can be appraised using standard appraisal methodology and employing standard appraisal criteria. It is the record-keeping systems of agencies which must be appraised; looking at information flows, the interrelationships between elements of the record-keeping system, what (or even if) records are created in the documentation of activities and functions, and crucially what information needs to be provided about the records in disposal authorities to ensure that there is sufficient detail for IT practitioners to do their job.

The archival value of records needs to be determined at creation and preservation strategies employed from that point to avoid loss through planned obsolescence or separation from business context or unsystematic destruction.

Records as evidence: not documents or data

It is fairly clear, in the new technological environment, that the distinct roles of archivist, records manager and IT personnel are becoming less distinct and have begun to intersect at the edges. This process presents us with some challenges when it comes to the use of language, particularly as it is influenced by technological change and the need to communicate across professional boundaries.

For the archivist and records manager the key concept to communicate is that of records as evidence of business transactions. A significant example of how we arrived at this point can be seen from the following;

The Archives Act 1983 defines a record as:

a document (including any written or printed material) or object (including a sound recording, coded storage device, magnetic tape or disc, microform, photograph, film, map, plan or model or a painting or other pictorial or graphic work) that is, or has been, kept by reason of any information or matter that it contains or can be obtained from it or by reason of its connection with any event, person, circumstance or thing.

This definition constructed in the early 1980s views records as physical objects such as paper files, tapes, disks etc. This type of definition presents us with major problems when we are dealing with electronic records. Firstly, records are neither purely documents nor objects. Secondly, in the electronic age of digital copying, records are independent of the medium (an object) on which they originally sat or currently sit. Thirdly, in electronic systems individual records and their components parts are not physically located in a logical sequence or (often) in the same physical space. Lastly, the definition does not provide any evidentiary test.

Because of these problems the Archives has decided, after some research and deliberation, that we needed an updated definition of a 'record' which has a generic set of characteristics and that is independent of format :

• A record is that which is created and kept as evidence of agency or individual functions, activities and transactions.
• To be considered evidence a record must possess content, structure and context and be part of a recordkeeping system.

By contrast in the IT environment a 'record' is defined as:

record

(1) an aggregate that consists of data objects, possibly with different attributes, that usually have identifiers attached to them or more succinctly

(2) a set of data treated as a unit or a set of one or more data items grouped for processing.

The closest link between the archival definition of a 'record' and IT terminology comes with the definition of a 'transaction'. For example:

transaction

an exchange between a work station and another device that accomplishes a particular action or result; e.g. the entry of a customer's deposit and the updating of a customer's balance or an item of business such as handling customer orders and customer billing.

The usage of the terms 'archive' and 'archiving' in the IT sphere has tended to give IT practitioners a misleading impression of records and archival processes.  In the IT environment these words have the following meanings:

archive

a copy of one or more files or a copy of a database that is saved for future reference or for recovery purposes in case the original data is damaged or lost.

archiving

the storage of backup files and any associated journals, usually for a given period of time.

The first is used as a verb ie to archive and the second is a corruption of the first and is used as a noun.

In the archives and records management world an 'archive(s)' has a two meanings:

archive(s)

(1) an institution responsible for the acquisition, preservation and communication of archives.

(2) non-current records preserved, with or without selection, by those responsible for their creation, or by their successors or by an appropriate archives because of their archival value [archival value = those values, administrative, fiscal, legal, evidential and / or informational, which justify the indefinite or permanent retention of records]

More recently the distinction between current and non-current records and archives has undergone a re-examination because of the need to appraise electronic records at the creation stage. This re-examination has now changed the way we look at archives in a distributed custody environment. In this environment the following definition is now more appropriate:

archives

those records that are appraised as having continuing value

Archiving is not a term in formal use in the archives / records management field although it is used colloquially. In any event, to do the archiving or to archive refers to the practice of preparing consignments of records and transferring them to archival custody.

Turning to the term 'document' we can see some measure of commonality in usage. The IT usage has 'document' defined in a number of ways:

document

(1) a named, structural unit of text that can be stored, retrieved, and exchanged among systems and users as a separate unit

(2) information and the medium on which it is recorded that generally have permanence and can be read by humans or machines.

The archives and records management definition is:

document

(1) a combination of a medium and the information recorded on or in it, which may be used as evidence or for consultation

(2) a single archival, record of manuscript item

There are some obvious similarities between the two but the crucial and significant difference occurs with the phrase in the latter definition, "may be used as evidence or for consultation"

If we revisit the definition of a 'record' we see that the concept of evidence is at its heart. And for a 'document' to be useful as an evidential record it, therefore, must possess, "content, structure and context and be part of a recordkeeping system." This is also the essential difference between a recordkeeping system and an information system. A 'document' in an information system which is, " used....for consultation" only, has limited value as evidence. For example a document management system which is used to create, receive, maintain and provide access to 'documents' which have no or insufficient evidential characteristics is an information system not a recordkeeping system.

Not surprisingly 'data' is a term which has been taken on by the IT industry as one of its own, most probably due to the ancestral link between scientific research and the development of computing. A standard dictionary definition of 'data' is:

data

(1) a series of observations, measurements, or facts; information

(2) Computer technol. the information operated on by a computer program

In the IT world 'data' is defined as:

data

a re-interpretable representation of information in a formalised manner suitable for communication, interpretation or processing. Operations can be performed upon data by humans or by automatic means.

Given that 'data' does not have a meaning or usage specific to archives and records management the meaning provided in the 'archival' dictionary is remarkably similar to the IT dictionary.

data

facts or instructions represented in a formalised manner, suitable for transmission, interpretation or processing manually or automatically.

To complete the circle the main link between data and records is through data elements. Data elements are the primary building blocks for records in electronic systems. Records, as has been explained, are not just data or documents, but that which are created and kept as business transactions.

Go to Part 2 of this article.

Greg O'Shea - Australian Archives National Office Canberra

1 - The content of this paper is largely drawn from Keeping Electronic Records Australian Archives Canberra 1995. Copies are available from me via e-mail gregos@aa.gov.au. The section on appraisal is based on an excerpt from my paper, "The medium is not the message: appraisal of electronic records by the Australian Archives" in Archives and Manuscripts, vol . 22 No. 1 1994 .

2 - Greg is currently Assistant Director Electronic Records working on electronic records policy at the National level. The Australian Archives is the archival authority for the Federal Government .