You are here: Home > Publications > Final Evaluation Report, Phase I

Final Evaluation Report, Phase I

Final Evaluation Report - JISC Information Environment Service Registry (IESR) Pilot, phase I.

Amanda Closier

27th Feb 2004

Executive summary

This report details the evaluation activities of phase I of the JISC funded Information Environment Service Registry (IESR) pilot project. This 14 month pilot was split between MIMAS at the University of Manchester, UKOLN based at the University of Bath and the Cheshire development team at the University of Liverpool.

The report assesses the achievements of the project against the projects aims and in relation to the requirements elicited from its stakeholder community.

Introduction
Purpose of the evaluation
Methodology
Achievements
Issues
The future

1. Introduction

1.1. Purpose of this report

This report forms a key deliverable for UKOLN's contribution to the JISC Information Environment Service Registry (IESR) pilot project. It falls under Task 38 in Work Package 7 of the IESR pilot project's plan. UKOLN were required to evaluate the pilot service against user requirements and against quality attributes as identified by appropriate quality assurance models such as those used by the EDNER.

It was decided that the outcomes of the IESR's evaluation activities outlined in the interim evaluation report, including any areas of difficulty in achieving the goals of the evaluation should be recorded in this final evaluation report to be presented as a deliverable of WP 7 in February 2004.

1.2. The JISC Information Environment

The development of the Information Environment is one of the JISC's current strategic activities. This is envisaged as an online environment where access to 'scholarly and educational materials' is made as easy and as relevant as possible for the UK HE and FE communities. One of the programmes supported by this development work is the Shared Services Programme. More information about the JISC IE can be found at http://www.jisc.ac.uk/whatwedo/themes/information_environment.aspx. At the time of writing the structure of the JISC IE is represented in Fig. 1 below (from http://www.ukoln.ac.uk/distributed-systems/jisc-ie/arch/).

Fig. 1

1.3. Shared Services Programme

The Shared Services programme encompasses those services that fall under the 'Shared infrastructure' area of Fig. 1. In essence the

"... programme aims to develop a suite of shared services that will provide machine to machine (M2M) interfaces that allow machines to survey the information environment for end users. The programme will do this by exploring what common technical standards, terminologies, description schemas and user profiles are required and by determining sustainability and management issues."

The Shared Services programme sets out to ascertain the requirements for the operation of shared services within the JISC IE and to reach an integrated technical solution. It aims to roll out full services from 2005.

Work under this programme is being undertaken through studies, scoping work and pilot service development.

1.4. The JISC IESR Pilot Project

The JISC IESR Pilot started in November 2002 and ended in December 2003. The work was being carried out between three centres. These were MIMAS at the University of Manchester, UKOLN at the University of Bath and the Cheshire team at the University of Liverpool.

The IE Service Registry was proposed as 'a machine-readable catalogue of the high-quality resources that form the Information Environment.' The aim is to enable portals and other services to discover which resources are available and appropriate for their users, and to supply information about how these resources are accessed, through a machine-to-machine interface.

The IESR pilot project sets out to:

Establish the requirements for a Service Registry
Test a technical solution based on Cheshire II software
Obtain metadata from JISC-funded services to populate the pilot registry
Report findings and making recommendations about future developments

During the development of the project further funding was obtained to develop the pilot IESR further. The second phase of the IESR began in January 2004 and is due to run for another 14 months. Details of this second phase can be found in brief later in this document and in more detail elsewhere on the project's website.

2. Purpose Of The Evaluation

The purposes of the IESR phase I evaluation activities are:

To ensure that the pilot IESR has incorporated the key user requirements for its stakeholder and potential user communities as outlined in the Interim Requirements Report, and that the registry offers other desirable features and services requested by these user groups in the IESR Stakeholder analysis.
To ensure that the pilot IESR can be offered to the UK academic community as a basis for further development of a full Service Registry service due to its compliance with the IESR pilot project user requirements survey findings.
To assess the effectiveness of the IESR pilot project user requirements reports as statements of the true needs of the registry's potential user groups.
To ensure that the IESR pilot offers its potential users an acceptable and effective tool for meeting their resource discovery needs.

3. Methodology

In the interim evaluation report the IESR pilot project categorised user requirements according to timescales and importance and gave them a level. Details of how these levels were defined are included in the breakdown of achievements below.

The suggestion of the interim evaluation report was that the original scope of the pilot be amended to include the twelve level I requirements put forward by the IESR user community and that some attempt should be made to incorporate the level II requirements into the pilot registry. Level III requirements may or may not have been incorporated into the pilot registry. The incorporation of additional functionality into the pilot registry was an on-going activity throughout the duration of the project.

The registry's development was to be reviewed by internal project staff to ensure where possible that these requirements were being met.

It was envisaged that a series of user testing sessions should be conducted to explore whether the pilot registry offered acceptable and effective machine-to-machine functionality that potential users could successfully use to deliver a wider range of information to their end users.

These testing sessions were to review the processes enabling users to:

Create descriptions
Upload descriptions
Make descriptions available for harvesting
Connect to, interrogate and download records dynamically
Connect to and harvest records via a batch process

Development of the IESR pilot proved a longer and more complex task than originally predicted and at the end of December 2003 it was not feasible to run a useful user testing session. This was because IESR software was still under development. This user testing has been deferred until early in the phase II of the IESR project. More information about future IESR development work is available later in this report.

4. Achievements

4.1. Summary

There is now a demo IESR application available for web searching at http://www.mimas.ac.uk/iesr/iesr. It can also be searched by Z39.50 - there is an explanation of how this can be done available through a link from the above URL. The database contains records supplied by the project's key stakeholders and this demo application gives an indication of the IESR's functionality. IESR records may be one of two types, those describing informational or transactional services. Informational services consist of a collection, all the services that provide access to that collection and all the agent records for owners and administrators. IESR records alternatively may refer to a single transactional service (for example an OpenURL resolver) with agent records for its administrators.

During the pilot the IESR domain has been scoped in terms of types of services it will include; defined and documented a set of metadata for collections and services; set up a prototype process, which includes templates for data supply for receiving descriptions along with user guidelines; accepted descriptions from an initial set of users and done a significant study of stakeholder requirements.

4.2. Scoping requirements

The evaluation sought to answer the following questions:

I. Has the IESR pilot project incorporated the key functionality defined in its project scope document? Any failure to do so should be recorded and explained in the final evaluation report.

The project scoping document was revised and updated in Summer 2003 and its full content can be at http://www.mimas.ac.uk/iesr/scope.html.

Displayed below is a summary of the tasks that were derived from the scoping document and the progress the IESR pilot made with incorporating them into the registry's development in its first phase.

This brief summary includes some comments on some of the problems that were incurred in developing a pilot registry. After six months work on the pilot registry it became clear that some of the development and the issues surrounding it would take longer than originally planned for. Project staff felt that this was almost inevitable for a project of this nature and whilst every effort was made to incorporate every aspect and requirement gathered together in the user requirements report it has not proven possible to do so in phase I. Details of some of the issues faced during the first phase of the IESR are discussed in more detail later in this document.

In some cases the development of functional requirements has been delayed until phase II. Where this is the case it has been indicated in the tables.

Have the following organisations provided descriptions for the IESR pilot?
Note: not all of these records are currently 'live' in the IESR, but all will appear there early in phase II.

Arts and Humanities Data Service: Discussions about the records held in November 2003. Full records describing AHDS as a whole, each AHDS centre, and sub-collections to be supplied by the end of January 2004. If it is not possible to describe all sub-collections then a number of typical sub-collections will be described. YES
EDINA: Subset of records for testing in October 2003. Full records supplied in December 2003. YES
MIMAS: Full records describing top-level MIMAS collections in December 2003. Records describing sub-collections in January 2004. YES
Resource Discovery Network: Some records for RDN hubs in December 2003, others in January and February 2004. YES
UK Data Archive: Full set of records supplied, from September through to December 2003. YES
UK Mirror Service: Full set of records supplied in September 2003. YES

Are the following service types supported?

OAI-PMH: YES
OpenURL: YES
RSS: YES
SOAP: YES
SRU: YES
SRW: YES
Web CGI: YES
Web pages: YES
Z39.50: YES

Has the metadata schema defined in the project scope document been implemented?

YES: Implemented in Registry; documented as a human-readable Application Profile and an XML DTD is available for a single IESR record as output from the IESR (see http://www.mimas.ac.uk/iesr/metadata/docs/iesrdtd.html). Note that the development of the metadata schema took up a large proportion of the project's time: more than originally estimated.

Have there been any alterations to the defined metadata schema?

YES: Documented in the change history of the Application Profile.

Was an input and editing template for contributors offered by October 2003?

YES & NO: Static templates are available but can't be used for direct input. These were devised at the request of JISC for early supply of records (an addition to the original project plan) and were developed instead of an online web input form for phase I.

Has the pilot IESR provided the following output formats from the IESR metadata:

OAI-PMH: NO (In phase II)
Simple Dublin Core XML: NO (Early in phase II)
IESR XML: NO (Early in phase II)
Web: YES
Summary descriptions: YES
Full IESR descriptions: YES
Z39.50: YES
SUTRS - brief and full: YES
GRS-1: YES
Simple Dublin Core XML: YES
IESR XML: YES

Have any users been in a position to make use of the pilot service by December 2003?

YES & NO: Available but contains only test data. Users not advised of availability.

Has the Resource Discovery Network been able to harvest and re-use the metadata from the IESR in this timescale?

NO:This was a suggestion included in the scoping document if development had progressed far enough. OAI-PMH not yet available.

4.3. Level I requirements

II. Have the key user requirements identified in the Interim Requirements Report been successfully incorporated into phase I of the pilot IESR? Key user requirements are those that have been identified as essential within the timescale of phase I of the pilot and will henceforth be known as Level I requirements.

Level I Requirements

1 Technical documentation detailing standards, schema etc
YES - The following documents have been produced: Metadata Application Profile. Documents detailing data mappings to output formats. Examples. Templates. XML DTD.

2 Examples of use. Details on how the Registry is being or may be used would be useful for the AHDS to determine what use they could make of the registry.
NO - This requirement has become part of a wider Level II requirement (26) and has been deferred to Phase II of the IESR

3 Interworking with the Geo services.
YES - This requires a deeper understanding of the nature of the services and so further consultation work is taking place Further discussions have taken place since the requirements gathering phase. Web-cgi may be a sufficient format to successfully incorporate these resources into the IESR. More will be discovered in Phase II.

4 Collection descriptions must include classification number(s) describing content of collections.
YES - Various encoding schemes for dc:subject

5 Collection descriptions must include description of which subject schemes and/or classification schemes are used within the collection records, and specify the version of that scheme.
YES - Schemes have been included but no scheme version. Not likely to be included - this information is at too low a level of detail. The element 'usesControlledList' has been included in the input metadata.

6 A guide to good practice i.e. for record generation etc.
YES - This has been provided in the form of metadata creation guidelines which can be found at http://www.mimas.ac.uk/iesr/metadata/guidelines/Data_Creation_Guidelines.html.

7 Descriptions should be updated regularly to ensure currency
NO - Not yet applicable

8 Thesaurus 'Hassett' to be added to the list in dc:subject
YES

9 Dewey Decimal Classification number added to each record
YES

10 Clarification is required on the creation of distinct descriptions for a) a service as an organisation; b) collections managed by that service; c) services offered by the organisation which enable access to collections (e.g. OAI, Z39.50, RSS etc).
YES - This has been provided in the metadata creation guidelines.

11 Comprehensive data creation guidelines/training Requirements
YES - Metadata creation guidelines

12 Practical support for record creation and maintenance
YES - Metadata creation guidelines and a contact at MIMAS for help with creating and providing records for use in the IESR

4.4. Level II Requirements

III. Have other level II features requested by IESR stakeholders and potential users in the initial requirements gathering phase been incorporated into the pilot registry? These features being those requirements categorised as:
desirable within the scope of the pilot;
given no priority but suggested as useful within the scope of the pilot;
those requirements, which have been categorised as essential but have not been given any specific timeframe for development.

Level II Requirements

13 We may need to be able to distinguish sub-collections of a collection to provide an optimal level of service to users in respect of some services.
YES - isPartOf has been included in the input metadata

14 It must operate in M2M mode. It must be easily accessible and contain high quality, valid, timely and relevant data. Data will be high quality.
YES and NO - At the time of writing only a Z39.50 interface was available.

15 M2m interfaces that fit with current structures in uPortal are available in the very near future as funding ends Feb 2004
NO - More details about the specific requirements of potential users will be gathered in the next phase once further development has been completed. Whilst IESR has no specific requirement to fit with other projects' timescales incorporating as many user requirements as feasible into the registry is a high priority

16 Collection descriptions may need to include a unique identifier for a collection.
YES Will be allocated on registration with IESR

17 IESR development should prioritise the provision of landscaping mechanisms for portals
NO - further discussions will take place in user testing sessions early in phase II

18 Must also support (eGMS)
NO - More research needs to be done - will probably need a SOAP interface, so deferred to IESR phase II

19 Recording of media types available in a collection to be described in record. To allow searches for example to find image repositories/collections or music repositories.
YES - Use dc:type

20 Records within the IESR should display some indication of the described service(s) status. ILRT displayed an interest in having access to logs generated by IESR on last known status of service targets. Automated email system to send out messages to services indicating prolonged downtime periods. This would help avert problems where for example services are moved onto different machines but the record has not been updated.
NO - deferred to IESR Phase II

21 Business case for a need to generate metadata about InforM25
NO Outside IESR scope.

22 Unique identifier for each service described and access to a people friendly description of that service for example access to a list of unique i.d.s and the corresponding titles of the service(s) described.
YES - Unique identifiers will be generated on registration of entities. Lists available from Web interface in January 2004

23 Information about licensing and copyright issues
YES - Further clarification has been given in the metadata creation guidelines and details are included in various fields of the metadata

24 Detailed subject coverage description
NO - further discussions will take place in user testing sessions early in phase II

25 Alerting service for updates to the registry, i.e. when a record is edited, created or deleted - all those using the records should be alerted.
NO - Incorporated into plans for further development work in phase II

26 User documentation describing the IESR and the interfaces provided Incorporated into plans for further development work in phase II NO - deferred to IESR phase II. However some documentation is already available on the project website

4.5. Level III Requirements

IV. Have other desirable features suggested by IESR stakeholders and potential users for longer-term development activities been incorporated into the pilot registry? If not have any of the potential issues surrounding these developments been addressed? These are the IESR Level III requirements.

Level III Requirements

27 M2M SOAP interface
NO - Incorporated into plans for further development work in phase II

28 Manual checking of descriptions
YES - for the duration of the IESR project

29 M2M relationship between the IESR and access management services such as Athens.
NO - Currently there is only a very basic indication of authentication in service metadata - needs more discussion

30 Professionals to classify content of collections
NO - further discussions will take place in user testing sessions

31 It may be useful to record that a service enables searches by geographical co- ordinates
YES - Use dc:description

32 Logo to be added to Service metadata. Specifically for OpenURL resolvers. They may wish to indicate a preference for the button shown to the user.
NO - This is one of the possibilities to be discussed in the early stages of phase II when the project undergoes a further review of its metadata

33 Checks on the information provided against the actual service described e.g. Z39.50 information should be checked against target
NO - May be part of quality control record checking. This could be automated or manually checked.

34 Automated prompt to review/update records within the registry
NO - Needs discussion, but likely to form part of a production service.

35 Human editor for IESR records to act as a point of contact for support
YES - This is currently available through MIMAS and full details are made available through the project's website

36 A 'human' interface would be required to browse information on collections and services.
YES - the IESR Web interface has been provided at the end of phase I and it is expected that user feedback in phase II will guide any further development in this area

37 Automated checking of described service's status.
NO - see 20.

38 Automated email system to send out messages to services indicating prolonged downtime periods. This would help avert problems where for example services are moved onto different machines but the record has not been updated.
NO - see 37

39 There are only a few mandatory elements: some elements probably need to be mandatory in certain cases e.g. Administrator - at least one of the contact elements should be required
YES - Checked by input software

40 Some form of editorial control/checks on the information provided
YES - see 28

41 Checks on items such as machine addresses and port numbers
NO - see 33

42 A contact point to refer problems to; with an expectation of quick resolution.
YES - For the duration of the project there are full contact details including an email address and phone number on the project's website.

43 Integration with Athens and other JISC services
NO - For Athens, see 29. For other JISC services it is too soon to tell. User testing sessions with portals will take place during phase II

44 Ability to access the IESR via SOAP in addition to z39.50 and OAI.
NO - see 27

45 Access to statistics to enable record revision where necessary
NO - Access statistics will be available early in phase II

The Level III requirements have been rationalised and are summarised below.

Level III Requirements (Revised)

27 M2M SOAP interface
NO - Incorporated into plans for further development work in phase II

28 Manual checking of descriptions
YES - for the duration of the IESR project

29 M2M relationship between the IESR and access management services such as Athens.
NO - Currently there is only a very basic indication of authentication in service metadata - needs more discussion

30 Professionals to classify content of collections
NO - further discussions will take place in user testing sessions

31 It may be useful to record that a service enables searches by geographical co- ordinates
YES - Use dc:description

34 Automated prompt to review/update records within the registry
NO - Needs discussion, but likely to form part of a production service.

35 Human editor for IESR records to act as a point of contact for support
YES - This is currently available through MIMAS and full details are made available through the project's website

36 A 'human' interface would be required to check information on collections and services.
YES - the IESR Web interface has been provided at the end of phase I and it is expected that user feedback in phase II will guide any further development in this area

37 There are only a few mandatory elements: some elements probably need to be mandatory in certain cases e.g. Administrator - at least one of the contact elements should be required
YES - Checked by input software

38 A contact point to refer problems to; with an expectation of quick resolution.
YES - For the duration of the project there are full contact details including an email address and phone number on the project's website.

39 Integration with Athens and other JISC services
NO - For Athens, see 29. For other JISC services it is too soon to tell. User testing sessions with portals will take place during phase II

40 Access to statistics to enable record revision where necessary
NO - Access statistics will be available early in phase II

V. Have any other issues of concern raised by the IESR user groups at the Stakeholder meeting, through the IESR Stakeholder list or in the user requirements survey been addressed?

Every effort was made to incorporate changes arising as a result of the stakeholder meeting held in June 2003 and these fed into the requirements report. Until the project is in a position to run rigorous testing sessions with users it is unlikely that further feedback will be gathered.

VI. Does user testing with potential IESR users suggest that the registry is likely to prove an effective tool for facilitating this community's various resource discovery needs?

VII. Does user testing with members of the IESR stakeholder and potential user communities suggest that the solution developed for the pilot registry should be used as a basis for further development work on a full registry?

It has not been possible to run effective user testing sessions for the purpose of evaluation. Development of the IESR has taken considerably longer than envisaged and as a result of this the IESR phase I pilot was not in a position to offer a system for potential users to test. Web and Z39.50 interfaces were available however the registry only contained test data at the end of phase I. Changes to the input metadata have been made in response to the responses of the key user group who have provided descriptions for the pilot. The user testing sessions will be run early in phase II.

5. Issues

The issues encountered during phase I of the IESR pilot can be loosely categorised under the following headings.

5.1. Scoping

The IESR phase I pilot was planned with stakeholder analysis and user requirements gathering proceeding in parallel. This lead to some tension as there was pressure to proceed with the development leaving user requirements to be incorporated after this had started.

Stakeholder analysis and User requirements gathering was delayed by problems recruiting full time project staff. Requirements gathering was started in parallel with the initial development phase
In March 2003 it was flagged up that SOAP and UDDI development work may be needed but it would be highly unlikely that the project would be able to implement either within the timescale of the pilot. In June 2003 it was agreed that UDDI is beyond the capacity of the project, but that in any continuation bid it would be highlighted as a priority, to be incorporated in the first four months.
In May 2003 it became apparent that the requirements of the IESR's potential user community were far more wide ranging than initially expected and that it was unrealistic to commit to implementing all of them at that stage. From this point the project had to revise the scope of the pilot and this was later published on the website [http://www.mimas.ac.uk/iesr/scope.html]. It was also at about this point where it was acknowledged that further funding might be required to develop the IESR further in order to encompass emerging technologies.
In June 2003 it was highlighted that the development of the pilot registry would potentially require more than the planned effort predicted in the project plan. In most cases project staff were not funded to work full time on the registry.
In April 2003 some consideration was given to how much time the project could or should spend doing research looking at different models. Or whether the project should concentrate on producing an implementation. RDF was mentioned at this point but the issue of time was raised and the constraints put upon the pilot, the strongest of which was time.
In the longer term the project staff felt that there was a need resolve whether the IESR, both registered services and potential service users, will be restricted to the closed JISC .ac.uk domain, or whether it will become more global both geographically and commercially. Not all JISC services are restricted to UK or to HE/FE, e.g. Zetoc is available to some of the NHS .

5.2. Communication and Collaboration

It was flagged up during the early stages of the project that it would be useful for technical staff to be involved in JISC programme meetings to facilitate communication at the technical level where the development work being completed.

There was some concern that other JISC funded projects developing in parallel with the IESR had expectations that it will be up-and-running within their timescale. Given the amount of development activity needed and the pilot nature of the project it was unlikely that this would be the case.

5.3. Development

The development of the metadata schema took up a larger proportion of the project's time than originally estimated. However as this key task underpinned the registry's development it was impossible to avoid this delay.

The short timescale of a 12-month project has meant the project has had a high level of inflexibility because of the scheduling restraints. The IESR phase I has had little room for research and investigative work. It was felt that projects of less than 2 years are limited in what they can achieve.

Project staff felt that it would have been useful to flag up staff training needs at the beginning of the IESR pilot, unfortunately in some cases this was not possible. It was also recognised that in a research and development project it would be useful to allocate more time in the project plan to staff training needs.

Some additional effort was required to implement the IESR data model. There is a mismatch between the abstract model (3 component entities) and the records needed for the implementation in Cheshire (composed of flat XML files).

The web and Z interface to the database have been implemented in OmniMark for the pilot. This sped up the implementation because it enabled development to proceed in parallel with other MIMAS projects. This has not been regarded as a long-term solution however it has considerably improved the speed of the development work achieved in phase I. The licensing implications for this now mean that in any further development work the IESR may need be redeveloped in C++.

5.4. Evaluation

Late in phase I the evaluation of the first phase was deferred until early in phase II.

6. The future

6.1. IESR Phase II

At the IESR (phase I) project meeting on the 11th June 2003 plans were made to put together a bid for funding for a second development phase for the IESR. During the requirements gathering phase issues were raised which would lead to a considerably larger project in terms of research, development and time. The bid for phase II was put together and submitted to the JISC in mid October 2003.

The IESR project team are delighted to announce that the JISC have agreed to fund further development work on the IESR. The second phase of the project began in January 2004 and will continue for a further 14 months. A full project plan is available [PDF format].

Site Navigation