IESR Data Load from Templates

Note: These instructions have now been superseded by the IESR Editor and should no longer be followed. They have been kept for historical reasons.

Initial data supply to IESR is by templates, either XML or Excel. These instructions detail data load into IESR using data supplied according to these templates. These instructions will be superseded by a Webform data entry editor.

This data supply method is temporary. All data entry into IESR and updating of records in IESR is currently performed by Ann Apps. These instructions are documented as a backup in case someone else has to perform the task.

Note: these instructions are for loading data into IESR. However it is a good idea to load data into IESR-DEV first in case of unexpected problems.

Checking and Registering Supplied XML Records

Data Load Directory

Create a directory for data load under /home/zzieradm/dbs/metadata_src/editor/load/ (eg. using supplier name) and copy supplied XML files into it. [.../loaddev/ for IESR-DEV]

Check XML Records

Check XML records by running scripts in the load directory, separately for each entity type (assuming files are named appropriately):

• check_colls *coll*.xml
• check_svcs *svc*.xml
• check_agts *agt*.xml

These scripts append to a dated log file in the current directory.

Check log file. If XML errors are shown correct them and re-run scripts (delete log file fisrt).

[Scripts for IESR-DEV are: devchk_colls; devchk_svcs; devchk_agts.]

Register XML Records

When there are no errors in the XML records, register them in the Meta-Registry. It is prudent, before registration, to check that all local identifiers in the supplied records corespond.

Register entities by running scripts in the load directory, separately for each entity type (assuming files are named appropriately):

• register_agts <supplier> *agt*.xml
• register_svcs <supplier> *svc*.xml
• register_colls <supplier> *coll*.xml

where <supplier> is a 3-letter code for the data supplier (see below).

For example: register_agts mim *agt*.xml

[Scripts for IESR-DEV are: devreg_colls; devreg_svcs; devreg_agts.]

Registration Results

These scripts:

• Append to a dated log file in /home/zzieradm/dbs/metadata_src/editor/editlogs/ [IESR-DEV: deveditlogs]. This log file should be checked for errors. If errors are shown (except in keys or zeerex files), restore Meta-Registry to its orginal state and remove created XML files (see next points). For errors in keys or zeerex files it is probably OK to correct these files and proceed with data loading.
  
  
• Create a new version of the Meta-Registry in /home/zzieradm/dbs/iesrmeta/iesrmeta.xml [IESR-DEV: iesrmetadev/iesrmetadev.xml]. Before registration the Meta-Registry is copied to iesrmeta.bak in the same directory. If problems occur during registration it can be restored to its original state by: cp iesrmeta.bak iesrmeta.xml .
  During processing, command line output should say:
  
  New Meta-registry created
  New entitiy created
  
  If the second line is missing, the Meta-registry is corrupt.
  
  
• Create entity XML files, named by their IESR identifier, in /home/zzieradm/dbs/metadata_src/editor/iesr/<supplier> [IESR-DEV: iesrdev].
  
  
• Copy XML files (.xml) in load directory to .bak files before starting registration. These .bak files may be used to restore the original XML to recover from problems.
  
  
• Add IESR identifier to XML records in load directory for new records.

Checking and Registering Supplied Excel Records

Data Load Directory

Create a directory for data load under /home/zzieradm/dbs/metadata_src/editor/load/ (eg. using supplier name). [.../loaddev/ for IESR-DEV]. Convert Excel files to CSV files and copy .csv files to load directory.

Convert and Check CSV Records

Convert and check XML records by running scripts in the load directory, separately for each entity type (assuming files are named appropriately):

• cvtcsv_colls *coll*.csv
• cvtcsv_svcs *svc*.csv
• cvtcsv_agts *agt*.csv

These scripts create and check XML files corresponding to the CSV files, and append to a dated log file in the current directory.

Check log file. If convert or XML errors are shown correct them and re-run scripts (delete log file first).

[Scripts for IESR-DEV are: devcvt_colls; devcvt_svcs; devcvt_agts.]

When there are no errors in the CSV or the generated XML files, register the XML entities using the instructions above.

Index IESR

After registering new entities, load them into IESR and index by:
indexiesrnow
[IESR-DEV: indexdevnow]

[It is possible to perform an interim composition of the data by: makereg . This logs to the editlogs directory.]

Indexing IESR:

• Assembles composite IESR XML records (ie. for collections and transactional services) in /home/zzieradm/dbs/metadata_src/iesr/<supplier>/ named by the IESR identifier of the collection or transactional service. Collections that have no service, and agents that are not owners or administrators are not included - these are noted in the log file.
  
  
• Copies these composite records into single supplier files in the database: /home/zzieradm/dbs/iesr/DATA/<supplier>.xml
  
  
• Performs Cheshire index. Appends to dated log file in /home/zzieradm/dbs/iesr/IndexLogs/
  
  
• Logs full process, including abbreviated index log, appended to dated log file in /home/zzieradm/dbs/iesr/LoadLogs/
  
  
• Note that at the end of a month, log files in LoadLogs for that month are tarred and gzipped, log files in IndexLogs are removed.

Updating IESR Records

To update an IESR entity the IESR identifier must be included in the XML record on registration. Thus the best way to update a record is to edit the XML entity file in the load directory. Then check this XML and register as above.

New Supplier

Currently IESR has registered the following suppliers and their codes:

To add a new supplier:

• Create a new directory, named as the supplier code, in:
  • /home/zzieradm/dbs/metadata_src/editor/iesr/ [and iesrdev]
  • /home/zzieradm/dbs/metadata_src/iesr/ [and iesrdev]
  
  
• Add to supplier-names and supplier-codes structures in source code (remember to update size) in:
  • /home/zzieradm/XML/XOM/Upd/iesrconst.xin for registering entities [and Upd/Dev/deviesrconst.xin]
  • /home/zzieradm/XML/XOM/iesrconst.xin for IESR Web application [and Dev/deviesrconst.xin]

20 June 2006, ann.apps@man.ac.uk