Adding a Harvester tool to your collection

If you already have a collection and would like to get your Metadata Records harvested to the BEN portal you need to create a Harvester tool for your collection that is capable of creating Metadata Records conforming to the BEN LOM structure and sending it in a OAI PMH format to the BEN Portal upon the Portal's request.

Before you start working on the Harvester Tool for you collection take a look at how various components of the Harvester in the BEN Portal and other collections are designed to interact with each other. This is important for you to know since your Harvester tool should be designed in a similar way so that the BEN Portal can “talk” to your collection and get the data it needs in a format that it understands.

Also have a look at the actual Harvesting Process where we have described exactly how we perform harvests on a quarterly basis and what results in a failed harvest and what needs to be done in the case of a failed harvest.

Requirements for your Harvester tool

Your Harvester tool needs to be able to do the following two things:

  • The produced file must be presented to the BEN Portal wrapped in a OAI PMH (version 1.1 or 2.0) envelop
  • The produced file must contain two required classifications: Discipline, Pedagogical Use. See the BEN LOM XML schema “REQUIRED: Classifications” section for more details.

Which means that the Metadata Records in the XML file your Harvester tool generates must contain all the required fields as indicated in the BEN LOM specification and the actual values of these fields have to exactly match one of the eligible values for that field as descried in the BEN LOM specification. Also the format of the file must conform the the supported versions 1.1 and 2.0 of OAI PMH.

This would be easy to do if your collection was set up to store data using the BEN LOM specification. If that is not the case then you might have to massage your data a little so that it meets the requirements of the specification.

Helpful resources

Available Software

We created a generalized version of the collection side Harvesting tool so that each new partner can install it in their collection, but it requires a lot of configuration and tweaking before it becomes fully functional. Never the less, it is a good starting point for a new partner collection trying to set up their Harvester component.. We called this component the Reaper and it can be downloaded from here.