Data Rescue

To stimulate and coordinate data rescue activities, recently the International Data Rescue (I-DARE) Portal has been set up, http://www.idare-portal.org. The Portal provides a single point of entry for information on the status of past and present worldwide to be rescued data and data rescue projects, and on best methods and technologies involved in data rescue. The Portal is supervised by the WMO's Expert Team on Data Rescue, under the auspices of the Global Framework for Climate Services, and is operated by the Royal Netherlands Meteorological Institute (KNMI).

/Users/Vicky/Desktop/Mtl_McGillObs_Baro_Jan1_1877.png
An image of some of the sub-daily McGill pressure readings. Work in progress to follow Christa and get an online data rescue & digitization project going.

Every year the Australian Broadcasting Corporation (ABC) runs an ABC Science's National Science Week Citizen Science Project. I have submitted a proposal to get the thousands of Clement Wragge’s log book images I have photographed digitised as part of the project. My proposal has been successful. The project runs for the months of August. Christa Pudmenzky International Centre for Applied Climate Sciences (ICACS) University of Southern Queensland Toowoomba, Australia

In digitising the lighthouse records, the thermometer is so constant that we have to conclude the readings were done indoors. Can we presume these readings are still valid for the ISPD? thanks, Mac

We are digitising pressure readings from original lighthouse records. There are up to 14 sub daily readings. Is this level of detail useful for the reanalysis model or should we be aiming for somwthing less. The data entry person works rapidly so I suggest a useful answer would be either 1-3 readings only or 'all in".

Gil, I've just downloaded them, start at http://surfacetemperatures.blogspot.com.au/ and about a thrid the way down is the link (it's very long)
Mac

Mac, for land meteorological data: 1. We would love to have the images if nobody else can curate them. Or even if they can ... 2. We would take any elements of digitized data but particularly temperature. It is great to have data with a full provenance trail back to the original observation. There is a series of blogposts on the recently beta version released global land surface databank at http://surfacetemperatures.blogspot.com/ and a static summary at www.surfacetemperatures.org/databank/ . The latter includes a pdf describing how to submit the data. We would love to see some early Australian data. Good luck Peter

Just to put the strains on the good nature of the ispd folk, I'd like to propose a discussion on what we do with the images we create or discover during our data digitisation activities. Our project ensures we take high quality images of all our data and then store them in a retrievable format so they are available for future reference and as part of the audit trail. Importantly we are only digitising the surface pressure and some temperature data but often we pass by other structured and unstructured weather information that could have use in the future. In our case we have 50,000+ images which contain a wealth of background on Australian meteorology. I'm aware of very scattered repositories for these images, but nothing structural enough for us all to use or indeed to act as the final backstop for the Data Manifest I've suggested in the previous posting in this WIKI. We're not only creating a resource of data but we're also creating a resource of images - weather journals, ship's logs, records kept by early weather enthusiasts, government reports, explorers' journals, etc. Surely there is a curating job here that needs to be considered and acted on.

Any thoughts? Presumably the WMO should be taking an interest (!!).

On submitting documents to archives, a data manifest is usually included. For the purposes of the ispd, a manifest would be a super-metadata document that would describe each collection as it is submitted. This would give researchers the opportunity judge the quality of the data, be informed about it's provenance and notable issue associated with the digitising of that data. These issues may not have been important for the electronic data that originally populated the ispd, presumably it came from NMS datasets and the provenance was unassailable.

However, with the increasing shift to deep historical records, matters of interpretation are creeping in and issues of second and third hand data are appearing. For instance, our project group in Australia has been working from newsprint which raises issues of multiple handling, typos, rigours of data entry, etc. Though this might make some of you feel insecure, the ispd is becoming increasingly reliant on human systems as it moves away from the e-record.

Thus, I propose we establish a format for a dataset manifest that can referenced form any of its data items in the ispd. This would require another filed in the ispd, the manifest i.d.. It would also require the archiving and retrieval of manifests.

Having completed a career in Systems Analysis, I'm aware that you inevitably get one run at data uptake and you have to get it right, otherwise future generations will curse you. It's essential that you specify for future reference where the data came from, what its special properties are (eg. handwritten/typed, nms data or enthusiastic amateur, etc) and who/how it was digitised. These are the starting points of a manifest document. We have a standard set by the Australian National Archives that we've worked to in our imaging project, but it's too comprehensive for our needs.

Thus, I'd be interested in hearing from anyone with ideas on the subject. I'll then collate them for submission to Gil inclusion in the ispd submission guidelines.

In amongst the WMO recognised stations we are digitising, we have a clutch of smaller stations appearing erratically in our records. They don't appear in either of the ispd, WMO or local weather bureau station listings. However, we are able to identify geographically where the data was observed. Should we include these items with their best-guess lat/long and give them an ID code 9999... ?

Since our data is derived from secondary sources, sometimes with multiple handling involved, how can we flag that there is a degree of unreliability in it? I cannot find an explanation for the QC indicator referenced in this field, the "ISH data quality flag". Is this something we can use as a metadata quality flag?

Most of the readings we are digitising come from secondary sources, newspapers, synoptic charts, weather journals. Should there be a code here to reflect this environment or just use code 000?

(and that should do us, hopefully!)

Is this an item we should be concerned about or is it done at your end?

Given that competent 19th century meteorologists converted raw pressure readings to acceptable mslp's, should we be using code 1 here?

I've checked the UDUNIT text file and it's a complex choice for the correct reference to a reading done with a barometer reading units of mercury. Given that our observations were made by competent late 19th Century metaorologists, which of the codes do we use in field 19?

In an earlier post, it was indicated that these fields were adjusted to GMT. Since we're working with continent-wide historical data with no proof of how the observers determined their local time (this was before std time zones were set in Australia), we cannot truthfully convert to GMT. We note that there are no "original" time/date fields. Do we just make a best guess or do you have some way of making the conversion yourself?

A complication for us is that most observations are 9 am which, for eastern continental times, are most probably UTC plus 10. Thus, beginning of month observations have to be dated the last day of the previuos month which would require some fairly fancy software coding! thanks

Field 2, "Observation ID Type" shows the type of station ID in Field 1. We are using either the station ID in the ispd database or the one assigned on the Australian Bureau of Met website. What code(s) do we put in this field?

Fields 23, 24 & 25 (Latitude, Longitude and elevation) appear to be the same as fields 11, 12 & 13 "Observed lat/long/elevation"

Is there a list of UD UNITS compliant unit names?

Field 10 requests "Time Code". We are working with data 19th C data that used local time, generally based on their longitute, so do we enter "001"?

Field 42 asks for "International Surface Pressure Data Bank Collection ID". Do we leave enter 999999 if our organisation is not listed in the table? Or do we get assigned an ID.

Field 4, Pos: 19-22, of the submission guidelines is "Year (GMT) of the observation record". Is this the local year or is it corrected to GMT? eg. a 9am 1st Jan 1900 reading for a station that is GMT+10 would be corrected to GMT 1899?. Similar issues aries for month and day, hour.

I am interested with the comparison methods: the reanalysis value for comparison with station was obtained from the weighted average of the reanalysis values of the four grid boxes whose centers lie closest to the station. The average of the four grid boxes is obtained from the inverse distance weighted average. how to derive the four grid boxes and average of the four grid boxes is obtained from the inverse distance weighted average? Do people here have some codes for Matlab? Thanks.

Mac, This a great question. The ISPD contains the information on how the reanalysis system used the data, including whether there were rejections. This so-called "feedback" information is intended to do just what you suggest. The idea is to preserve all of the necessary metadata from the source so that the data can be returned to the source for just the sort of improvement you are mentioning. That full feedback step has not happened, yet, but it is possible, and we will be following up with the collection providers to do just that. thanks, gil compo

We do several levels of data cleaning to ready our data for submission to the ispd. Knowing that the ispd data-uptake system includes an even more refined form of data cleaning, further data outliers must be discovered. Presumably the ispd system can clean most of these outliers, BUT there must still be the occasional mysteries leftover. Do you 'return' these mysteries to the owners for possible resolution or do your just ignore those particular data points?

Though I'm sure the data-uptake routines of the ispd look for outliers in data banks that are submitted, but we would like to do our own data cleaning first because we may be able to make corrections on the spot before it's discovered by the ispd routines. Thus, if comparing data day to day, what is an acceptable range of daily variance in one station's pressure readings. We are working in inches of mercury, so would a daily change of +/- 1 be significant? 1.5 or 1.8 or 2? Is there a meteorologist who has an opinion on this? With the range defined, we can then determine the outliers and get to work checking the validity of our data entry, or indeed, the original observation.

Where can I get a succinct and refined list of pressure data already digitised for my area of interest? It would be ideal if this list is specific to a geographical area and a time period. With this in hand, I know we won't be expending effort on readings already digitised. BTW, the graphical listing at http://www.esrl.noaa.gov/psd/data/ISPD/ doesn't seem to show station names, meaning you have to guess what the dots on the map are.

I am not a weather professional, but I can see the logic and value of pressure data because it can be used to construct isobaric maps. I can see and touch this product of our digitising work. Why should my team of volunteers spend time digitising temperatures (rainfall, cloud, wind direction, river levels, etc)?

How useful is geographically contiguous data? Given that digitising data is a time-consuming event, how important to the reanalyses models is it that geograhically contiguous data is digitised? Are we wasting our time by doing a series of stations only 100/200/300/etc kms apart? Yes, you want EVERYTHING, but is there a priority issue here?

Where in this website do we find the definitive source of information on the long/lat of each station? Do we need to apply this data if we have applied the persistent station identifier (station id) for reading?

In some cases we have multiple data sources for the same reading. Occasionally they disagree. Do we submit all multiple readings, or submit sibgle readings and choose best fit for those that have conflicting readings, or dump conflicting readings all together?

Old pressure readings were recorded as inches of mercury. Do these have to be converted to pascals or is this done elctronically during the upload process?

We have dozens of stations digitised but a couple of them are consistently incorrect (they just don't tally with nearby stations). Do we attempt a correction ourselves (eg. determine average error and adjust series accordingly) or does the ispd input process identify and correct/quarantine these readings?

How important is it that our data is normalised to sea level, taking into account temperature, height above sea level, height of instrument, etc.? Do the data input processes of the ispd determine in some fashion if these activities have been carried out?