Observations

To stimulate and coordinate data rescue activities, recently the International Data Rescue (I-DARE) Portal has been set up, http://www.idare-portal.org. The Portal provides a single point of entry for information on the status of past and present worldwide to be rescued data and data rescue projects, and on best methods and technologies involved in data rescue. The Portal is supervised by the WMO's Expert Team on Data Rescue, under the auspices of the Global Framework for Climate Services, and is operated by the Royal Netherlands Meteorological Institute (KNMI).

/Users/Vicky/Desktop/Mtl_McGillObs_Baro_Jan1_1877.png
An image of some of the sub-daily McGill pressure readings. Work in progress to follow Christa and get an online data rescue & digitization project going.

Every year the Australian Broadcasting Corporation (ABC) runs an ABC Science's National Science Week Citizen Science Project. I have submitted a proposal to get the thousands of Clement Wragge’s log book images I have photographed digitised as part of the project. My proposal has been successful. The project runs for the months of August. Christa Pudmenzky International Centre for Applied Climate Sciences (ICACS) University of Southern Queensland Toowoomba, Australia

In digitising the lighthouse records, the thermometer is so constant that we have to conclude the readings were done indoors. Can we presume these readings are still valid for the ISPD? thanks, Mac

We are digitising pressure readings from original lighthouse records. There are up to 14 sub daily readings. Is this level of detail useful for the reanalysis model or should we be aiming for somwthing less. The data entry person works rapidly so I suggest a useful answer would be either 1-3 readings only or 'all in".

Gil, I've just downloaded them, start at http://surfacetemperatures.blogspot.com.au/ and about a thrid the way down is the link (it's very long)
Mac

Mac, for land meteorological data: 1. We would love to have the images if nobody else can curate them. Or even if they can ... 2. We would take any elements of digitized data but particularly temperature. It is great to have data with a full provenance trail back to the original observation. There is a series of blogposts on the recently beta version released global land surface databank at http://surfacetemperatures.blogspot.com/ and a static summary at www.surfacetemperatures.org/databank/ . The latter includes a pdf describing how to submit the data. We would love to see some early Australian data. Good luck Peter

Just to put the strains on the good nature of the ispd folk, I'd like to propose a discussion on what we do with the images we create or discover during our data digitisation activities. Our project ensures we take high quality images of all our data and then store them in a retrievable format so they are available for future reference and as part of the audit trail. Importantly we are only digitising the surface pressure and some temperature data but often we pass by other structured and unstructured weather information that could have use in the future. In our case we have 50,000+ images which contain a wealth of background on Australian meteorology. I'm aware of very scattered repositories for these images, but nothing structural enough for us all to use or indeed to act as the final backstop for the Data Manifest I've suggested in the previous posting in this WIKI. We're not only creating a resource of data but we're also creating a resource of images - weather journals, ship's logs, records kept by early weather enthusiasts, government reports, explorers' journals, etc. Surely there is a curating job here that needs to be considered and acted on.

Any thoughts? Presumably the WMO should be taking an interest (!!).

On submitting documents to archives, a data manifest is usually included. For the purposes of the ispd, a manifest would be a super-metadata document that would describe each collection as it is submitted. This would give researchers the opportunity judge the quality of the data, be informed about it's provenance and notable issue associated with the digitising of that data. These issues may not have been important for the electronic data that originally populated the ispd, presumably it came from NMS datasets and the provenance was unassailable.

However, with the increasing shift to deep historical records, matters of interpretation are creeping in and issues of second and third hand data are appearing. For instance, our project group in Australia has been working from newsprint which raises issues of multiple handling, typos, rigours of data entry, etc. Though this might make some of you feel insecure, the ispd is becoming increasingly reliant on human systems as it moves away from the e-record.

Thus, I propose we establish a format for a dataset manifest that can referenced form any of its data items in the ispd. This would require another filed in the ispd, the manifest i.d.. It would also require the archiving and retrieval of manifests.

Having completed a career in Systems Analysis, I'm aware that you inevitably get one run at data uptake and you have to get it right, otherwise future generations will curse you. It's essential that you specify for future reference where the data came from, what its special properties are (eg. handwritten/typed, nms data or enthusiastic amateur, etc) and who/how it was digitised. These are the starting points of a manifest document. We have a standard set by the Australian National Archives that we've worked to in our imaging project, but it's too comprehensive for our needs.

Thus, I'd be interested in hearing from anyone with ideas on the subject. I'll then collate them for submission to Gil inclusion in the ispd submission guidelines.

In amongst the WMO recognised stations we are digitising, we have a clutch of smaller stations appearing erratically in our records. They don't appear in either of the ispd, WMO or local weather bureau station listings. However, we are able to identify geographically where the data was observed. Should we include these items with their best-guess lat/long and give them an ID code 9999... ?

Since our data is derived from secondary sources, sometimes with multiple handling involved, how can we flag that there is a degree of unreliability in it? I cannot find an explanation for the QC indicator referenced in this field, the "ISH data quality flag". Is this something we can use as a metadata quality flag?

Most of the readings we are digitising come from secondary sources, newspapers, synoptic charts, weather journals. Should there be a code here to reflect this environment or just use code 000?

(and that should do us, hopefully!)

Is this an item we should be concerned about or is it done at your end?

Given that competent 19th century meteorologists converted raw pressure readings to acceptable mslp's, should we be using code 1 here?

I've checked the UDUNIT text file and it's a complex choice for the correct reference to a reading done with a barometer reading units of mercury. Given that our observations were made by competent late 19th Century metaorologists, which of the codes do we use in field 19?

In an earlier post, it was indicated that these fields were adjusted to GMT. Since we're working with continent-wide historical data with no proof of how the observers determined their local time (this was before std time zones were set in Australia), we cannot truthfully convert to GMT. We note that there are no "original" time/date fields. Do we just make a best guess or do you have some way of making the conversion yourself?

A complication for us is that most observations are 9 am which, for eastern continental times, are most probably UTC plus 10. Thus, beginning of month observations have to be dated the last day of the previuos month which would require some fairly fancy software coding! thanks

Field 2, "Observation ID Type" shows the type of station ID in Field 1. We are using either the station ID in the ispd database or the one assigned on the Australian Bureau of Met website. What code(s) do we put in this field?

Fields 23, 24 & 25 (Latitude, Longitude and elevation) appear to be the same as fields 11, 12 & 13 "Observed lat/long/elevation"

Is there a list of UD UNITS compliant unit names?

Field 10 requests "Time Code". We are working with data 19th C data that used local time, generally based on their longitute, so do we enter "001"?

Field 42 asks for "International Surface Pressure Data Bank Collection ID". Do we leave enter 999999 if our organisation is not listed in the table? Or do we get assigned an ID.

Field 4, Pos: 19-22, of the submission guidelines is "Year (GMT) of the observation record". Is this the local year or is it corrected to GMT? eg. a 9am 1st Jan 1900 reading for a station that is GMT+10 would be corrected to GMT 1899?. Similar issues aries for month and day, hour.

I am interested with the comparison methods: the reanalysis value for comparison with station was obtained from the weighted average of the reanalysis values of the four grid boxes whose centers lie closest to the station. The average of the four grid boxes is obtained from the inverse distance weighted average. how to derive the four grid boxes and average of the four grid boxes is obtained from the inverse distance weighted average? Do people here have some codes for Matlab? Thanks.

Mac, This a great question. The ISPD contains the information on how the reanalysis system used the data, including whether there were rejections. This so-called "feedback" information is intended to do just what you suggest. The idea is to preserve all of the necessary metadata from the source so that the data can be returned to the source for just the sort of improvement you are mentioning. That full feedback step has not happened, yet, but it is possible, and we will be following up with the collection providers to do just that. thanks, gil compo

We do several levels of data cleaning to ready our data for submission to the ispd. Knowing that the ispd data-uptake system includes an even more refined form of data cleaning, further data outliers must be discovered. Presumably the ispd system can clean most of these outliers, BUT there must still be the occasional mysteries leftover. Do you 'return' these mysteries to the owners for possible resolution or do your just ignore those particular data points?

Though I'm sure the data-uptake routines of the ispd look for outliers in data banks that are submitted, but we would like to do our own data cleaning first because we may be able to make corrections on the spot before it's discovered by the ispd routines. Thus, if comparing data day to day, what is an acceptable range of daily variance in one station's pressure readings. We are working in inches of mercury, so would a daily change of +/- 1 be significant? 1.5 or 1.8 or 2? Is there a meteorologist who has an opinion on this? With the range defined, we can then determine the outliers and get to work checking the validity of our data entry, or indeed, the original observation.

Where can I get a succinct and refined list of pressure data already digitised for my area of interest? It would be ideal if this list is specific to a geographical area and a time period. With this in hand, I know we won't be expending effort on readings already digitised. BTW, the graphical listing at http://www.esrl.noaa.gov/psd/data/ISPD/ doesn't seem to show station names, meaning you have to guess what the dots on the map are.

I am not a weather professional, but I can see the logic and value of pressure data because it can be used to construct isobaric maps. I can see and touch this product of our digitising work. Why should my team of volunteers spend time digitising temperatures (rainfall, cloud, wind direction, river levels, etc)?

How useful is geographically contiguous data? Given that digitising data is a time-consuming event, how important to the reanalyses models is it that geograhically contiguous data is digitised? Are we wasting our time by doing a series of stations only 100/200/300/etc kms apart? Yes, you want EVERYTHING, but is there a priority issue here?

Where in this website do we find the definitive source of information on the long/lat of each station? Do we need to apply this data if we have applied the persistent station identifier (station id) for reading?

In some cases we have multiple data sources for the same reading. Occasionally they disagree. Do we submit all multiple readings, or submit sibgle readings and choose best fit for those that have conflicting readings, or dump conflicting readings all together?

Old pressure readings were recorded as inches of mercury. Do these have to be converted to pascals or is this done elctronically during the upload process?

We have dozens of stations digitised but a couple of them are consistently incorrect (they just don't tally with nearby stations). Do we attempt a correction ourselves (eg. determine average error and adjust series accordingly) or does the ispd input process identify and correct/quarantine these readings?

How important is it that our data is normalised to sea level, taking into account temperature, height above sea level, height of instrument, etc.? Do the data input processes of the ispd determine in some fashion if these activities have been carried out?

Dear Indira,

Great news! The new regional reanalysis area of reanalyses.org will be just the place to put some information and links to your new dataset. If you want to post some information before the dataset is released, please feel free to do so. Once the dataset is released, please let us know, and we'll put the information as a highlight on the homepage. 

Best wishes,

Hi,

We at National Centre for Medium Range Weather Forecasting (NCMRWF), India,  are running a re-analysis over the Indian Monsoon region known as Indian Monsoon Data Assimilation and Analysis (IMDAA) in collaboration with Met Office, UK. 

I am compiling a list of the documentation for the observation operator used by each reanalysis system to obtain surface pressure. For the 20CR, the H(x) to obtain surface pressure is documented in Compo et al., QJRMS, 2011, http://dx.doi.org/10.1002/qj.776 equations 5 to 7. Please add the appropriate reference for each system as a reply to this comment. I'll make a page with the results. best wishes, gil compo

Gil, I know I cleared this query under private cover, but herewith for the record: The two maps you refer to came from different meteorologists.- the large coloured one done under the management of Charles Todd, Astronomer and Meteorological Observer of the Colony of South Australia, while the newsprint map was done by his equal, Ellery (I believe), of the Colony of New South Wales. There were 6 colonies until Australian Federation in 1901 and each ran its own Met Service. (the true date was about 1909 when the Federation process got around to fully integrating these services). I suppose the take on these synoptic charts is the difference of interpretations of the data. It's a topic that contemporary Australian forecasters can discuss endlessly between each of the (now) States - all taking their slant on the "true synoptic situation". However, I have to admit that the two maps you have identified here seemingly have major differences. Not being qualified to comment, I leave it to my betters to judge the two meteorologists involved. On the Todd map, you can actually read the datapoints that both men would have used to construct their charts, so the data is there for contemporary analysis. And, of course, we have digitised a large amount of this data and it now resides in the ISPD.

There is no difference between the B&W and colour maps of the same date - they are the same page. In our workflow, if a page contained printed text, it was scanned (slow) for later OCR. If it contained graphics (eg. synoptic chart) it was photographed (fast) to retain the colour copy..... thus there is redundancy in the images. I will send you under separate cover a pdf of the manifest and post it to charlestodd.net at a later date.

Mac Benoy

Todd Project team, On many of the pages, there is more than one map of Australia and the contours are different. For example, at http://www.charlestodd.net/Todd_Hi-Res/1886/18860121c_hi-res.jpg what is the difference in the source between the apparent newspaper printed black and white map and the hand-drawn map with red isobars? Also, could the Manifest at http://www.charlestodd.net/Todd_Hi-Res/index1_html_files/Series%20description%20-%20for%20portable%20disk%20copies.jpg be made into a PDF? It currently is not searchable. thanks, gil compo University of Colorado/CIRES NOAA/Earth System Research Laboratory

Dears,

How can i obtain long-term wind data (1980-2018)  for the specific location Adama, Ethiopia

please?

I want to pose a series of questions regarding wave spectrum analysis with limited data and without the use of wave modeling software. Let's assume that the data sources is ICOADS, and that for a particular point in space and time, we have these two known variables: wave height (WH): 1 meter; swell height (SH): 2 meters Note: The wave and swell heights are reported in ICOADS as half-meters, but assume that I have already converted them to the values shown above in whole meters. Next, let's assume that the data were observed at sea in deep water, say, 250 m. If I want to determine the significant wave height (Hs or H 1/3), I can use this formula: Hs = ((WH^2) + (SH^2))^0.5 ≈ 2.236 m Then... Hmax = 2.0 (Hs) ≈ 4.472 m Question: From these known variables, how is maximum crest height (Hc) determined? During my review of DNV published literature, it was suggested that, "the maximum crest to wave height ratio for a Stokes wave is 0.635." Following that, is it correct to say that the maximum crest height, based on Hmax is: Hc = 0.635 (Hmax) ≈ 2.840 m Or, is there a more appropriate formula (and/or conversion ratio) to use? Again, please assume that there are no additional data available and that no wave spectrum modeling is available. I want to pose more questions along the same line, but I will stop with this one for now until it is resolved. Thank you.

Dear Sir, is it possible to obtain long series of daily short/long wave radiations measurements for mediterranean bacin and particularly for Egypt? Thank you for your kind reply Giuseppe Provenzano University of Palermo, Italy

For the marine surface go to http://rda.ucar.edu/datasets/ds540.0/ - sign in go do data access and create a subset - try it as often as you like. For 20C reanalysis, http://rda.ucar.edu/datasets/ds131.1, do as above For the ISPD, noted above, go to http://rda.ucar.edu/datasets/ds132.0, do as above For global surface observations, 1999 - ongoing, go to http://rda.ucar.edu/datasets/ds461.0, do as above There are many other option in the Research Data Archive - look around.

From the reanalysis datasets, see https://reanalyses.org/atmosphere/how-obtainplotanalyze-data for many options. Please post there with more questions about obtaining variables from reanalysis, and include more details: what time resolution, what period, what region?

How to obtain long term time series temperature, pressure and humidity data for any location... Thanks and regards, Suresh

Gil Compo encouraged me to post a note about Ameriflux and Fluxnet. Nearly 500 sites have run for anywhere for the past several to past 20 years monitoring near surface meteorology (T, RH, P, Precip), surface energy (H, LE) and carbon fluxes, soil properties (T and Q), and radiative fluxes. Because Fluxnet is a "cooperative" of independent sites, there is large variance in types of variables, instruments used, calibration protocols across networks. Still significant effort has gone into make this as interoperable as possible. Most of these data are now freely available, though it is encouraged to contact PIs of running sites to clarify any issues with data quality or site specific considerations. Ameriflux observations are now funded under a Dept of Energy core site facility model, which requires free access and frequent updating for most sites. These sites can be downloaded via http://ameriflux.lbl.gov/

Most useful for this group is the new Fluxnet2015 release, a harmonized global flux tower product, just released in early January 2016, with an update planned in April and December 2016. The new Fluxnet data release has a more open data policy, more consistent naming convention, and better job at detection of trends, energy balance closure, identification of QA/QC flags, gap filling, and uncertainty estimation. These can be downloaded at http://fluxnet.fluxdata.org/data/fluxnet2015-dataset/ After creating an account, all the data can be downloaded by making a web request, which sends a link to an FTP server.

I am happy to help be a conduit to any Ameriflux or Fluxnet datasets or help with understanding any issues you see in those. I run a set of 4 Ameriflux core forest/wetland sites in N Wisconsin/Michigan and 1 lake site in Madison, WI, including the Park Falls very tall tower with temperature and humidity profiles to 400 m. We provide fluxes and meteorology profiles nightly (near real-time) along with lots of ancillary data (sap-flow, soil profiles, vegetation inventories, leaf chemistry) and so on at: http://flux.aos.wisc.edu/twiki/bin/view/Main/ChEASData

Thanks. -ankur desai, UW-Madison, desai@aos.wisc.edu