Here is my current understanding of our process for assembling the new ISPD V4.3, using ICOADS 3.0, ISPD 4.0 and IBTrACS v03r08.
ICOADS 3.0 has applied a land mask that we want to ignore for the ship decks from the ACRE collection.
- inorder to use all the data from the ACRE I will need to remove the observations for those ship decks from the ICOADS3.0 (from 1870 - 1946) and blend ACRE data in.
This will include the old weather from the Oldweather US log books and the crowdsourced Weather Detective (DECK 710 and 711)
From Gil: there should also be another Oldweather related deck of UK royal navy.
From Chesley: I think you are referring to the Source ID ?
Here is the link to documentation for the ICOADS 3.0 Talking with Sandy Lubker she thinks the data used in ICOADS3 was downloaded from
This may be where they are:
These are the particular snippets for the DECK ID's resently added old weather oberservations
Newly added ICOADS SourceID's
Once the data is removed from the ICOADS the UniqueID they have will be lost... ** note that ICOADS has all entries that were not "trimmed" for land mask, even those that do not have a pressure observation
From Gil:The IMMA files provided by Philip Brohan would need to retain the ICOADS UniqueID
From Chesley: I see two problems with trying to maintain the UID from ICOADS
1) The record itself will have been altered for the SLP, so if the user were to look at ICOADS3 for that UID they would not see the same obervation
2) There will be additional oberservations that have no ICOADS UID for obs that were "trimmed"
We want to add a bias correction to all ICOADS observations, including the ACRE observations, (from 1780-1871)
One idea is to pull all the IMMA records from ICOADS3 that are from the specified Decks, merge with all the ACRE IMMA files, remove the duplicates that contain no UID, then edit the SLP values and merge back with the full marine collection.
or if we don't need the UID remove the data that are from the specified Decks, edit all the ACRE IMMA files with "corrected" SLP values and merge back. ***
or we could create a table of ShipName/YYYYMM/bias, and I would write a wrapper to check this file, if bias correction exists apply to the observation.
*** probably the easiest, most reliable way
The German Maury is included in the ICOADS3.0, some of the measurements have both temp and gravity corrections and some not, so not sure if you want to "remove" the Maury also, and add back the observations with the corrections you wanted applied.
From Gil: We would want to remove German Maury (Deck 721 SID 152) and replace with consistently corrected for gravity and temp.