Here is my current understanding of our process for assembling the new ISPD V4.3, using ICOADS 3.0, ISPD 4.0 and IBTrACS v03r08.
ICOADS 3.0 has applied a land mask that we want to ignore for the ship decks from the ACRE collection.
- inorder to use all the data from the ACRE I will need to remove the observations for those ship decks from the ICOADS3.0 (from 1870 - 1946) and blend ACRE data in.
This will include the old weather from the Oldweather US log books and the crowdsourced Weather Detective (DECK 710 and 711)
From Gil: there should also be another Oldweather related deck of UK royal navy.
From Chesley: I think you are referring to the Source ID ?
Here is the link to documentation for the ICOADS 3.0 Talking with Sandy Lubker she thinks the data used in ICOADS3 was downloaded from
This may be where they are:
https://github.com/oldweather/
https://github.com/oldweather/
https://github.com/oldweather/
https://github.com/oldweather/
Sandy
These are the particular snippets for the DECK ID's resently added old weather oberservations
Newly added ICOADS SourceID's
Once the data is removed from the ICOADS the UniqueID they have will be lost... ** note that ICOADS has all entries that were not "trimmed" for land mask, even those that do not have a pressure observation
From Gil:The IMMA files provided by Philip Brohan would need to retain the ICOADS UniqueID
From Chesley: I see two problems with trying to maintain the UID from ICOADS
1) The record itself will have been altered for the SLP, so if the user were to look at ICOADS3 for that UID they would not see the same obervation
2) There will be additional oberservations that have no ICOADS UID for obs that were "trimmed"
We want to add a bias correction to all ICOADS observations, including the ACRE observations, (from 1780-1871)
One idea is to pull all the IMMA records from ICOADS3 that are from the specified Decks, merge with all the ACRE IMMA files, remove the duplicates that contain no UID, then edit the SLP values and merge back with the full marine collection.
or if we don't need the UID remove the data that are from the specified Decks, edit all the ACRE IMMA files with "corrected" SLP values and merge back. ***
or we could create a table of ShipName/YYYYMM/bias, and I would write a wrapper to check this file, if bias correction exists apply to the observation.
*** probably the easiest, most reliable way
The German Maury is included in the ICOADS3.0, some of the measurements have both temp and gravity corrections and some not, so not sure if you want to "remove" the Maury also, and add back the observations with the corrections you wanted applied.
From Gil: We would want to remove German Maury (Deck 721 SID 152) and replace with consistently corrected for gravity and temp.
Re: Proposal for assembling the ISPD V4.3
It's not just the land mask - the ICOADS QC process is missing background fields for some areas (notably the polar latitudes) for older obs, but the upshot is that many of the early obs are rejected by the QC process for bad reasons. So for our early obs, because we have many useful obs in ports and at high latitude, we would do better to use all the obs, rather than just the obs retained by the ICOADS QC.
I think the simplest way to do this is to get both the released ICOADS3.0, and the 'all-observations file' for ICOADS3.0 (could limit to pre-1950 to save space), and just to swap sources - delete all of SID 165 (oW WW1 ships) in the released file and replace with SID165 in the all file. I'd do this for SIDs 165, 166, 167, 168, 148 128 and 247.
Both released and All files are at NCAR, though an NCAR account is necessary to get the all files (ask Sandy, or Steve Worley)
What does altered SLP mean?
Chesley,
Would you clarify what you mean by
I thought that SLP is in the ICOADS3 record. In the ISPD, we keep the original value as obtained from the source, as well as any corrected or adjusted value.
thanks for clarifying,
gil