To utilize these directions, an account at NERSC is required.
First, untar the convert scripts in your home area:
cd ; tar xf ~cmccoll/convert_grb2_netcdf.tar
cd ~/convert_grb2_netcdf; ./setup_directories.csh
Prior to post-processing, download the data at NERSC (takes about a day):
-
Login to a Data Transfer Node (ssh dtn01.nersc.gov or ssh dtn02.nersc.gov)
-
Enter your name indicating you are processing that year in the Post_Process_Table (you may need to request access). Take note of the experiment, stream_year and year you will be processing
-
Download the data (be extra careful to not use spinup data “stream_year" or "stream_year +1”):
-
cd $CSCRATCH/20CRv3_grib2
-
Check that all the data is available (Example for experiment 451 stream 1899 year 1904)
-
~/convert_grb2_netcdf/check_grb2.csh 451 1899 1904
-
-
Get a listing in tape order of the files to download:
-
./get_data_by_member.csh 451 1899 1904
-
-
Start the download:
-
./hsi_YYYY.csh (where YYYY is the year of data you just downloaded, e.g. 1904)
-
-
- Run a check that all the data is present
- ./check_download.csh YYYY
Now you can submit the jobs to the queue for processing:
-
Login to Cori
-
Change directory to cd ~/convert_grb2_netcdf
-
Submit all the jobs to the queue ./submit_all.csh YYYY
-
Once complete check that all the members are completed and have the proper number of records:
-
cd $CSCRATCH/ncfiles_20CRv3
-
./check_complete.csh YYYY ( YYYY is the year just post processed)
-
Follow output messages if errors; otherwise, move on to step 4 (below)
-
Error could be entire variables didn’t run:
- This requires (replacing YYYY with your year processing):
- cd $HOME/convert_grb2_netcdf
- cp table_of_variables_YYYY table_of_variables
- ./submit.csh YYYY
- and/or
- cd $HOME/convert_grb2_netcdf
- cp table_of_variables_fg_YYYY table_of_variables_fg
- ./submit_fg.csh YYYY
-
Or, specific members don’t have the correct number of records:
- (if processing on Cori max to fix at one time is 32, edison 12. This means when copying the tables_to_fix you have to be careful how many rows you copy at one time and run, hopefully always less than 32 ;-) and you have to change the corresponding SBATCH -n to match this number.
- Removing the files that are incomplete (replacing YYYY with your year processing):
- cd $CSCRATCH/ncfiles_20CRv3; ./rm_bad_members_YYYY
- Then resubmitting the now missing members:
- cd $HOME/convert_grb2_netcdf
- cp table_of_variables_to_fix_YYYY table_of_variables_to_fix
- wc -l table_of_variables_to_fix
- vi qsub_plev_fix.csh (change SBATCH -n line to be this number of fix variables)
- ./submit_fix.csh
- and/or
- cd $HOME/convert_grb2_netcdf
- cp table_of_variables_to_fix_fg_YYYY table_of_variables_to_fix_fg
- ./submit_fg_fix.csh
3. Rerun check_complete.csh
- cd $CSCRATCH/ncfiles_20CRv3; ./check_complete.csh YYYY
4. Backup to the HPSS
- ./project_backup.csh YYYY
5. Delete the original data
- rm -rf $CSCRATCH/20CRv3_grib2/YYYY/mem*