converting every member to netCDF

To utilize these directions, an account at NERSC is required. 

First, untar the convert scripts in your home area:

    cd ; tar xf ~cmccoll/convert_grb2_netcdf.tar

    cd ~/convert_grb2_netcdf; ./setup_directories.csh

Prior to post-processing, download the data at NERSC (takes about a day):

1. Login to a Data Transfer Node (ssh dtn01.nersc.gov or ssh dtn02.nersc.gov)

2. Enter your name indicating you are processing that year in the Post_Process_Table (you may need to request access). Take note of the experiment, stream_year and year you will be processing

3. Download the data (be extra careful to not use spinup data “stream_year" or "stream_year +1”):

   • cd $CSCRATCH/20CRv3_grib2

   • Check that all the data is available (Example for experiment 451 stream 1899 year 1904)

     • ~/convert_grb2_netcdf/check_grb2.csh 451 1899 1904

   • Get a listing in tape order of the files to download:

     • ./get_data_by_member.csh  451 1899 1904

   • Start the download:

     • ./hsi_YYYY.csh (where YYYY is the year of data you just downloaded, e.g. 1904)

4. Run a check that all the data is present
   • ./check_download.csh YYYY

Now you can submit the jobs to the queue for processing:

1. Login to Cori

2. Change directory to  cd ~/convert_grb2_netcdf

3. Submit all the jobs to the queue  ./submit_all.csh YYYY

4. Once complete check that all the members are completed and have the proper number of records:

   • cd $CSCRATCH/ncfiles_20CRv3

   • ./check_complete.csh YYYY ( YYYY is the year just post processed)

Follow output messages if errors; otherwise, move on to step 4 (below)

1. Error could be entire variables didn’t run:

• This requires (replacing YYYY with your year processing):
  • cd $HOME/convert_grb2_netcdf
  • cp table_of_variables_YYYY  table_of_variables
  • ./submit.csh YYYY
• and/or
  • cd $HOME/convert_grb2_netcdf
  • cp table_of_variables_fg_YYYY table_of_variables_fg
  • ./submit_fg.csh YYYY

2. Or, specific members don’t have the correct number of records:

• (if processing on Cori max to fix at one time is 32, edison 12. This means when copying the tables_to_fix you have to be careful how many rows you copy at one time and run, hopefully always less than 32 ;-) and you have to change the corresponding SBATCH -n to match this number.
• Removing the files that are incomplete (replacing YYYY with your year processing):
  • cd $CSCRATCH/ncfiles_20CRv3; ./rm_bad_members_YYYY
• Then resubmitting the now missing members:
  • cd $HOME/convert_grb2_netcdf
  • cp table_of_variables_to_fix_YYYY  table_of_variables_to_fix
  • wc -l table_of_variables_to_fix
  • vi qsub_plev_fix.csh (change SBATCH -n line to be this number of fix variables)
  • ./submit_fix.csh
• and/or
  • cd $HOME/convert_grb2_netcdf
  • cp table_of_variables_to_fix_fg_YYYY table_of_variables_to_fix_fg
  • ./submit_fg_fix.csh

3.  Rerun check_complete.csh

• cd $CSCRATCH/ncfiles_20CRv3; ./check_complete.csh YYYY

4.  Backup to the HPSS

• ./project_backup.csh YYYY

5.  Delete the original data

• rm -rf $CSCRATCH/20CRv3_grib2/YYYY/mem*