Instructions for publishing AWS data on this website

Contents

1. Extract ascii data from argos_main

The first station is labelled "LGB00" in all the files relevant to that AWS. The next station to operate at that site is called "LGB00-A", and the last "LGB00-B". Should a new station operate there, it would be called "LGB00-C", the next "LGB00-D" and so on. We want to extract an ascii version of the all data for the first station "LGB00" from argos_main. To do this, one first needs to choose the data year to process by choosing an appropriate configuration file. Since LGB00 starts operation in 1982, that is the first year to process. Assuming you've already picked the 1982 config file, go to the "File" menu in argos_main, and choose "Show databases". The subsequent screen that appears will provide a summary of all databases for the year 1982, but you'll have to change to the updated set of databases by clicking on "List sensor database". To extract the ascii data of this database, click on "Save as text file", and choose to save all 12 months of data. Save the file using the following nomenclature LGB001982.txt. 

You should save data in a working directory named after the station, the location of which is listed under The Starcalc file and directory structure section below. 

2. Import ascii data into Staroffice

A GUI will appear on screen that asks how to allocate spreadsheet columns to the data. Choose "Fixed width", and manually add the columns to each field of the AWS data. Once this is done, open the dataset and a spreadsheet should pop up containing one year of the station's data. This LGB001982 spreadsheet contains a lot of extra characters and data not required. Below is an example of a similar raw spreadsheet to LGB001982. In the example, only the cells coloured grey are the ones to be kept. The others should be deleted. Note that "Battery voltage" is deleted, because it is not published. Solar power output (abbreviated "Spa") is also deleted should the station you are processing contain the Spa field, as are any other fields not to be published. A list of all fields that are published in the AWS dataset can be seen in Appendix C. Note that text output from the argos program no longer contains separate headings for each month, and there is no indication of the start of each month in the raw 12 month argos_main text output. This is not a problem, since the algorithm for translating time of observations (see section 3.2) automatically take this into account.

You should also delete any blank lines in the dataset, and make sure that only one heading of Day, Time etc is given at the top of the spreadsheet, and delete all other headings further down the file if they exist. In short, clean up spreadsheet so it does not contain any gaps, only has one heading, and only has the fields of use. Continuing our example, the spreadsheet above would end up looking like this: 

Once all cleaning up has been done for the given year's dataset, save it as a "Starcalc" file. As an example, one would save LGB001982 in the LGB00 working directory under the name LGB001982.sdc (the .sdc extension is given to Starcalc files). This importing and cleaning up step must be done for each individual year of the station's data before one can proceed to the next step, and it is important to be done thoroughly to avoid errors later. So, if you were to continue with the example of LGB00, you would go back to the previous step and extract ascii files for all the years of LGB00 operation (1982-1989), then turn each of them into clean Starcalc files. 

3. Process data in Staroffice

Algorithms to filter and convert data have already been set up in Starcalc, and creating a new dataset is mostly a matter of cutting and pasting the appropriate columns from another AWS Starcalc file to the one you are using. As always, when cutting and pasting algorithms in spreadsheets, exercise extreme caution to avoid inadvertent mistakes you may not detect. It is strongly recommended you copy all algorithms from /v/argos/awswebsite/staff/A028-B_1998_2000.sdc.Z (note this file needs to be uncompressed before opening). which has data from all sensors used on AWSs, contains the most refined algorithms, and has been carefully checked for errors. 

This section is written to guide you through setting up a new station's spreadsheet from scratch, and along the way explains what columns do what and why. The best way to understand it is to work through an example as you read the instructions. 

3.1 The Starcalc file and directory structure

• The datasets are very large to be manipulating in spreadsheets. Limit Starcalc file size to 2-3 years of data. 
• Run Staroffice on a computer containing at least 4GB of RAM.
• Never hand edit data in the Preprocessing spreadsheet, as it can cause significant problems in the algorithms used to filter data. 
• Never graph data in Staroffice as it may crash the computer.
• Cells coloured in blue in proceeding examples must be changed for each new Starcalc file.

The Preprocessing spreadsheet contains only "cleaned up" raw data previously discussed in section 2. The raw data prepared for each individual year should be concatenated to form a multiple year raw dataset. 

The Processing spreadsheet always has six initial columns that process the observation times. Of these columns, column "C" is automatically copied from the Preprocessing spreadsheet. This spreadsheet also contains copies of all other data from the Preprocessing spreadsheet, followed by several columns for each data type (e.g. temperature or pressure) which filter and process the data using algorithms discussed in sections 3.3 to 3.6 below.

The Postprocessing spreadsheet copies all columns with a heading ending in "(ed)" from the algorithms of the Processing spreadsheet. This spreadsheet contains only data to be published on this website. 

File nomenclature and directory location should be consistent so that it is immediately apparent what each file and directory contains. File nomenclature is {station name}_{starting year of data in file}_{ending year of data in file}{extension} where extension is .sdc for starcalc files and .txt for text files. The files should all be kept in a subdirectory named after the station being processed, together with all other files involved in processing data from that station. Remember to regularly save your work. The approximate memory required per year per station is 15 MB. Current (27/2/2002) file directory locations on Berg are:

• /v/aws (data)
• /v/argos/AWS (software)
• /v/argos/awswebsite (website)

3.2 Converting time to "days since 1980"

In this example of the first 6 columns of the A028-B_1998_2000.sdc "Processing" spreadsheet, the blue cells signify the cells you must change in each new Processing spreadsheet established. Cell A2 contains the first year of the data being processed. Change this as required and all other years will automatically change. Cell B2 contains the first month of the data. Change this as required, and each time a day number decreases, the month will click over to the new month. Beware that if a station's data misses an entire month, you will need to search through your data to find the place where this occurs, and manually change the month number of the cell where the skip occurs.

Once you have made these two changes, fill down (to do this go to the top menu: edit>fill>down) from row 25 to the number of rows in the Preprocessing spreadsheet. This action simply copies the time algorithm for each set of observations. The total number of rows in the Processing spreadsheet should then be entered in cell E1. 

You now need to setup the "Postprocessing" in your spreadsheet. Click over to that one in the A028-B_1998_2000.sdc file and see what appears in the first column: It is simply the "Days since 1980", because this is the timestamp we want to publish for each AWS observation. Set up a similar column for each row of data in your Postprocessing spreadsheet. You can do this by setting cell A1 of the Postprocessing spreadsheet to equal cell F1 of the Processing spreadsheet, and then filling down to the total number of rows you require to complete the time conversion. 

3.3 Filtering temperature, relative humidity and barometric pressure data

In this example, cell S1 is the raw air temperature at 4 metres equal to the value given on the "Preprocessing" spreadsheet. The header cells are given values as follows:

• T1 = (factor1)*(mean backward looking standard deviation)
• U1 = (factor2)*(mean forward looking standard deviation), 
• V1 = filtered data using the following algorithm:


A standard deviation is taken of the current observation and the previous five observations, regardless of observation time. These are the numbers in the T column starting at row 2. (Note, however that a backward looking standard deviation can only be taken when more than five observations exist, so the first five rows contain forward looking or centered looking standard deviations in cells T2:T6) 

A standard deviation is also taken of the current observation and the next six observations, regardless of observation time. These are the numbers in the U column starting at row 2. 

If both of the two standard deviations are greater than the values in their respective orange header cells, the datum is considered erroneous. If both standard deviations are zero, the datum is also considered erroneous. If one standard deviation is zero, and the other is greater than it's respective orange header cell, the datum is also considered erroneous and all erroneous values are automatically given the value 99999.9. All values not matching these criteria are accepted and appear in column V.

The tolerance on the number of standard deviations accepted past the mean standard deviation is controlled by changing the values of factor1 and factor2. Typical values for air temperature calculations are

factor1=1.0, factor2=3.5

for sub-surface temperatures are

factor1=5.0, factor2=5.0

and for pressure calculations are 

factor1=4.0, factor2=4.0

The larger the values of factor1 and factor2, the more sudden the jump in the trace is required to reject datum as erroneous. There is no reason for both factors to be the same, and they should simply be tuned to filter out the data you know to be suspect or erroneous. Factors may change between datasets, but have been found to be similar for all processed datasets. For appropriate values of the factors for relative humidity and electronics box temperature, please refer to the A028-B_1998_2000.sdc spreadsheet. 

Fill down this formula to the end of your Processing spreadsheet in order that all temperature data in your Preprocessing spreadsheet are copied across to your Processing spreadsheet. The header in your equivalent cell to V1 will automatically change, showing that you have addressed the correct column of your Preprocessing spreadsheet. 

Although the algorithm described here is applicable to all temperature and pressure data, there will be isolated problems with raw data that have not been taken into account here. For this reason the processing of AWS data was set up in spreadsheets rather than a C or Fortran program as it is easier to alter the algorithm in a spreadsheet. Two problems that will arise are wrapping or reflection of some sub-surface temperature data, and for the the station LGB20 only, a faulty barometer. For the latter case, a correction to raw pressure data must be applied, and this procedure is documented in Appendix A. For the occasional problem of wrapping or reflection in raw sub-surface temperature data, please refer to Appendix B. 

3.4 Processing pyranometer data

To install this algorithm into a new spreadsheet, copy the first 25 rows of these eight columns into the next available blank columns of your Processing spreadsheet. In the same manor as described in the previous subsection, change the reference of the raw data cell G1 to the correct column containing raw pyranometer data in the Preprocessing spreadsheet, and then fill down the eight columns from row 25 to the last row of your Processing spreadsheet.

Notice that the above figure shows two blue cells and one orange cell. The blue cell H1 is the maximum value of pyranometer data allowed when processing in argos_main. You must manually change this number in your new spreadsheet: To find out what the number should be, graph the pyranometer data in argos_main for any year of the station's data you are processing. Find out the y-axis range in argos_main for that graph, and the maximum number of that range is the number to be entered in cell H1. The blue cell J1 is the decimal longitude of the station in degrees east from north. This number controls the time calculated as local solar midnight and it is imperative it is changed for each new station.

The orange cell M1 controls the filtering of pyranometer data for errors. Filtering pyranometer data for errors is difficult, as it cannot be done using variance since the data has a natural high variance due to diurnal oscillations. Neither can the filtering be done using change in variance, because the data has a high change in variance as a result of seasonal variation of solar zenith angle and daily variation of downward solar radiation affected by cloud. The algorithm adopted here searches the data to see if the day's maximum solar irradiance is greater than a median of surrounding maximums by the quantity in the orange cell M1 (set at 13 in the A028-B_1998_2000.sdc spreadsheet). If so, all data for that day are filtered out. This filtering is only effectual for extreme errors in data.

Using this processing, there is a data loss resulting from the limit of the number of years that can be processed in one Starcalc file. You may notice that the first few days of data are lost at the beginning of the pyronometer data series in each file. There are ways around this problem, however they tend to create several other problems in spreadsheets. Therefore, the data loss has been incurred for the present until a need arises to fix the problem.

Be aware that pyranometers in the Antarctic may collect snow, particularly during winter, and consequently strange results by arise. One such case where snow is suspected of resting on the pyranometer at the end of winter, while slowly being melted off by the time of the summer solstice is for LGB20 during 1991 and 1992 shown in the graph below:

3.5 Filtering wind speed and wind direction data

Wind speed and wind direction data often have errors during winter when temperatures are cold enough to increase the viscosity of lubricants in wind vanes and anemometers. These instruments may also seize as a result snow or ice accumulation on the instruments. A sign of wind vanes and anemometer freeze-up is a sudden jump in the respective data trace, often followed by a zero data trace. That is, the data trend is well in excess of the mean trend of a station's total wind direction or wind speed record, followed by no trend at all. This error is filtered out using similar algorithms to those used for temperature, pressure and humidity data as explained in section 3.3.

For anemometers, another sign of freeze-up is the decay of wind speed with decreasing surface height in excess of the logarthmic decrease expected. This means that as wind speeds recorded at 1m and 2m decrease to less than about 0.8*(wind speed at 4m), the anemometers are beginning to seize, and the data can be treated as erroneous. This is calculated using a roughness length between 0.01mm and 1mm, and takes into account the fact that an AWS is slowly buried with accumulating snow. An explanation of the wind speed filtering for the highest anemometer (3m or 4m) spreadsheet cells is provided here:

In filtering wind direction data, we can assume that high data variance is most likely at low wind speeds for "light and variable" winds, but for moderate and greater wind speed, lower directional variance is usual. Hence a cuttoff is used for which variable wind direction is accepted. Since the wind direction can pass through 360 degrees under normal operation, it is really the variance of change in wind direction used to filter data, rather than simply wind direction. In other words, a change in wind direction between one observation of 355 degrees and the next of 5 degrees is 10 degrees. An explanation of the wind direction filtering spreadsheet cells is provided here:

In treating the alorithms for wind direction and wind speed together, it should be noted the there is cross referencing between cells in the wind direction filter and wind speed filters in A028-B_1998_2000.sdc, and so one must copy the first 25 rows of columns AV through BP from that spreadsheet. Note that the columns of the highest-above-surface wind speed are those used in filtering wind direction and closer-to-ground wind speeds. So all other wind filtering algorithms pivot around the 4m (or 3m) wind speed filter routine. If you intend to only use some of the wind data algorithms from A028-B_1998_2000.sdc, be sure the correct cells of the 4m wind speed filter routine are referenced by remaining algorithms after you have made your changes.

Once you have copied the wind filtering algorithm from A028-B_1998_2000.sdc and changed the referencing to the appropriate data fields on the Preprocessing spreadsheet, fill down to the base of your data, as explained in section 3.2. Make sure a duplicate of the edited data is set up in the Postprocessing spreasheet.

3.6 Processing acoustic accumulation data

The data must have basic temperature corrections applied, and this is done using the 4m air temperature. A lot of noisy data can also be removed by only accepting observations below a certain windspeed, so that blowing snow is less abundant, the wind speed is set in the spreadsheet as shown below. Temperature corrections to data are done using the formula:

An extract of the top 10 rows of the accumulation sensor algorithm from A028-B_1998_2000.sdc is shown in the figure below. In the figure, note that the two orange cells and the columns BV and BW serve the same purpose as the filtering columns for barometric pressure, temperature and relative humidity documented in section 3.3 above. There are also five blue cells in column BZ which must be changed for each new station. The green cells only indicate what data is being used to correct the sensor for air temperature variation and to filter out high wind speed data. These green cells are not changed by the user but will change according to what values are inserted in the five blue cells in column BZ.

Definitions of the values in the blue cells are:

• Initial Height: The height at which the sensor was set when it was installed (taken from first observation).
• Initial Temp: The temperature of the first observation after installation used to correct the initial height. Initial Temp and Initial Height provide the Correct height in the next cell down which is used as the reference for accumulation.
• Column of Ait4(ed): The number of columns right of column A in which the Ait4(ed) data appears in the Processing spreadsheet. When set correctly, this will automatically change the top green row to Ait4(ed).
• Column of WD4(ed): The number of columns right of column A in which WM4(ed) data appears in the Processing spreadsheet. When set correctly, this will automatically change the bottom green row to WM4(ed).
• Max wind speed: The windspeed above which data is not published (data set at 99999.9 in such cases).

You should copy the first 25 rows of columns BU through CB from the A028-B_1998_2000.sdc spreadsheet when using this algorithm, and then fill down only columns BU through BX to the base of your spreadsheet to obtain edited accumulation data (denoted Acc (ed)). Of course, the necessary change to the referenced raw data in the Preprocessing spreadsheet must be made in column BU as described for barametric pressure, temperature and relative humidity documented in section 3.3. Similarly, you must duplicate the Acc (ed) data in the Postprocessing spreadsheet in order to complete the processing of accumulation data.

4. Extract processed ascii data from Staroffice

Check you have done these things before extracting data from Staroffice:

• Are the number of rows the same in each of the Preprocessing, Processing and Postprocessing spreadsheets? 
• Have you changed all blue cells in your spreadsheet to values unique to the station you are processing?
• Is the start year given in cell A2 of the processing spreadsheet correct? 
• Is the start month given in cell B2 of the processing spreadsheet correct? 
• Is the last observation's year and month correct for the station?
• Are the correct number of data rows in the spreadsheet given in cell E1? 

There is a significant problem that Staroffice can sometimes write error messages in output text files, even though no such error exists. This is a software problem that is best fixed by 'getting out of the software and getting in again', in the same manor adopted by software engineers to fix broken cars (joke!). The best way to check if the error has occurred is to open up your text output file and run a search on "Err" in the file. A non-erroneous file will not yield a find. If the problem persists, physically copy the Pyronometer data to the Postprocessing spreadsheet and save as text. Remember NOT to save this altered version as your calculating spreadsheet. Another way of fixing the problem is to change the orange cell of the pyronometer header, and the change it back to the original number. This sometimes works.

You must obtain text output files for each set of data processed in individual starcalc files for the station. LGB20, for example, had five such text output files at the time of writing these instructions: LGB20_1991_1992.txt, LGB20_1993_1995.txt, LGB20_1996_1997.txt, LGB20_1998_2000.txt and LGB20_2001_.txt. 

5. Convert processed ascii data into netcdf format

Converting Starcalc text output discussed in the previous section to netcdf is easy, because a Fortran 95 program exists to do it for you. In fact, it is the same program available for download in the software section of this website under "Software to convert automatic weather station netcdf datafiles to text". Although unknown to the general user, this software can also convert text to netcdf when a "-w" option is specified on the command line. This software is already set up as "antarcticaws" on the Glaciology system and shouldn't have to be installed. Once you have checked the executable exists by typing "antarcticaws" at the unix prompt (you'll get an error message rather than "command not found"), you need to create an input file for your particular station that contains all the essential information on the station. The file is in a special format, and an example of the text input file for GC41 is given here: GC41.input.

In the file, all information after "!" on a line is a comment and will not be read by the software. Information under "&INFO" must be changed to the station name you are processing, the location of the station, the elevation, and the total number of sensors you are publishing with the dataset, respectively. Information under "&VARIABLES" indicates the column order in which stations appear in your text output files after the time column. If the sensor does not exist, set the sensor number to zero. All information under "&VARIABLES" must also be changed for each new station, however all input files have already been created for existing stations, and are kept under their appropriate directories. 

The text datafiles from each Starcalc text output must be concatenated to create a text file of the entire dataset. Here the example of LGB20 is used, and you should repeat this procedure for any stations your are processing. All commands are in unix.

Before copying your netcdf file to the web page, check the data thoroughly, as errors not obvious by poring over the data in a spreadsheet often exist. The best way to do this is to graph the data using the NCAR tool provided at this website, and the next section explains how to use that software to check data and publish it on this website.

6. Publish on the AWS website and graph data with NCAR graphics

To check the data, plot your netcdf dataset in graphaws as described on the software page. To do a broad check of the data, plot each variable for the entire period (multiple years) on an individual graph. This usually shows up most large errors. You can also plot any three variables for a single year on the same plot, and this is also a good way of doing more detailed error checking. The program gives you the option to output the data in postscript if you would like to print any of the graphs. Once you are satisfied with your dataset, and don't need to fix data by jumping back to step 3, copy the netcdf file to /v/argos/awswebsite/data.

Now construct an HTML file of the station's data to publish on this website. graphaws does this for you. Select to plot a single year of the data, and then the program will ask you if you require and HTML page. Answer "y" for yes. You will be asked what year you want to plot, and you should answer with the first year of the dataset (for LGB20, this would be 1991). A series of questions will be asked about station location and elevation, and the number of years in the dataset, and when you are asked which data to plot, you must choose 4m air temperature as the first graph, barometric pressure as the second graph, and wind speed as the third graph. When giving the ranges of the graphs, choose values that accommodate the entire dataset, not just the first year's data. Upon plotting the data, the program will automatically plot all subsequent years in the dataset, and write a web page in the background which will have the nomenclature "stationname.html". Copy this file to /v/argos/awswebsite. 

You now need to convert the graphics output to "gif" files. The graphics output is in the NCAR cgm (ncgm) format, and this can be converted to gif files by first splitting the output file, called "gmeta", into individual frames. You do this by typing med gmeta at the command line, and then at the prompt typing split {number of years in dataset}. This will place files called "med{frame number}.ncgm" in your directory. Quit out of med, and then rename each of these files to the name of the station, followed by the year the frame represents. For example, for LGB20, the dataset from 1991 to 2001 would have 11 frames, and so one would enter the following sequence of commands:

Each of these files is an individual ncgm file, and must be converted to a gif file using the following ncgm2gif script (which can be download by clicking here):

Continuing with the LGB20 example, the following sequence of commands would be entered to convert the files to gif format:

The ncgm2gif script places gif files in your directory with the nomenclature {station name}{year}.gif. Copy these gif files to /v/argos/awswebsite/graphs. A postscript version of the gmeta file must also be constructed, but this will contain all the frames in a single file. To do this first move gmeta to a file of the station name. So, for example, LGB20's gmeta file would be renamed LGB20. Now use the ncgm2ps script (which you can download here): 

Type at the command line (again using the LGB20 example): ncgm2ps LGB20 which will provide an output file called LGB20.ps. This is the printable version of the graphs available at the website, and this should also be copied to /v/argos/awswebsite/graphs.

Now move to the /v/argos/awswebsite directory, and open the data web page datapage.html. You will notice, as you scroll down, that there are java scripts set up in the data table to open windows on each station's data. Make sure one is setup for your new dataset, and also be sure it is named uniquely.

Finally, go to the web page, and check your new dataset can be viewed, and all the files can be downloaded. Tidy up your working directory so others can come after you and follow your work. 
 

Appendix A: Corrections to raw LGB20 barometric pressure data

Appendix B: Problems that may occur with sub-surface temperature data

B.1 Wrapped data

In the case of wrapped data, the data has reached a minimum (maximum) value, and for some reason, has wrapped to the maximum (minimum) value and continued the data record. An example of is provided here:

B.2 Reflected data

Appendix C: List of all fields published in the dataset (listed by abbreviation)

Appendix D: Ongoing processing notes

1. Berg B9B - Andrew Roberts 11/05/2004

2. STATION - NAME DAY/MONTH/YEAR

3. STATION - NAME DAY/MONTH/YEAR

4. STATION - NAME DAY/MONTH/YEAR

5. STATION - NAME DAY/MONTH/YEAR

6. STATION - NAME DAY/MONTH/YEAR

7. STATION - NAME DAY/MONTH/YEAR

8. STATION - NAME DAY/MONTH/YEAR