ASTM D6589 EXAMPLE MODEL EVALUATION OF THE AVERAGE CENTERLINE CONCENTRATION VALUE
ASTM D6589 Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance provides concepts that are useful for the comparison of modeled air concentrations with observed field data. Such comparisons provide a means for assessing a model's performance (e.g. bias and precision or uncertainty) and for determining whether a model’s skill is significantly better relative to other candidate models.
There are two primary concepts espoused by D6589:
1) Since all atmospheric transport and dispersion involves effects from random air currents and eddies, no model can predict explicitly what will occur from a release to the atmosphere. At best, models can predict the characteristics of the distribution of possible outcomes (e.g., mean, variance, frequency distribution). Stated another way, we cannot predict the sequence of outcomes from tossing a pair of dice, but we can test whether a pair of dice are ‘fair’ by comparing the characteristics of the distribution of outcomes (mean, variance, frequency of occurrence of each outcome) with what we know should occur.
2) The truest definition of “performance” is obtained through the comparison of a model’s skill with several other candidate models on several comprehensive data sets. Stated another way, you do not know who is the current fastest runner for the 100m dash, until you see the outcome for a race involving all the best 100m runners, running at the same time on several 100m courses.
AVAILABLE SOFTWARE AND DATA SETS
From this web page, you can download a zip file entitled D6589Archive (44.5MB). Please, let me know if you have problems downloading files this large. If so, I will try to break the archive apart into smaller sections.
The contents of this zip file implement the procedures discussed in D6589 for comparison of centerline concentration values from several dispersion models with observed values obtained at three field experiments: Project Prairie Grass, EPRI Kincaid and EPRI Indianapolis. More information on these field experiments will eventually be made available via this web site. Currently you can read a brief description of these data sets here (242KB).
D6589Archive.zip was originally made available from an Environmental Protection Agency (EPA) ftp site but they have discontinued this service. Hence, I am now making this file available from my own web site.
When D6589Archive is unzipped, it creates six (6) subdirectories. Below is a brief description of the subdirectories.
1. ASTMsoftware – Provides the FORTRAN code and executable to perform the comparisons described in the appendixes of D6589. In order to run the examples provided, you need to copy Design2.exe to a field experiments’ “ASTMEvaluation" subdirectory.
2. Documentation –Three (3) files are provided in the subdirectory. There is a description of the steps taken to characterize the meteorological inputs needed to drive the dispersion models. There is an Excel file that summarizes the meteorology developed for the three field experiments. There is a User’s Guide for the ASTM software (Design2.exe).
3. Indianapolis - has three subdirectories.
a) ASTMEvaluation - all files required to run ASTM software with example input and output files.
b) DispersionModeling - dispersion model results by model
c) Meteorology - the meteorology developed for each dispersion model
4. Kincaid - has three subdirectories.
a) ASTMEvaluation - all files required to run ASTM software with example input and output files.
b) DispersionModeling - dispersion model results by model
c) Meteorology - the meteorology developed for each dispersion model
5. Kincaid Reanalysis - As I understand it, there are two (2) “Kincaid” data sets. One was used to develop the Hesitant Plume Dispersion Model (HPDM). Another was reserved for final determination of model performance. The data file currently available from this web site is the one used for development of HPDM. The second Kincaid data set (call it the evaluation data set), has not yet been packaged for release to the public.
A tracer was injected into the Kincaid power plant stacks and samplers were placed along available roadways downwind of the stacks. The samplers were not placed precisely along “receptor rings,” hence the investigators assigned samplers to “receptor rings” and in some cases these assignments seemed questionable. In several cases, concentration values were discarded that by all appearances looked in accord with surrounding values.
I thought that perhaps objective criteria (or rules) might be a fairer way to assign samplers to the “receptor rings”, and might make better use of the available data. In this subdirectory you see the results of my endeavors.
As an interesting aside, when I have used the D6589 procedures to define model performance, HPDM performs better using the sampler assignments developed by HPDM developers, than using my assignments which are sterile and objective. This should not surprise you, as these data (and their “receptor ring” assignments) were used to develop HPDM, so in essence HPDM was “tuned” to perform best to these sampler assignments.
By way of this discussion, I am illustrating the sensitivity of model performance to something as innocuous as the assignment of samplers to “receptor rings.”
6. Prairie Grass - has three subdirectories.
a) ASTMEvaluation - all files required to run ASTM software with example input and output files.
b) DispersionModeling - dispersion model results by model
c) Meteorology - the meteorology developed for each dispersion model
Look for filenames that include "notes", as these provide brief descriptions of the file contents.
DISCUSSION
It is not a given that the concepts espoused by D6589 are universally accepted. There are some who yet insist that it is sufficient to compare the maximum concentration seen at each downwind distance with that simulated by the model. It is my opinion that such comparisons are only sufficient if there is convincing evidence that the variability in the observed maximum concentration value is small in comparison to the magnitude of the value of the maximum concentration.
As discussed in Irwin et al. (2007): Probabilistic Characterization of Atmospheric Transport and Diffusion. J. of Applied Meteorology and Climatology. (46):980-993, for averaging times of 1-hr the natural stochastic variability in concentration values is on the order of a factor of 2 times the value of the concentration value (i.e., too large for direct comparison of modeled and observed concentration values). ASTM D6589 was developed to provide a way of making valid model performance assessments when stochastic variability in concentration values is large.
The ASTM Standard D6589 was first published in 2000, and is reviewed and updated (as needed) every five (5) years. It represents a continuing effort to develop a consensus on how the performance of air quality models should be defined.
The Harmonization Initiative (which was begun in 1991) is another continuing effort to reach consensus on best practices in air quality characterization. As part of the Harmonization Initiative, conferences are held every 18 months to stimulate development of ideas and ultimately consensus on best practices. As part of this effort, a model validation kit was created to stimulate discussion on how best to evaluate model performance.