View source for Verification and validation of computer simulation models

'''Verification and validation of computer simulation models''' is conducted during the development of a [[simulation]] model with the ultimate goal of producing an accurate and credible model.<ref name ="Banks">Banks, Jerry; Carson, John S.; Nelson, Barry L.; Nicol, David M. ''Discrete-Event System Simulation'' Fifth Edition, Upper Saddle River, Pearson Education, Inc. 2010 {{ISBN|0136062121}}</ref><ref name = "Schlesinger">{{cite journal|author=Schlesinger, S.|year=1979|display-authors=etal|title=Terminology for model credibility|journal=Simulation|volume=32|issue=3|pages=103–104|doi=10.1177/003754977903200304}}</ref>  "Simulation models are increasingly being used to solve problems and to aid in decision-making. The developers and users of these models, the decision makers using information obtained from the results of these models, and the individuals affected by decisions based on such models are all rightly concerned with whether a model and its results are "correct".<ref name ="Sargent">Sargent, Robert G. [http://www.informs-sim.org/wsc11papers/016.pdf "VERIFICATION AND VALIDATION OF SIMULATION MODELS"]. Proceedings of the 2011 Winter Simulation Conference.</ref> This concern is addressed through verification and validation of the simulation model.

Simulation models are approximate imitations of real-world systems and they never exactly imitate the real-world system.  Due to that, a model should be [[Verification and validation|verified]] and validated to the degree needed for the model's intended purpose or application.<ref name ="Sargent" />

The verification and validation of a simulation model starts after functional specifications have been documented and initial model development has been completed.<ref name ="Carson" >Carson, John, [http://informs-sim.org/wsc02papers/008.pdf "MODEL VERIFICATION AND VALIDATION"]. Proceedings of the 2002 Winter Simulation Conference.</ref> Verification and validation is an iterative process that takes place throughout the development of a model.<ref name ="Banks" /><ref name ="Carson" />

== Verification ==
In the context of computer simulation, '''verification''' of a model is the process of confirming that it is correctly implemented with respect to the conceptual model (it matches specifications and assumptions deemed acceptable for the given purpose of application).<ref name ="Banks" /><ref name ="Carson" />
During verification the model is tested to find and fix errors in the implementation of the model.<ref name ="Carson" />
Various processes and techniques are used to assure the model matches specifications and assumptions with respect to the model concept. 
The objective of model verification is to ensure that the implementation of the model is correct.

There are many techniques that can be utilized to verify a model.
These include, but are not limited to, having the model checked by an expert, making logic flow diagrams that include each logically possible action, examining the model output for reasonableness under a variety of settings of the input parameters, and using an interactive debugger.<ref name ="Banks" />
Many software engineering techniques used for [[software verification]] are applicable to simulation model verification.<ref name ="Banks" />

== Validation ==
Validation checks the accuracy of the model's representation of the real system.  Model validation is defined to mean "substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model".<ref name ="Sargent" /> A model should be built for a specific purpose or set of objectives and its validity determined for that purpose.<ref name ="Sargent" />

There are many approaches that can be used to validate a computer model.  The approaches range from subjective reviews to objective statistical tests.  One approach that is commonly used is to have the model builders determine validity of the model through a series of tests.<ref name ="Sargent" />

Naylor and Finger [1967] formulated a three-step approach to model validation that has been widely followed:<ref name ="Banks" />

Step 1. Build a model that has high face validity.

Step 2. Validate model assumptions.

Step 3. Compare the model input-output transformations to corresponding input-output transformations for the real system.<ref>NAYLOR, T. H., AND J. M. FINGER [1967], [http://mansci.journal.informs.org/content/14/2/B-92 "Verification of Computer Simulation Models"], Management Science, Vol. 2, pp. B92– B101., cited in Banks, Jerry; Carson, John S.; Nelson, Barry L.; Nicol, David M. ''Discrete-Event System Simulation'' Fifth Edition, Upper Saddle River, Pearson Education, Inc. 2010 p. 396. {{ISBN|0136062121}}</ref>

=== Face validity ===
A model that has '''face validity''' appears to be a reasonable imitation of a real-world system to people who are knowledgeable of the real world system.<ref name ="Carson" /> Face validity is tested by having users and people knowledgeable with the system examine model output for reasonableness and in the process identify deficiencies.<ref name ="Banks" /> An added advantage of having the users involved in validation is that the model's credibility to the users and the user's confidence in the model increases.<ref name ="Banks" /><ref name ="Carson" /> Sensitivity to model inputs can also be used to judge face validity.<ref name ="Banks" /> For example, if a simulation of a fast food restaurant drive through was run twice with customer arrival rates of 20 per hour and 40 per hour then model outputs such as average wait time or maximum number of customers waiting would be expected to increase with the arrival rate.

=== Validation of model assumptions ===
Assumptions made about a model generally fall into two categories: structural assumptions about how system works and data assumptions. Also we can consider the simplification assumptions that are those that we use to simplify the reality.<ref name = "Fonseca i  Casas, P.">1. Fonseca, P. Simulation hypotheses. In Proceedings of SIMUL 2011; 2011; pp. 114–119. https://www.researchgate.net/publication/262187532_Simulation_hypotheses_A_proposed_taxonomy_for_the_hypotheses_used_in_a_simulation_model</ref>

==== Structural assumptions ====
Assumptions made about how the system operates and how it is physically arranged are structural assumptions.  For example, the number of servers in a fast food drive through lane and if there is more than one how are they utilized?  Do the servers work in parallel where a customer completes a transaction by visiting a single server or does one server take orders and handle payment while the other prepares and serves the order.  Many structural problems in the model come from poor or incorrect assumptions.<ref name="Carson" />  If possible the workings of the actual system should be closely observed to understand how it operates.<ref name="Carson" />  The systems structure and operation should also be verified with users of the actual system.<ref name="Banks" />

==== Data assumptions ====
There must be a sufficient amount of appropriate data available to build a conceptual model and validate a model.  Lack of appropriate data is often the reason attempts to validate a model fail.<ref name ="Sargent" />  Data should be verified to come from a reliable source.  A typical error is assuming an inappropriate statistical distribution for the data.<ref name="Banks" />  The assumed statistical model should be tested using [[goodness of fit]] tests and other techniques.<ref name ="Banks" /><ref name ="Sargent" /> Examples of goodness of fit tests are the [[Kolmogorov–Smirnov test]] and the [[chi-square test]].  Any outliers in the data should be checked.<ref name ="Sargent" />

==== Simplification assumptions ====
Are those assumptions that we know that are not true, but are needed to simplify the problem we want to solve.<ref name = "Fonseca i  Casas, P."/> The use of this assumptions must be restricted to assure that the model is correct enough to serve as an answer for the problem we want to solve.

=== Validating input-output transformations ===
The model is viewed as an input-output transformation for these tests.  The validation test consists of comparing outputs from the system under consideration to model outputs for the same set of input conditions.  Data recorded while observing the system must be available in order to perform this test.<ref name ="Sargent" /> The model output that is of primary interest should be used as the measure of performance.<ref name ="Banks" /> For example, if system under consideration is a fast food drive through where input to model is customer arrival time and the output measure of performance is average customer time in line, then the actual arrival time and time spent in line for customers at the drive through would be recorded.  The model would be run with the actual arrival times and the model average time in line would be compared with the actual average time spent in line using one or more tests.

==== Hypothesis testing ====
[[Statistical hypothesis testing]] using the [[Student's t-test|t-test]] can be used as a basis to accept the model as valid or reject it as invalid.

The hypothesis to be tested is
:H<sub>0</sub> the model measure of performance = the system measure of performance
versus
:H<sub>1</sub> the model measure of performance ≠ the system measure of performance.

The test is conducted for a given sample size and level of significance or α.  To perform the test a number ''n'' statistically independent runs of the model are conducted and an average or expected value, E(Y), for the variable of interest is produced.  Then the [[test statistic]], ''t''<sub>0</sub> is computed for the given α, ''n'', E(Y) and the observed value for the system  μ<sup>0</sup>

: <math>t_0 = {(E(Y)-u_0)}/{(S/\sqrt{n})}</math> and the critical value for α and n-1 the degrees of freedom

: <math>t_{a/2,n-1}</math> is calculated.

If 
: <math> \left\vert t_0 \right\vert > t_{a/2,n-1}</math>
reject H<sub>0</sub>, the model needs adjustment.

There are two types of error that can occur using hypothesis testing, rejecting a valid model called type I error or "model builders risk" and accepting an invalid model called Type II error, β, or "model user's risk".<ref name ="Sargent" /> The level of significance or α is equal the probability of type I error.<ref name ="Sargent" /> If α is small then rejecting the null hypothesis is a strong conclusion.<ref name ="Banks" /> For example, if α = 0.05 and the null hypothesis is rejected there is only a 0.05 probability of rejecting a model that is valid.  Decreasing the probability of a type II error is very important.<ref name ="Banks" /><ref name ="Sargent" /> The probability of correctly detecting an invalid model is 1 - β.  The probability of a type II error is dependent of the sample size and the actual difference between the sample value and the observed value.  Increasing the sample size decreases the risk of a type II error.

===== Model accuracy as a range =====
A statistical technique where the amount of model accuracy is specified as a range has recently been developed.  The technique uses hypothesis testing to accept a model if the difference between a model's variable of interest and a system's variable of interest is within a specified range of accuracy.<ref name="Sargent2">Sargent, R. G. 2010. "A New Statistical Procedure for Validation of Simulation and Stochastic Models." Technical Report SYR-EECS-2010-06, Department of Electrical Engineering and Computer Science, Syracuse University, Syracuse, New York.</ref> A requirement is that both the system data and model data be approximately [[Normal distribution|Normally]] [[Independent and identically distributed random variables|Independent and Identically Distributed (NIID)]].  The [[Student's t-test|t-test]] statistic is used in this technique.  If the mean of the model is μ<sup>m</sup> and the mean of system is μ<sup>s</sup> then the difference between the model and the system is D = μ<sup>m</sup> - μ<sup>s</sup>. The hypothesis to be tested is if D is within the acceptable range of accuracy. Let L = the lower limit for accuracy and U = upper limit for accuracy.  Then

:H<sub>0</sub> L ≤ D ≤ U
versus
:H<sub>1</sub> D < L or D > U

is to be tested.

The operating characteristic (OC) curve is the probability that the null hypothesis is accepted when it is true.  The OC curve characterizes the probabilities of both type I and II errors.  Risk curves for model builder's risk and model user's can be developed from the OC curves.  Comparing curves with fixed sample size tradeoffs between model builder's risk and model user's risk can be seen easily in the risk curves.<ref name="Sargent2" /> If model builder's risk, model user's risk, and the upper and lower limits for the range of accuracy are all specified then the sample size needed can be calculated.<ref name="Sargent2" />

==== Confidence intervals ====
Confidence intervals can be used to evaluate if a model is "close enough"<ref name ="Banks" /> to a system for some variable of interest.  The difference between the known model value, μ<sub>0</sub>, and the system value, μ, is checked to see if it is less than a value small enough that the model is valid with respect that variable of interest.  The value is denoted by the symbol ε.  To perform the test a number, ''n'', statistically independent runs of the model are conducted and a mean or expected value, E(Y) or μ for simulation output variable of interest Y, with a standard deviation ''S'' is produced.  A confidence level is selected, 100(1-α).  An interval, [a,b], is constructed by

: <math>a = E(Y) - t_{a/2,n-1}S/\sqrt{n} \qquad and \qquad b = E(Y) + t_{a/2,n-1}S/\sqrt{n}</math>,
where 
: <math>t_{a/2,n-1}</math>
is the critical value from the t-distribution for the given level of significance and n-1 degrees of freedom.
: If |a-μ<sub>0</sub>| > ε and |b-μ<sub>0</sub>| > ε then the model needs to be calibrated since in both cases the difference is larger than acceptable.
: If |a-μ<sub>0</sub>| < ε and |b-μ<sub>0</sub>| < ε then the model is acceptable as in both cases the error is close enough.
: If |a-μ<sub>0</sub>| < ε and |b-μ<sub>0</sub>| > ε or [[vice versa]] then additional runs of the model are needed to shrink the interval.

==== Graphical comparisons ====
If statistical assumptions cannot be satisfied or there is insufficient data for the system a graphical comparisons of model outputs to system outputs can be used to make a subjective decisions, however other objective tests are preferable.<ref name ="Sargent" />

== ASME Standards ==
Documents and standards involving verification and validation of computational modeling and simulation are developed by the [[American Society of Mechanical Engineers]] (ASME) Verification and Validation (V&V) Committee. ASME V&V 10 provides guidance in assessing and increasing the credibility of computational solid mechanics models through the processes of verification, validation, and [[uncertainty quantification]].<ref>[https://www.asme.org/products/codes-standards/v-v-10-2006-guide-verification-validation “V&V 10 – 2006 Guide for Verification and Validation in Computational Solid Mechanics”]. ''Standards''. ASME. Retrieved 2 September 2018.</ref> ASME V&V 10.1 provides a detailed example to illustrate the concepts described in ASME V&V 10.<ref>[https://www.asme.org/products/codes-standards/v-v-101-2012-illustration-concepts-verification “V&V 10.1 – 2012 An Illustration of the Concepts of Verification and Validation in Computational Solid Mechanics”]. ''Standards''. ASME. Retrieved 2 September 2018.</ref> ASME V&V 20 provides a detailed methodology for validating computational simulations as applied to fluid dynamics and heat transfer.<ref>[https://www.asme.org/products/codes-standards/v-v-20-2009-standard-verification-validation “V&V 20 – 2009 Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer”]. ''Standards''. ASME. Retrieved 2 September 2018.</ref> ASME V&V 40 provides a framework for establishing model credibility requirements for computational modeling, and presents examples specific in the medical device industry. <ref>[https://www.asme.org/events/vandv/program/v-v40-day “V&V 40 Industry Day”]. ''Verification and Validation Symposium''. ASME. Retrieved 2 September 2018.</ref>

== See also ==
* [[Verification and validation]]
* [[Software verification and validation]]

== References ==
{{Reflist}}

[[Category:Simulation software| ]]
[[Category:Formal methods]]