On Credibility of Simulation Studies of Telecommunication Networks

[attachment=18436]

INTRODUCTION
The last decade of the 20th century will be
remembered as a time when computers found
their place in primary schools and private
homes, and became ordinary items of equipment
on desks in offices and businesses. This is
also a time when the computing paradigm
began its drift from computer networks to network
computing. There is enormous interest, in
both industry and academia, in creating an
AAA network: a worldwide computer network
able to offer Any information service, accessible
from Any place and at Any time. Before it
happens, scientists and engineers will have to
investigate many challenging problems of network
technology and evaluate possible solutions.
These research activities are certainly
accelerated by achievements in the area of scientific
computing, with various easy-to-use software
packages specifically designed for
conducting performance evaluation studies of
telecommunication networks.



THE ISSUE OF CREDIBILITY
P. J. Kiviat, in his opening address of the Summer
Computer Simulation Conference (SCSC ’90) [2],
stated that “… succeeding in simulation requires
more than the ability to build useful models….”
Some experts assess that the modeling phase of
simulation consumes only 30–40 percent of the
total effort in most successful simulation projects.
The first necessary step of any performance evaluation
studies based on stochastic simulation is to
use a valid simulation model. In the case of
telecommunication networks, it means a valid
conceptual model of the network, based on appropriate
assumptions about the network’s internal
mechanisms, limitations, stochastic characteristics
of processes that will be simulated, and so on. A
good discussion of general guidelines on how to
build valid simulation models can be found, for
example, in [3]. However, this is only the first step
in ensuring the credibility of the final results of
simulation studies.


SIMULATION OUTPUT DATA ANALYSIS
Any stochastic computer simulation, in which random
processes are simulated, has to be regarded
as a (simulated) statistical experiment and,
because of that, application of statistical methods
of analysis of (random) simulation output data is
mandatory. Otherwise, J. Kleijnen of the University
of Tilburg, the Netherlands, warns that “…computer
runs yield a mass of data but this mass may
turn into a mess <if the random nature of such
output data is ignored, and then> …instead of an
expensive simulation model, a toss of the coin had
better be used” [10]. Von Neumann, having noticed
a similarity between computer simulators producing
random output data and a roulette, coined the
term Monte Carlo simulation.


FINAL COMMENTS
In this article we have considered two important
necessary conditions of credible simulation studies:
use of appropriate pseudo-random generators
of independent uniformly distributed
numbers, and appropriate analysis of simulation
output data. Our survey of recent research publications
in the area of telecommunication networks
suggests that the majority of recently
published results of simulation studies do not
satisfy the basic criteria of credibility. The situation
could easily be corrected if scientists and
engineers who apply stochastic simulation as the
tool for studying performance of networks accept
fuller responsibility for credibility of their results.
An adoption of the basic guidelines indicated in
the previous section would be the first step in
this direction.