A number of methodological papers published during the last years
testify that a need for a thorough revision of the research
methodology is felt by the operations research community – see, for
example, [Barr et al., J. Heuristics1 (1995) 9–32; Eiben and Jelasity,
Proceedings of the 2002 Congress on Evolutionary Computation
(CEC'2002) 582–587; Hooker,
J. Heuristics1 (1995) 33–42; Rardin and Uzsoy,
J. Heuristics7 (2001) 261–304]. In particular, the
performance evaluation of nondeterministic methods, including widely
studied metaheuristics such as evolutionary computation and ant colony
optimization, requires the definition of new experimental protocols.
A careful and thorough analysis of the problem of evaluating
metaheuristics reveals strong similarities between this problem and
the problem of evaluating learning methods in the machine learning
field.
In this paper, we show that several conceptual tools commonly used in
machine learning – such as, for example, the probabilistic notion of
class of instances and the separation between the training and the
testing datasets – fit naturally in the context of metaheuristics
evaluation.
Accordingly, we propose and discuss some principles inspired by the
experimental practice in machine learning for guiding the performance
evaluation of optimization algorithms.
Among these principles, a clear separation between the instances that
are used for tuning algorithms and those that are used in the actual
evaluation is particularly important for a proper assessment.