Journal of Applied and Computational Mechanics
Performance measure and tool for benchmarking metaheuristic optimization algorithms
Document Type : Research Paper
Authors
1 Percipio Robotics, Maison des Microtechniques 18, rue Alain Savary, Besançon, France
2 ICB UMR 6303, CNRS, UBFC, UTBM, Belfort, France
3 FEMTO-ST institute, Univ. Bourgogne Franche-Comté, CNRS, ENSMM Time and frequency dept., Besançon, France
4 My-OCCS, Besançon, France
5 Univ. Savoie Mont Blanc, SYMME, FR-74000 Annecy, France
Abstract
In the last decade, many new algorithms have been proposed to solve optimization problems. Most of them are meta-heuristic algorithms. The issue of accurate performance measure of algorithms is still under discussion in the scientific community. Therefore, a new scoring strategy via a new benchmark is proposed. The idea of this new tool is to determine a score, a measure of efficiency taking into account both the end value of the optimization and the convergence speed. This measure is based on an aggregate of statistical results of different optimization problems. These problems are judiciously chosen to cover as broad a spectrum of resolution configurations as possible. They are defined by combinations of several parameters: dimensions, objective functions and evaluation limit on dimension ratios. Aggregation methods are chosen and set in order to make the problem weight relevant according to the computed score. This scoring strategy is compared to the CEC one thanks to the results of different algorithms: PSO, CMAES, Genetic Algorithm, Cuttlefish and simulated annealing.
Keywords
Main Subjects
Publisher’s Note Shahid Chamran University of Ahvaz remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
[2] Nguyen, A.T., Reiter, S., Rigo, P., A review on simulation-based optimization methods applied to building performance analysis, Applied Energy, 2014, 113(Supplement C), 1043 – 1058.
[3] Roy, R., Hinduja, S., Teti, R., Recent advances in engineering design optimisation: Challenges and future trends, CIRP Annals, 2008, 57(2), 697–715.
[4] Wolpert, D.H., Macready, W.G., No free lunch theorems for optimization, IEEE Transactions on Evolutionary Computation, 1997, 1(1), 67–82.
[5] García-Martínez, C., Gutiérrez, P.D., Molina, D., Lozano, M., Herrera, F., Since cec 2005 competition on real-parameter optimisation: a decade of research, progress and comparative analysis’s weakness, Soft Computing, 2017, 21(19), 5573–5583.
[6] De Jong, K.A., Analysis of the behavior of a class of genetic adaptive systems, Ph.D. thesis, University of Michigan, USA, 1975.
[7] Liu, Q., Gehrlein, W.V., Wang, L., Yan, Y., Cao, Y., Chen, W., Li, Y., Paradoxes in numerical comparison of optimization algorithms, IEEE Transactions on Evolutionary Computation, 2020, 24(4), 777–791.
[8] Cheng, R., Li, M., Tian, Y., Zhang, X., Yang, S., Jin, Y., Yao, X., Benchmark functions for the cec 2017 competition on many-objective optimization, Tech. Rep., University of Birmingham, UK, 2017.
[9] Hansen, N., Auger, A., Finck, S., Ros, R., Real-parameter black-box optimization benchmarking 2010: Experimental setup, Research Report RR-7215, INRIA, 2010.
[10] Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., Brockhoff, D., Coco: a platform for comparing continuous optimizers in a black-box setting, Optimization Methods and Software, 2021, 36(1), 114–144.
[11] Hansen, N., Auger, A., Mersmann, O., Tusar, T., Brockhoff, D., Coco: A platform for comparing continuous optimizers in a black-box setting, 2016, arXiv e-prints, arXiv:1603.08785.
[12] Hansen, N., Tusar, T., Mersmann, O., Auger, A., Brockhoff, D., Coco: The experimental procedure, 2016, arXiv e-prints, arXiv:1603.08776.
[13] Garden, R.W., Engelbrecht, A.P., Analysis and classification of optimisation benchmark functions and benchmark suites, 2014 IEEE Congress on Evolutionary Computation (CEC), IEEE, 1641–1649.
[14] Weise, T., Chiong, R., Lassig, J., Tang, K., Tsutsui, S., Chen, W., Michalewicz, Z., Yao, X., Benchmarking optimization algorithms: An open source framework for the traveling salesman problem, IEEE Computational Intelligence Magazine, 2014, 9(3), 40–52.
[15] Chamoret, D., Salmon, S., Di Cesare, N., Xu, Y.J., Bsg-starcraft radius improvements of particle swarm optimization algorithm: An application to ceramic matrix composites, P. Siarry, L. Idoumghar, J. Lepagnot, eds., Swarm Intelligence Based Optimization, Springer International Publishing, 166–174.
[16] Beiranvand, V., Hare, W., Lucet, Y., Best practices for comparing optimization algorithms, Optimization and Engineering, 2017, 18(4), 815–848.
[17] Collignan, A., Méthode d’optimisation et d’aide à la décision en conception mécanique : Application à une structure aéronautique, Ph.D. thesis, Université Sciences et Technologies - Bordeaux I, 2011.
[18] Gomes, W.J., Beck, A.T., Lopez, R.H., Miguel, L.F., A probabilistic metric for comparing metaheuristic optimization algorithms, Structural Safety, 2018, 70, 59–70.
[19] Huang, Y., Jin, Y., Ding, Y., New performance indicators for robust optimization over time, 2015 IEEE Congress on Evolutionary Computation (CEC), 1380–1387.
[20] Quirante, T., Sebastian, P., Ledoux, Y., A trade-off function to tackle robust design problems in engineering, Journal of Engineering Design, 2013, 24(1), 64–81.
[21] Mobin, M., Mousavi, S.M., Komaki, M., Tavana, M., A hybrid desirability function approach for tuning parameters in evolutionary optimization algorithms, Measurement, 2018, 114, 417 – 427.
[22] Qu, N.H.A.M.Z.A.P.N.S.J.J.L.B.Y., Problem definitions and evaluation criteria for the cec 2017 special session and competition on single objective realparameter numerical optimization, Tech. rep., Nanyang Technological University, Singapore And Jordan University of Science and Technology, Jordan And Zhengzhou University, Zhengzhou China, 2016.
[23] Qu, B., Liang, J., Wang, Z., Chen, Q., Suganthan, P., Novel benchmark functions for continuous multimodal optimization with comparative results, Swarm and Evolutionary Computation, 2016, 26, 23–34.
[24] Liang, J., Qu, B., Suganthan, P., Problem definitions and evaluation criteria for the cec 2014 special session and competition on single objective real-parameter numerical optimization, Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore, 2013, 635.
[25] Hansen, N., Auger, A., Brockhoff, D., Tusar, D., Tusar, T., COCO: performance assessment, CoRR, 2016, abs/1605.03560.
[26] Piotrowski, A.P., Napiorkowski, M.J., Napiorkowski, J.J., Rowinski, P.M., Swarm intelligence and evolutionary algorithms: Performance versus speed, Information Sciences, 2017, 384, 34–85.
[27] Boskovic, B., Brest, J., Protein folding optimization using differential evolution extended with local search and component reinitialization, Information Sciences, 2018, 454-455, 178 – 199.
[28] Schott, F., Contribution to robust and fiabilist optimization application to the design of acoustic radio-frequency filters, Ph.D. thesis, Université de Bourgogne Franche Comté, UTBM, 2019.
[29] Tawfik, A.S., Badr, A.A., Abdel-Rahman, I.F., Comparative optimizer rank and score: A modern approach for performance analysis of optimization techniques, Expert Systems with Applications, 2016, 45, 118–130.
[30] Chen, Q., Liu, B., Zhang, Q., Liang, J., Suganthan, P.N., Qu, B., Problem definition and evaluation criteria for cec 2015 special session and competition on bound constrained single-objective computationally expensive numerical optimization, Tech. rep., Congress on Evolutionary Computation (CEC), 2013.
[31] Yang, X.S., Koziel, S., Leifsson, L., Computational Optimization, Modelling and Simulation: Past, Present and Future, Procedia Computer Science, 2014, 29(Supplement C), 754 – 758.
[32] Hansen, N., The cma evolution strategy: a comparing review, Towards a new evolutionary computation, Springer, 2006, 75–102.
[33] Holland, J.H., Adaptation in Natural and Artificial Systems, MIT Press, Cambridge, MA, USA, 1992.
[34] Eesa, A., Mohsin Abdulazeez Brifcani, A., Orman, Z., A novel bio-inspired optimization algorithm, International Journal of Scientific & Engineering Research, 2013, 4, 1978–1986.
[35] Neri, F., Tirronen, V., Tirronen, v.: Recent advances in differential evolution: a survey and experimental analysis. artif. intell. rev. 33(1-2), 61-106, Artif. Intell. Rev., 2010, 33, 61–106.
[36] Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P., Optimization by simulated annealing, Science, 1983, 220(4598), 671–680.
[37] Muller, C.L., Baumgartner, B., Sbalzarini, I.F., Particle swarm cma evolution strategy for the optimization of multi-funnel landscapes, 2009 IEEE Congress on Evolutionary Computation, 2685–2692.
)