Optimization Strategy for Generative Adversarial Networks Design
DOI:
https://doi.org/10.47839/ijc.22.3.3223Keywords:
artificial intelligence, machine learning, deep learning, generative adversarial network, design, optimization, loss functionAbstract
Generative Adversarial Networks (GANs) are a powerful class of deep learning models that can generate realistic synthetic data. However, designing and optimizing GANs can be a difficult task due to various technical challenges. The article provides a comprehensive analysis of solution methods for GAN performance optimization. The research covers a range of GAN design components, including loss functions, activation functions, batch normalization, weight clipping, gradient penalty, stability problems, performance evaluation, mini-batch discrimination, and other aspects. The article reviews various techniques used to address these challenges and highlights the advancements in the field. The article offers an up-to-date overview of the state-of-the-art methods for structuring, designing, and optimizing GANs, which will be valuable for researchers and practitioners. The implementation of the optimization strategy for the design of standard and deep convolutional GANs (handwritten digits and fingerprints) developed by the authors is discussed in detail, the obtained results confirm the effectiveness of the proposed optimization approach.
References
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, J. Bengio, “Generative adversarial networks,” Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2014, pp. 2672–2680.
N. Aldausari, A. Sowmya, N. Marcus, and G. Mohammadi, Video Generative Adversarial Networks: A Review, 2022. https://doi.org/10.1145/3487891.
O. S. Striuk, Y. P. Kondratenko, “Generative adversarial neural networks and deep learning: Successful cases and advanced approaches,” International Journal of Computing, vol. 20, issue 3, pp. 339-349, 2021. https://doi.org/10.47839/ijc.20.3.2278.
F. Di Mattia et al., A Survey on GANs for Anomaly Detection, 2021, [Online]. Available at: https://arxiv.org/abs/1906.11632
O. S. Striuk, Y. P. Kondratenko, “Generative adversarial networks in cybersecurity: Analysis and response,” in: Y. Kondratenko, V. Kreinovich, W. Pedrycz, A. Chilrii, A. M. Gil-Lafuente (Eds.), Artificial Intelligence in Control and Decision-making Systems: Dedicated to Prof. Janusz Kacprzyk. Studies in Computational Intelligence, vol. 1087, Springer, Cham, 2023, pp. 373-388. https://doi.org/10.1007/978-3-031-25759-9_18.
O. Striuk and Y. Kondratenko, “Adaptive deep convolutional GAN for fingerprint sample synthesis,” Proceedings of the 2021 IEEE 4th International Conference on Advanced Information and Communication Technologies (AICT), Lviv, Ukraine, September 21-25, 2021, pp. 193-196. https://doi.org/10.1109/AICT52120.2021.9628978.
O. Striuk, Y. Kondratenko, I. Sidenko, A. Vorobyova, “Generative adversarial neural network for creating photorealistic images,” Proceedings of 2020 IEEE 2nd International Conference on Advanced Trends in Information Theory, Kyiv, Ukraine, November 27, 2020, pp. 368-371. https://doi.org/10.1109/ATIT50783.2020.9349326.
M. Arjovsky, L. Bottou, Towards Principled Methods for Training Generative Adversarial Networks, 2017, [Online]. Available at: https://arxiv.org/abs/1701.04862.
R. Ayari, Generative Adversarial Networks, 2020, [Online]. Available at: https://bit.ly/3Uk4GBw.
A. Borji, Pros and Cons of GAN Evaluation Measures, 2018, [Online]. Available at: https://arxiv.org/abs/1802.03446.
M. Arjovsky, S. Chintala, L. Bottou, Wasserstein GAN, 2017, [Online]. Available at: https://arxiv.org/abs/1701.07875.
X. Mao et al., Least Squares Generative Adversarial Networks, 2016, [Online]. Available at: https://arxiv.org/abs/1611.04076.
J. H. Lim, J. C. Ye, Geometric GAN, 2017, [Online]. Available at: https://arxiv.org/abs/1705.02894.
C. Cortes, V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, issue 3, pp. 273-297, 1995. https://doi.org/10.1007/BF00994018.
C.-L. Li et al., MMD GAN: Towards Deeper Understanding of Moment Matching Network, 2017, [Online]. Available at: https://arxiv.org/abs/1705.08584.
I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, Cambridge, Massachusetts, 2016, 66 p., 178 p., 187 p., 189p.
M. P. Deisenroth, A. A. Faisal, C. S. Ong, Mathematics for Machine Learning, 1st ed., Cambridge University Press, Cambridge, 2020, 160 p., 213 p., 315 p. https://doi.org/10.1017/9781108679930.
S. Ioffe, C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, 2015, [Online]. Available at: https://arxiv.org/abs/1502.03167.
T. Salimans et al., Improved Techniques for Training GANs, 2016, [Online]. Available at: https://bit.ly/3L8qjBM.
A. Radford, L. Metz, S. Chintala, Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, 2016, [Online]. Available at: https://arxiv.org/abs/1511.06434,
S. Xiang, H. Li, On the Effects of Batch and Weight Normalization in Generative Adversarial Networks, 2017, [Online]. Available at: https://arxiv.org/abs/1704.03971.
M. Arjovsky, S. Chintala, L. Bottou, Wasserstein GAN, 2017, [Online]. Available at: https://arxiv.org/abs/1701.07875.
I. Gulrajani et al., Improved Training of Wasserstein GANs, 2017, [Online]. Available at: https://arxiv.org/abs/1704.00028.
K. Roth et al., Stabilizing Training of Generative Adversarial Networks through Regularization, 2017, [Online]. Available at: https://arxiv.org/abs/1705.09367.
J. Hui, GAN – Ways to improve GAN performance, 2018, [Online]. Available at: https://bit.ly/3A8d11Z.
T. Miyato et al., Spectral Normalization for Generative Adversarial Networks, 2018, [Online]. Available at: https://arxiv.org/abs/1802.05957.
B. O. Ayinde, K. Nishihama, J. M. Zurada, Diversity Regularized Adversarial Learning, 2019, [Online]. Available at: https://arxiv.org/abs/1901.10824. https://doi.org/10.1007/978-3-030-19823-7_24.
V. Dumoulin, F. Visin, A Guide to Convolution Arithmetic for Deep Learning, 2018, [Online]. Available at: https://arxiv.org/abs/1603.07285.
J. Brownlee, How to Use the UpSampling2D and Conv2DTranspose Layers in Keras, 2019, [Online]. Available at: https://bit.ly/3oq94TA.
J. Brownlee, A Gentle Introduction to Transfer Learning for Deep Learning, 2017, [Online]. Available at: https://bit.ly/3GTmdeC.
M. Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, 2018, [Online]. Available at: https://arxiv.org/abs/1706.08500.
S. Barratt, R. Sharma, A Note on the Inception Score, 2018, [Online]. Available at: https://arxiv.org/abs/1801.01973.
M. S. M. Sajjadi, Assessing Generative Models via Precision and Recall, 2018, [Online]. Available at: https://arxiv.org/abs/1806.00035.
T. Karras, S. Laine and T. Aila, “A style-based generator architecture for generative adversarial networks,” Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June 15-20, 2019, pp. 4396-4405. https://doi.org/10.1109/CVPR.2019.00453.
T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen and T. Aila, “Analyzing and improving the image quality of StyleGAN,” Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, June 13-19, 2020, pp. 8107-8116. https://doi.org/10.1109/CVPR42600.2020.00813.
J. Brownlee, A Gentle Introduction to BigGAN the Big Generative Adversarial Network, 2019, [Online]. Available at: https://bit.ly/3om09CM.
D. P. Kingma, J. Ba, Adam: A Method for Stochastic Optimization, 2014, [Online]. Available at: https://arxiv.org/abs/1412.6980.
Y. Kondratenko, I. Atamanyuk, I. Sidenko, G. Kondratenko, S. Sichevskyi, “Machine learning techniques for increasing efficiency of the robot’s sensor and control information processing,” Sensors, vol. 22, issue 3, 1062, 2022. https://doi.org/10.3390/s22031062.
M. Derkach, I. Skarga-Bandurova, D. Matiuk and N. Zagorodna, “Autonomous quadrotor flight stabilisation based on a complementary filter and a PID controller,” Proceedings of the 2022 12th International Conference on Dependable Systems, Services and Technologies (DESSERT), Athens, Greece, December 09-11, 2022, pp. 1-7. https://doi.org/10.1109/DESSERT58054.2022.10018623.
A. Shevchenko, M. Vakulenko, and M. Klymenko, “The Ukrainian AI strategy: Premises and outlooks,” Proceedings of the 12th International Conference on Advanced Computer Information Technologies (ACIT), Ruzomberok, Slovakia, September 26-28, 2022, pp. 511-515. https://doi.org/10.1109/ACIT54803.2022.9913094.
V. N. Opanasenko, S. L. Kryvyi, “Synthesis of neural-like networks on the basis of conversion of cyclic Hamming codes,” Cybernetics and Systems Analysis, vol. 53, issue 4, pp. 627–635, 2017. https://doi.org/10.1007/s10559-017-9965-z.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014.
Downloads
Published
How to Cite
Issue
Section
License
International Journal of Computing is an open access journal. Authors who publish with this journal agree to the following terms:• Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
• Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
• Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
