Evaluation of Synthetic Data Effectiveness using Generative Adversarial Networks (GAN) in Improving Javanese Script Recognition on Ancient Manuscript
DOI:
https://doi.org/10.12962/j24068535.v23i1.a1256Abstract
The imbalance of Javanese script data in ancient manuscript recognition poses a challenge due to the limited availability of data. A potential approach to addressing this issue is the use of Generative Adversarial Networks (GAN). This study evaluates the effectiveness of synthetic data generated using Enhanced Balancing GAN (EBGAN) in mitigating data imbalance. Various evaluation scenarios are conducted, including: (i) assessing the impact of syn-thetic data as augmentation, (ii) evaluating the sufficiency of synthetic data for recognition models, (iii) analyzing minority class oversampling with different selection strategies, and (iv) evaluating model generalization through cross-validation. Quantitative analysis of the generated synthetic data, based on Fréchet Inception Distance (FID) and Structural Similarity Index (SSIM), as well as visual assessment, indicates that the quality of synthetic data closely resembles real data. Additionally, experimental results demonstrate that combining real and synthetic data improves accuracy, precision, recall, and F1-score. The oversampling strategy for synthetic data has proven effective in meeting the data sufficiency requirements for training recognition models. Meanwhile, selecting minority classes and determining threshold values based on percentage, distribution, and model performance in oversampling can serve as guidelines for enhancing script recognition performance. Compared to previous methods, the use of EBGAN has been shown to produce more diverse synthetic data with better visual quality. However, further research is still needed to optimize GAN performance in supporting script recognition.References
D. Iskandar, S. Hidayat, U. Jamaludin, and S. Mukti Leksono, “Javanese script digitalization and its utilization as learning media: an etnopedagog-ical approach,” International Journal of Mathematics and Sciences Education, vol. 1, no. 1, pp. 21–30, Jun. 2023, doi: 10.59965/ijmsed.v1i1.24.
S. O. Robson, “Javanese script as cultural artifact: Historical background,” RIMA: Review of Indonesian and Malaysian Affairs, vol. 45, no. 1 and 2, pp. 9 – 36, 2011, doi: 10.3316/ielapa.422100940117015.
Y. Sugianela and N. Suciati, “Ekstraksi Fitur pada Pengenalan Karakter Aksara Jawa Berbasis Histogram of Oriented Gradient,” JUTI: Jurnal Ilmiah Teknologi Informasi, vol. 17, no. 1, pp. 64–72, 2019.
E. Paulus, M. Suryani, S. Hadi, and F. Natsir, “An initial study to solve imbalance sundanese handwritten dataset in character recognition,” Pro-ceedings of the 3rd International Conference on Informatics and Computing, ICIC 2018, Oct. 2018, doi: 10.1109/IAC.2018.8780496.
A. R. Widiarti, R. Pulungan, A. Harjoko, Marsono, and S. Hartati, “A Proposed Model for Javanese Manuscript Images Transliteration,” in Journal of Physics: Conference Series, Institute of Physics Publishing, Oct. 2018. doi: 10.1088/1742-6596/1098/1/012014.
D. Tursina, S. R. Anggraeni, C. Fatichah, M. Munir, and I. Subakti, “Metode Hibrida Oversampling untuk Menangani Imbalanced Multi-Label,” JUTI: Jurnal Ilmiah Teknologi Informasi, vol. 22, no. 1, p. 32, 2024, doi: 10.12962/j24068535.v22i1.a1208.
C. Shorten and T. M. Khoshgoftaar, “A survey on Image Data Augmentation for Deep Learning,” J Big Data, vol. 6, no. 1, p. 60, 2019, doi: 10.1186/s40537-019-0197-0.
T. Dam, M. M. Ferdaus, S. G. Anavatti, S. Jayavelu, and H. A. Abbass, “Does Adversarial Oversampling Help us?,” Aug. 2021.
G. Huang and A. H. Jafari, “Enhanced balancing GAN: minority-class image generation,” Neural Comput Appl, vol. 35, no. 7, pp. 5145–5154, Mar. 2023, doi: 10.1007/s00521-021-06163-8.
N. V Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of Artificial Intel-ligence Research, vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953.
A. Fernandez, S. Garcia, F. Herrera, and N. V. Chawla, “SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year Anniversary,” Journal of Artificial Intelligence Research, vol. 61, pp. 863–905, Apr. 2018, doi: 10.1613/jair.1.11192.
H. Han, W.-Y. Wang, and B.-H. Mao, “Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning,” 2005, pp. 878–887. doi: 10.1007/11538059_91.
Y. Sun et al., “Borderline SMOTE Algorithm and Feature Selection-Based Network Anomalies Detection Strategy,” Energies (Basel), vol. 15, no. 13, p. 4751, Jun. 2022, doi: 10.3390/en15134751.
Haibo He, Yang Bai, E. A. Garcia, and Shutao Li, “ADASYN: Adaptive synthetic sampling approach for imbalanced learning,” in 2008 IEEE In-ternational Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), IEEE, Jun. 2008, pp. 1322–1328. doi: 10.1109/IJCNN.2008.4633969.
Z. Hu, L. Wang, L. Qi, Y. Li, and W. Yang, “A Novel Wireless Network Intrusion Detection Method Based on Adaptive Synthetic Sampling and an Improved Convolutional Neural Network,” IEEE Access, vol. 8, pp. 195741–195751, 2020, doi: 10.1109/ACCESS.2020.3034015.
W. Zhang and G. Ding, “A Study on the GAN-Stacking Model Framework for Fraud Dataset,” in 2023 4th International Conference on Machine Learning and Computer Application, New York, NY, USA: ACM, Oct. 2023, pp. 310–315. doi: 10.1145/3650215.3650270.
I. J. Goodfellow et al., “Generative Adversarial Networks,” Jun. 2014.
T. Chakraborty, U. Reddy K S, S. M. Naik, M. Panja, and B. Manvitha, “Ten years of generative adversarial nets (GANs): a survey of the state-of-the-art,” Mach Learn Sci Technol, vol. 5, no. 1, p. 011001, Mar. 2024, doi: 10.1088/2632-2153/ad1f77.
M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” arXiv preprint, Nov. 2014, doi: 10.48550/arXiv.1610.09585.
G. Zhao, P. Liu, K. Sun, Y. Yang, T. Lan, and H. Yang, “Research on data imbalance in intrusion detection using CGAN,” PLoS One, vol. 18, no. 10, p. e0291750, Oct. 2023, doi: 10.1371/journal.pone.0291750.
A. Odena, C. Olah, and J. Shlens, “Conditional Image Synthesis with Auxiliary Classifier GANs,” in Proc. ICML ’17: 34th International Conference on Machine Learning, May 2017, pp. 2642–2651. doi: 10.5555/3305890.3305954.
Y. Luo and B.-L. Lu, “EEG Data Augmentation for Emotion Recognition Using a Conditional Wasserstein GAN,” in 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, Jul. 2018, pp. 2535–2538. doi: 10.1109/EMBC.2018.8512865.
G. Mariani, F. Scheidegger, R. Istrate, C. Bekas, and C. Malossi, “BAGAN: Data Augmentation with Balancing GAN,” arXiv preprint, Mar. 2018.
G. Cao and S.-I. Kamata, “Data Augmentation for Historical Documents via Cascade Variational Auto-Encoder,” in 2019 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), IEEE, Sep. 2019, pp. 340–345. doi: 10.1109/ICSIPA45851.2019.8977737.
A. Ali-Gombe and E. Elyan, “MFC-GAN: Class-imbalanced dataset classification using Multiple Fake Class Generative Adversarial Network,” Neurocomputing, vol. 361, pp. 212–221, Oct. 2019, doi: 10.1016/j.neucom.2019.06.043.
M. Eltay, A. Zidouri, I. Ahmad, and Y. Elarian, “Generative adversarial network based adaptive data augmentation for handwritten Arabic text recognition,” PeerJ Comput Sci, vol. 8, p. e861, Jan. 2022, doi: 10.7717/peerj-cs.861.
J. Cai, L. Peng, Y. Tang, C. Liu, and P. Li, “TH-GAN: Generative Adversarial Network Based Transfer Learning for Historical Chinese Character Recognition,” in 2019 International Conference on Document Analysis and Recognition (ICDAR), IEEE, Sep. 2019, pp. 178–183. doi: 10.1109/ICDAR.2019.00037.
Z. Yuan and S. Kamata, “Data Augmentation for Ancient Characters via Semi-MixFontGan,” in 2020 Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 2020 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR), IEEE, Aug. 2020, pp. 1–6. doi: 10.1109/ICIEVicIVPR48672.2020.9306588.
M. A. Faizin, “Deteksi Aksara Jawa Menggunakan YOLO untuk Transliterasi Berbasis LSTM pada Manuskrip Jawa Kuno,” Institut Teknologi Sepuluh Nopember, 2023. Accessed: May 19, 2024. [Online]. Available: https://repository.its.ac.id/102757/
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved Training of Wasserstein GANs,” arXiv preprint, Mar. 2017.
Z. Qu, G. Fan, Z. Zhao, L. Jia, J. Shi, and J. Ai, “Synthetic aperture radar ground target image generation based on improved Wasserstein generative adversarial networks with gradient penalty,” J Appl Remote Sens, vol. 17, no. 03, Jul. 2023, doi: 10.1117/1.JRS.17.036501.
L. O. Guertler, A. Ashfahani, and A. T. Luu, “How to train your draGAN: A task oriented solution to imbalanced classification,” arXiv preprint, Nov. 2022, doi: 10.48550/arXiv.2211.10065.
G. James, D. Witten, T. Hastie, and R. Tibshirani, “Resampling Methods,” 2021, pp. 197–223. doi: 10.1007/978-1-0716-1418-1_5.
Y. N. Dauphin and Y. Bengio, “Big Neural Networks Waste Capacity,” Jan. 2013.
Downloads
Published
Issue
Section
License
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in JUTI unless they receive approval for doing so from the Editor-in-Chief.
JUTI open access articles are distributed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
