IMPLEMENTASI DEEP LEARNING HYBRID RESNET50V2-BIGRU PADA APLIKASI WEB VISUAL SPEECH RECOGNITION BAGI KOMUNITAS TULI
Abstract
Visual Speech Recognition (VSR) merupakan teknologi komunikasi krusial bagi komunitas Tuli, terutama di lingkungan bising di mana sistem berbasis audio gagal berfungsi. Pengembangan VSR untuk bahasa Indonesia menghadapi tantangan kelangkaan dataset audio-visual dan tingginya ambiguitas kata homofon. Selain itu, tingginya beban komputasi data video seringkali menghambat implementasi pada platform web. Oleh karena itu, penelitian ini bertujuan mengembangkan sistem VSR bahasa Indonesia yang efisien menggunakan arsitektur hibrida untuk aplikasi web responsif. Metodologi yang digunakan adalah CRISP-ML(Q). Ekstraksi fitur visual spasial dilakukan menggunakan jaringan pre-trained ResNet50V2, dikombinasikan dengan Bidirectional Gated Recurrent Unit (Bi-GRU) untuk memodelkan dependensi sekuensial temporal. Model dilatih dengan dataset publik LUMINA dan dievaluasi menggunakan 250 sampel video dataset privat. Hasil eksperimen menunjukkan akurasi validasi mencapai 95,49%. Pada pengujian data baru, sistem mencatat Word Error Rate (WER) sebesar 11,18% dan Character Error Rate (CER) 8,74%. Implementasi berbasis web menggunakan Streamlit mencapai waktu respons inferensi 2-4 detik. Kesimpulannya, model ini sukses menerjemahkan gerakan bibir menjadi teks secara akurat dan menyediakan alat komunikasi responsif bagi penyandang disabilitas rungu.
Downloads
References
A. Muhawarman, “Kemenkes Ajak Masyarakat Peduli Kesehatan Pendengaran,” Jakarta, Mar. 03, 2025. Accessed: Feb. 10, 2026. [Online]. Available: https://kemkes.go.id/eng/kemenkes-ajak-masyarakat-peduli-kesehatan-pendengaran
H. L. Handojo and B. A. Sukada, “Implementasi Design SaranaTerapi Bermain untuk Pengembangan Kemampuan Wicara dan Bahasa Anak Penyamdang Tunarungu,” Jurnal Sains, Teknologi, Urban, Perancangan, Arsitektur (Stupa), vol. 6, no. 1, pp. 603–612, Apr. 2024, doi: 10.24912/stupa.v6i1.27502.
M. Kim, H.-I. Kim, and Y. M. Ro, “Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition,” Oct. 2024, [Online]. Available: http://arxiv.org/abs/2302.08102
P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels,” Jun. 2023, doi: 10.1109/ICASSP49357.2023.10096889.
D. Ivanko, D. Ryumin, and A. Karpov, “A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition,” Jun. 01, 2023, MDPI. doi: 10.3390/math11122665.
R. Inamdar, K. sundarr, D. Khandelwal, and A. KB, “Lips Reading Using 3D Convolution and LSTM,” Dec. 13, 2023. doi: 10.20944/preprints202312.0928.v1.
Pradiptatmaka Feliciano Akwila, Suciati Nanik, and Navastara Adni Dini, “Analisis Pembacaan Gerakan Bibir Menggunakan Gabungan Arsitektur Convolutional Neural Network dan Recurrent Neural Network,” 2021.
G. M. Rahmatullah, S.-J. Ruan, and L. P.-H. Li, “Recognizing Indonesian words based on visual cues of lip movement using deep learning,” Measurement, vol. 250, p. 116968, Jun. 2025, doi: https://doi.org/10.1016/j.measurement.2025.116968.
E. R. Setyaningsih, A. N. Handayani, W. S. G. Irianto, Y. Kristian, and C. T. S. L. Chen, “LUMINA: Linguistic unified multimodal Indonesian natural audio-visual dataset,” Data Brief, vol. 54, p. 110279, Jun. 2024, doi: 10.1016/j.dib.2024.110279.
H. Laux, E. Mededovic, A. Hallawa, L. Martin, A. Peine, and A. Schmeink, “LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data,” Dec. 2023, [Online]. Available: http://arxiv.org/abs/2312.09727
N. Duklan, S. Kumar, H. Maheshwari, R. Singh, S. D. Sharma, and S. Swami, “CNN Architectures for Image Classification: A Comparative Study Using ResNet50V2, ResNet152V2, InceptionV3, Xception, and MobileNetV2,” SSRG International Journal of Electronics and Communication Engineering, vol. 11, no. 9, pp. 11–21, Sep. 2024, doi: 10.14445/23488549/IJECE-V11I9P102.
D. Xiao, F. Zhu, J. Jiang, and X. Niu, “Leveraging natural cognitive systems in conjunction with ResNet50-BiGRU model and attention mechanism for enhanced medical image analysis and sports injury prediction,” Front. Neurosci., vol. 17, 2023, doi: 10.3389/fnins.2023.1273931.
K. T. Bibish Kumar, S. John, K. M. Muraleedharan, and R. K. Sunil Kumar, “Linguistically involved data-driven approach for Malayalam phoneme-to-viseme mapping,” in Applied Speech Processing, Elsevier, 2021, pp. 117–145. doi: 10.1016/B978-0-12-823898-1.00003-5.
S. Agarwal, H. Farid, O. Fried, and M. Agrawala, “Detecting Deep-Fake Videos from Phoneme-Viseme Mismatches,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Jun. 2020, pp. 2814–2822. doi: 10.1109/CVPRW50498.2020.00338.
M. Ardiana, “Viseme Morphing and Text-to-Speech Integration for Indonesian News Broadcasting,” The Indonesian Journal of Computer Science, vol. 14, no. 3, Jun. 2025, doi: 10.33022/ijcs.v14i3.4841.
Sougatamoy Biswas, Asim Kumar Naskar, and Anup Nandy, “MediaPipe with LSTM Architecture for Real-Time Hand Gesture Recognization ⋆,” 2023.
J. Stephen and J. MK, “A Survey on Multilingual End-to-End Automatic Continuous Speech Recognition Using Deep Neural Network,” 2023. [Online]. Available: www.ijrti.org
M. Maskur and A. Zahra, “The Impact of Data Augmentation Techniques on Improving Speech Recognition Performance for English in Indonesian Children Based on Wav2Vec 2.0,” IJCCS (Indonesian Journal of Computing and Cybernetics Systems, vol. 19, no. 2, pp. 165–176, 2025, doi: 10.22146/ijccs.104646.
S. Studer et al., “Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology,” Mach. Learn. Knowl. Extr., vol. 3, no. 2, pp. 392–413, Jun. 2021, doi: 10.3390/make3020020.
A. Wangikar, A. Khavale, S. Sharma, U. Gadekar, and U. Sinha, “Ai-Powered Lip Reading Assistant For Individuals With Hearing Impairment,” 2025, [Online]. Available: www.ijcrt.org
N. Sarangpure, V. Dhamde, A. Roge, J. Doye, S. Patle, and S. Tamboli, “Automating the Machine Learning Process using PyCaret and Streamlit,” in 2023 2nd International Conference for Innovation in Technology (INOCON), IEEE, Mar. 2023, pp. 1–5. doi: 10.1109/INOCON57975.2023.10101357.
L. Poomhiran, P. Meesad, and S. Nuanmeesri, “Improving the Recognition Performance of Lip Reading Using the Concatenated Three Sequence Keyframe Image Technique,” 2021. [Online]. Available: www.etasr.com
I. I. Panagos, G. Sfikas, and C. Nikou, “Lightweight Operations for Visual Speech Recognition,” Feb. 2025, [Online]. Available: http://arxiv.org/abs/2502.04834
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
