SISTEM REKOMENDASI METADATA LAGU BERDASARKAN DETEKSI EMOSI WAJAH MENGGUNAKAN VIT-B/16

Ainan Zaky Nurrofiq; Bambang Irawan

doi:10.23960/jitet.v14i1.8950

Authors

Ainan Zaky Nurrofiq Mahasiswa
Bambang Irawan Universitas Muhadi Setiabudi

DOI:

https://doi.org/10.23960/jitet.v14i1.8950

Abstract Views: 65 File Views: 21

Keywords:

Deteksi emosi wajah, ViT-B/16, Rekomendasi lagu, Metadata lagu, Vision transformer

Abstract

This study presents a music metadata recommendation system based on facial emotion detection using the Vision Transformer (ViT-B/16) model. The system classifies user emotions into seven categories using the KDEF facial dataset and matches them with music metadata (title, artist, genre, mood) labeled with corresponding emotional tags. The ViT-B/16 model was trained using transfer learning and evaluated with accuracy, precision, recall, and F1-score. The model achieved an accuracy of 89% and an average F1-score of 0.89. The recommendation system was assessed by 30 participants, with 87% indicating that the suggested song metadata matched the detected emotion. The system offers real-time emotion recognition and automatic mood-based song suggestions. However, classification accuracy for visually similar emotions such as “fear” and “angry” remains a challenge. Future development may include audio and lyric analysis, as well as user preference integration, to enhance recommendation relevance.

Downloads

Download data is not yet available.

References

H. Nurdiyansah and E. Rochmawati, “Klasifikasi emosi wajah menggunakan CNN pada sistem interaksi cerdas,” J. Sains Komput. dan Inform., vol. 11, no. 2, pp. 104–113, 2025.

R. M. Putra and S. Sumarno, “Klasifikasi Ekspresi Wajah Menggunakan Convolutional Neural Network (CNN),” J. Ilm. Tek. Elektro Terap., vol. 9, no. 2, pp. 123–130, 2023, [Online]. Available: https://jurnal.polindra.ac.id/index.php/JITET/article/view/1131

R. A. Hakim, A. Nurhadi, and G. Pratama, “Real-time facial emotion recognition using YOLOv8 in smart systems,” J. Intell. Syst. Appl., vol. 8, no. 2, pp. 88–97, 2024.

A. Saepudin, T. Widodo, and S. Fitriani, “Eksplorasi sinyal EEG untuk deteksi emosi dalam sistem afektif,” Indones. J. Biomed. Eng., vol. 5, no. 2, pp. 112–120, 2024.

L. Michelle, A. M. Putri, and M. Yusuf, “Penerapan Vision Transformer pada dataset KDEF untuk pengenalan emosi,” J. Ilmu Komput. dan Inform., vol. 14, no. 1, pp. 30–39, 2025.

G. Sruthi, “Emotion-aware music recommendation systems: A survey,” Int. J. Comput. Appl., vol. 182, no. 10, pp. 18–24, 2024.

M. Kambham, R. Reddy, and A. Sharma, “CNN-based emotional genre classification for music recommendation,” J. Artif. Intell. Soft Comput. Res., vol. 15, no. 1, pp. 75–86, 2025.

R. Agustini, “Real-time facial expression detection and music recommendation using deep learning,” J. Teknol. Inf. dan Komput., vol. 10, no. 1, pp. 44–52, 2023.

V. Bagadi, “Emotion-based content filtering for personalized music recommendations,” Int. J. Affect. Comput., vol. 9, no. 2, pp. 133–141, 2025.

M. Amin and D. Setiawan, “Klasifikasi Emosi Menggunakan Kombinasi Data Audio dan Visual dengan Deep Learning,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 7, no. 3, pp. 456–462, 2023, [Online]. Available: https://ejournal.unsri.ac.id/index.php/resti/article/view/14567

Z. Mamieva, E. Gubanova, and D. Vasiliev, “Systematic review of emotion classification in song lyrics using NLP,” Procedia Comput. Sci., vol. 219, pp. 920–928, 2023, [Online]. Available: https://doi.org/10.1016/j.procs.2023.01.215

N. Sidora and M. Harani, “Pemanfaatan metadata musik untuk sistem rekomendasi berbasis emosi,” J. Teknol. dan Data, vol. 8, no. 1, pp. 25–36, 2024.

S. Gupta, “Affective computing for task management: Integrating facial expression and background music for stress mitigation,” IEEE Trans. Human-Machine Syst., vol. 54, no. 3, pp. 213–225, 2024.

N. Khadijah, “Emotion-aware music recommendation for postpartum mothers: A psychological approach,” Psikol. dan Kesehat. Ment., vol. 12, no. 2, pp. 99–109, 2023.

D. R. Nugraha, T. M. Sari, and Y. Hartono, “Neuropsikologi musik dan deteksi emosi: Kajian literatur integratif,” J. Psikol. Musik, vol. 2, no. 1, pp. 60–72, 2024.

A. P. Kusuma, “Cultural emotion analysis in TikTok music trends: The role of facial expression in popularity metrics,” J. Digit. Soc. Stud., vol. 6, no. 1, pp. 55–67, 2025.