Research on hybrid transformer-based autoencoders for user biometric verification
DOI:
https://doi.org/10.20535/SRIT.2308-8893.2023.3.03Keywords:
biometric verification, transformers, variational autoencoder, transformer autoencoderAbstract
Our current study extends previous work on motion-based biometric verification using sensory data by exploring new architectures and more complex input from various sensors. Biometric verification offers advantages like uniqueness and protection against fraud. The state-of-the-art transformer architecture in AI is known for its attention block and applications in various fields, including NLP and CV. We investigated its potential value for applications involving sensory data. The research proposes a hybrid architecture, integrating transformer attention blocks with different autoencoders, to evaluate its efficacy for biometric verification and user authentication. Various configurations were compared, including LSTM autoencoder, transformer autoencoder, LSTM VAE, and transformer VAE. Results showed that combining transformer blocks with an undercomplete deterministic autoencoder yields the best performance, but model performance is significantly influenced by data preprocessing and configuration parameters. The application of transformers for biometric verification and sensory data appears promising, performing on par with or surpassing LSTM-based models but with lower inference and training time.
References
S. Kokal, M. Vanamala, and R. Dave, “Deep Learning and Machine Learning, Better Together Than Apart: A Review on Biometrics Mobile Authentication,” Journal of Cybersecurity and Privacy, vol. 3, no. 2, pp. 227–258, Jun. 2023. doi: 10.3390/jcp3020013.
Q. Cao, F. Xu, and H. Li, “User Authentication by Gait Data from Smartphone Sensors Using Hybrid Deep Learning Network,” Mathematics, vol. 10, no. 13, p. 2283, Jun. 2022. doi: 10.3390/math10132283.
A. Vaswani et al., “Attention is All you Need,” in Advances in Neural Information Processing Systems (NIPS), vol. 30, 2017.
S.M. Lakew, M. Cettolo, and M. Federico, “A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation,” in International Conference on Computational Linguistics (COLING), 2018.
A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are Transformers Effective for Time Series Forecasting?,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Q. Wen et al., “Transformers in time series: A survey,” in International Joint Conference on Artificial Intelligence (IJCAI), 2023.
A. Katrompas, T. Ntakouris, and V. Metsis, “Recurrence and Self-attention vs the Transformer for Time-Series Classification: A Comparative Study,” in Michalowski M., Abidi S.S.R., Abidi S. (eds) Artificial Intelligence in Medicine (AIME), vol. 13263. Springer, Cham, 2022. doi: 10.1007/978-3-031-09342-5_10.
I. Dirgová Luptáková, M. Kubovčík, and J. Pospíchal, “Wearable Sensor-Based Human Activity Recognition with Transformer Model,” Sensors, vol. 22, no. 5, p. 1911, Mar. 2022. doi: 10.3390/s22051911.
A. Raza et al., “Lightweight Transformer in Federated Setting for Human Activity Recognition,” ArXiv, arXiv:2110.00244, Nov. 4, 2022. Accessed on: July 6, 2023. [Online]. Available: https://arxiv.org/abs/2110.00244
M. Havrylovych and V. Danylov, “Research of Autoencoder-Based User Biometric Verification with Motion Patterns,” System Research and Information Technologies, no. 2, 2022, pp. 128–136. doi: 10.20535/srit.2308-8893.2022.2.10.
M. Havrylovych, V. Danylov, and A. Gozhyj, “Comparative Analysis of using Recurrent Autoencoders for User Biometric Verification with Wearable Accelerometer,” Proceedings of the 9th International Conference “Information Control Systems & Technologies” (ICST), pp. 358–370, Sept. 2020.
F. Trad, A. Hussein and A. Chehab, “Free Text Keystroke Dynamics-based Authentication with Continuous Learning: A Case Study,” in 2022 IEEE 21st International Conference on Ubiquitous Computing and Communications (IUCC/CIT/DSCI/ SmartCNS), Chongqing, China, 2022, pp. 125–131. doi: 10.1109/IUCC-CIT-DSCI-SmartCNS57392.2022.00031S.
M. Roy, S. Majumder, A. Halder, and U. Biswas, “ECG-NET: A deep LSTM autoencoder for detecting anomalous ECG,” Engineering Applications of Artificial Intelligence, vol. 124, p. 106484, 2023. doi: 10.1016/j.engappai.2023.106484.
L. Shan et al., “Abnormal ECG detection based on an adversarial autoencoder,” Frontiers in Physiology, vol. 13, 2022. doi: 10.3389/fphys.2022.961724.
A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey, “Adversarial Autoencoders,” arXiv, arXiv:1511.05644, 2015. [Online]. Available: https://arxiv.org/abs/1511.05644.
A. Oluwasanmi, M.U. Aftab, E. Baagyere, Z. Qin, M. Ahmad, and M. Mazzara, “Attention Autoencoder for Generative Latent Representational Learning in Anomaly Detection,” Sensors, vol. 22, no. 1, p. 123, Dec. 2021. doi: 10.3390/s22010123.
M. Hu, K. Zhang, R. You, and B. Tu, “Relative Attention-based One-Class Adversarial Autoencoder for Continuous Authentication of Smartphone Users,” arXiv, arXiv:2210.16819, 2022. [Online]. Available: https://arxiv.org/abs/2210.16819.
A. Alamr and A. Artoli, “Unsupervised Transformer-Based Anomaly Detection in ECG Signals,” Algorithms, vol. 16, no. 3, p. 152, Mar. 2023. doi: 10.3390/a16030152.
M. Ma, C. Sun and X. Chen, “Deep Coupling Autoencoder for Fault Diagnosis With Multimodal Sensory Data,” in IEEE Transactions on Industrial Informatics, vol. 14, no. 3, pp. 1137–1145, March 2018. doi: 10.1109/TII.2018.2793246.
S. Qiu et al., “Deep Learning Techniques in Intelligent Fault Diagnosis and Prognosis for Industrial Systems: A Review,” Sensors, vol. 23, no. 3, p. 1305, Jan. 2023. doi: 10.3390/s23031305.
L. Weng, “From Autoencoder to Beta-VAE,” Lil’Log, Aug. 12, 2018. Accessed on: July 15, 2023. [Online]. Available: https://lilianweng.github.io/lil-log/2018/08/12/from-autoencoder-to-beta-vae.html
Qing Yang et al., “A Multimodal Data Set for Evaluating Continuous Authentication Performance in Smartphones,” in Proceedings of the 12th ACM Conference on Embedded Network Sensor Systems (SenSys ‘14). ACM, New York, NY, USA, pp. 358–359. doi: 10.1145/2668332.2668366.
Zdeňka Sitová et al., “HMOG: New Behavioral Biometric Features for Continuous Authentication of Smartphone Users,” in IEEE Transactions on Information Forensics and Security, vol.11, no.5, pp. 877–892, May 2016. doi: 10.1109/TIFS.2015.2506542.
“A Multimodal Data Set for Evaluating Continuous Authentication Performance,” H-MOG Dataset. Accessed on: July 15, 2023. [Online]. Available: https://hmog-dataset.github.io/hmog/
A. Allan, Basic Sensors in iOS: Programming the Accelerometer, Gyroscope, and More; 1st ed., O’Reilly Media, Beijing, Aug. 30, 2011. Accessed on: July 8, 2023. [Online]. Available: https://www.amazon.com/Basic-Sensors-iOS-Programming-Accelerometer/dp/1449308465
S. Balli, E.A. Sağbaş, and M. Peker, “A Mobile Solution Based on Soft Computing for Fall Detection,” in Mobile Solutions and Their Usefulness in Everyday Life, S. Paiva, Ed. Cham: Springer International Publishing, 2019, pp. 275–294. doi: 10.1007/978-3-319-93491-4_14.
M. El-Abed, C. Charrier, and C. Rosenberger, “Evaluation of Biometric Systems,” New Trends and Developments in Biometrics. InTech, Nov. 28, 2012. doi: 10.5772/52084.
