HYBRID QUANTUM OPTIMIZATION OF DISTILBERT FOR EFFICIENT NATURAL LANGUAGE PROCESSING
Abstract
The authors created a hybrid quantum-classical model for efficient sentiment classification based on Frozen Distilled Bidirectional Encoder Representations from Transformers (frozen dBERT), supervised dimensionality reduction, variational quantum learning, and residual integration. The binary Stanford Sentiment Treebank had a total of 67,349 training observations. Following contradiction, duplication and overlap removal, 66,971 observations were left, of which 60,274 were used for training, and 6,697 for internal validation. There were 872 observations on the official development partition that were not touched. The embeddings, which had a dimension of 768, were mean-pooled to 4, 6, or 8 dimensions. The six-dimensional bottleneck was chosen and represented in the variational circuit with six qubits. The selected circuit was composed of 12 completed trials and 14 pruned trials, and had 3 layers, 36 quantum parameters, 52 logical gates and a circuit depth of 16. The classical six-dimensional bottleneck achieved 0.8360 accuracy, 0.8427 harmonic-mean classification score, 0.9184 receiver operating characteristic area under the curve and 0.6724 Matthews correlation coefficient on the official development partition. It was able to lower the dimensionality of the representation by 99.219 percent and obtain a model-head latency of 0.000607 milliseconds per observation. The quantum-only classifier had an accuracy of 0.6663 and a harmonic-mean classification score of 0.6914, whereas the residual hybrid kept a zero-quantum contribution. The results show efficient classical compression with no quantum advantage for the simulation of six-qubit states.
Downloads
References
2.Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2623–2631). Association for Computing Machinery. https://doi.org/10.1145/3292500.3330701
3.Benedetti, M., Lloyd, E., Sack, S., & Fiorentini, M. (2019). Parameterized quantum circuits as machine learning models. Quantum Science and Technology, 4(4), Article 043001. https://doi.org/10.1088/2058-9565/ab4eb5
4.Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., & Lloyd, S. (2017). Quantum machine learning. Nature, 549(7671), 195–202. https://doi.org/10.1038/nature23474
5.Cerezo, M., Arrasmith, A., Babbush, R., Benjamin, S. C., Endo, S., Fujii, K., McClean, J. R., Mitarai, K., Yuan, X., Cincio, Ł., & Coles, P. J. (2021). Variational quantum algorithms. Nature Reviews Physics, 3(9), 625–644. https://doi.org/10.1038/s42254-021-00348-9
6.Cerezo, M., Sone, A., Volkoff, T., Cincio, Ł., & Coles, P. J. (2021). Cost function dependent barren plateaus in shallow parametrized quantum circuits. Nature Communications, 12, Article 1791. https://doi.org/10.1038/s41467-021-21728-w
7.Coecke, B., de Felice, G., Meichanetzidis, K., & Toumi, A. (2020). Foundations for near-term quantum natural language processing [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2012.03755
8.Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
9.Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021). Compressing large-scale transformer-based models: A case study on BERT. Transactions of the Association for Computational Linguistics, 9, 1061–1080. https://doi.org/10.1162/tacl_a_00413
10.Havlíček, V., Córcoles, A. D., Temme, K., Harrow, A. W., Kandala, A., Chow, J. M., & Gambetta, J. M. (2019). Supervised learning with quantum-enhanced feature spaces. Nature, 567(7747), 209–212. https://doi.org/10.1038/s41586-019-0980-2
11.Huang, H.-Y., Broughton, M., Mohseni, M., Babbush, R., Boixo, S., Neven, H., & McClean, J. R. (2021). Power of data in quantum machine learning. Nature Communications, 12, Article 2631. https://doi.org/10.1038/s41467-021-22539-9
12.Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020). TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 4163–4174). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.372
13.Kartsaklis, D., Fan, I., Yeung, R., Pearson, A., Lorenz, R., Toumi, A., de Felice, G., Meichanetzidis, K., Clark, S., & Coecke, B. (2021). lambeq: An efficient high-level Python library for quantum NLP [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2110.04236
14.Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). ALBERT: A lite BERT for self-supervised learning of language representations [Preprint]. arXiv. https://doi.org/10.48550/arXiv.1909.11942
15.Mari, A., Bromley, T. R., Izaac, J., Schuld, M., & Killoran, N. (2020). Transfer learning in hybrid classical-quantum neural networks. Quantum, 4, Article 340. https://doi.org/10.22331/q-2020-10-09-340
16.McClean, J. R., Boixo, S., Smelyanskiy, V. N., Babbush, R., & Neven, H. (2018). Barren plateaus in quantum neural network training landscapes. Nature Communications, 9, Article 4812. https://doi.org/10.1038/s41467-018-07090-4
17.Meichanetzidis, K., Gogioso, S., de Felice, G., Chiappori, N., Toumi, A., & Coecke, B. (2021). Quantum natural language processing on near-term quantum computers. Electronic Proceedings in Theoretical Computer Science, 340, 213–229. https://doi.org/10.4204/EPTCS.340.11
18.Mitarai, K., Negoro, M., Kitagawa, M., & Fujii, K. (2018). Quantum circuit learning. Physical Review A, 98(3), Article 032309. https://doi.org/10.1103/PhysRevA.98.032309
19.Pérez-Salinas, A., Cervera-Lierta, A., Gil-Fuster, E., & Latorre, J. I. (2020). Data re-uploading for a universal quantum classifier. Quantum, 4, Article 226. https://doi.org/10.22331/q-2020-02-06-226
20.Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter [Preprint]. arXiv. https://doi.org/10.48550/arXiv.1910.01108
21.Schuld, M., Bocharov, A., Svore, K. M., & Wiebe, N. (2020). Circuit-centric quantum classifiers. Physical Review A, 101(3), Article 032308. https://doi.org/10.1103/PhysRevA.101.032308
22.Schuld, M., & Killoran, N. (2019). Quantum machine learning in feature Hilbert spaces. Physical Review Letters, 122(4), Article 040504. https://doi.org/10.1103/PhysRevLett.122.040504
23.Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (pp. 1631–1642). Association for Computational Linguistics. https://aclanthology.org/D13-1170/
24.Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D. (2020). MobileBERT: A compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 2158–2170). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.195
25.Widdows, D., Alexander, A., Zhu, D., Zimmerman, C., & Majumder, A. (2024). Near-term advances in quantum natural language processing. Annals of Mathematics and Artificial Intelligence, 92, 1249–1272. https://doi.org/10.1007/s10472-024-09940-y
Copyright (c) 2026 IJRDO -Journal of Computer Science Engineering
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Author(s) and co-author(s) jointly and severally represent and warrant that the Article is original with the author(s) and does not infringe any copyright or violate any other right of any third parties, and that the Article has not been published elsewhere. Author(s) agree to the terms that the IJRDO Journal will have the full right to remove the published article on any misconduct found in the published article.
