Top

Published in:

2024 | OriginalPaper | Chapter

Transformer Models in Natural Language Processing

Authors : László Kovács, László Csépányi-Fürjes, Walelign Tewabe

Published in: The 17th International Conference Interdisciplinarity in Engineering

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The development of transformer-based language models brings a paradigm shift in the world of smart applications. The ChatGPT model opened new horizons in the field of natural language understanding and generation. This paper presents a survey on the history of transformer models, on the basic architecture and application areas. The last section is devoted to two use cases experiments on the application of ChatGPT. The first domain relates to Human-Level Programming and the second focuses on the semantic functional parsing of text sentences. The performed analysis demonstrates the big potential in the transformer language models.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Drag Effect on Prats Problem Using Power-Law Saturating Fluid: Convective Instability

next chapter Comparing Two Different Implementations of OPC UA Clients

Adam, M., Wessel, M., Benlian, A.: AI-based chatbots in customer service and their effects on user compliance. Electron. Markets 31(2), 427–445 (2021). https://doi.org/10.1007/s12525-020-00414-7

Averbukh, V.L.: Evolution of human computer interaction. Sci. Vis. 12(5), 130–164 (2021). https://doi.org/10.26583/SV.12.5.11

Heyman, G., Huysegems, R., Justen, P., van Cutsem, T.: Natural language-guided programming. In: Onward! 2021 - Proceedings of the 2021 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Co-Located with SPLASH 2021, pp. 39–55 (2021). https://doi.org/10.1145/3486607.3486749

Kovács, L.: Elemző és robotizált folyamatautomatizálási rendszer fejlesztése nagy terhelésű ügyfélszolgálatok számára: Kutatási jelentések 2022/1 Miskolc-Egyetemváros, Magyarország: Miskolci Egyetemi Kiadó, p. 167 (2022). ISBN: 9789633582626

OpenAI., ChatGPT. https://chat.openai.com/. Accessed 2023

Xu, F.F., Vasilescu, B., Neubig, G.: In-IDE code generation from natural language: promise and challenges. ACM Trans. Softw. Eng. Methodol. 31(2), 1–47 https://doi.org/10.1145/3487569 (2022)

Shi, Y., Keneshloo, N.R., Reddy, C.K.: Neural abstractive text summarization with sequence-to-sequence models. ACM/IMS Trans. Data Sci. 2(1), 1–37 (2021). https://doi.org/10.1145/3419106

Haque, S., Eberhart, Z., Bansal, A., McMillan, C.: Semantic similarity metrics for evaluating source code summarization. In: IEEE International Conference on Program Comprehension, IEEE Computer Society, pp. 36–47 (2022)

Vaswani, A., et al.: Attention Is All You Need (2017). http://arxiv.org/abs/1706.03762

10.

Khurana, D., Koli, A., Khatter, K., Singh, S.: Natural language processing: state of the art, current trends and challenges. Multimed. Tools Appl. 82(3), 3713–3744 (2023). https://doi.org/10.1007/s11042-022-13428-4

11.

Ray, P.P.: ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. In: Internet of Things and Cyber-Physical Systems, vol. 3. KeAi Communications Co., pp. 121–154 (2023). https://doi.org/10.1016/j.iotcps.2023.04.003

12.

Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International Conference on Machine Learning. PMLR (2013)

13.

Zhou, C., et al.: A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT (2023). arXiv preprint arXiv:2302.09419. x0x0

14.

Liu, Y., et al.: RoBERTa: A robustly optimized BERT pretraining approach (2019). CoRR

15.

Li, J., et al.: Pretrained language models for text generation: A survey (2022). arXiv preprint arXiv: 2201.05273

16.

https://lambdalabs.com/blog/demystifying-gpt-3

17.

Lewis, M., et al.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019). arXiv preprint arXiv: 1910.13461

18.

Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

19.

Biswas, S.S.: Role of chat GPT in public health. Ann. Biomed. Eng. 51(5), 868–869 (2023)CrossRef

20.

Li, Jianning, et al. “ChatGPT in Healthcare: A Taxonomy and Systematic Review.” medRxiv (2023): 2023–03

21.

Biswas, S.: ChatGPT and the future of medical writing. Radiology 307(2), e223312 (2023)CrossRef

22.

Wang, C., et al.: Ethical Considerations of Using ChatGPT in Health Care. J. Med. Internet Res. 25, e48009 (2023)

23.

Hill-Yardin, E.L., et al.: A Chat (GPT) about the future of scientific publishing. Brain Behav. Immun. 110, 152–154 (2023)

24.

Rahimi, F., Abadi, A.T.B.: ChatGPT and publication ethics. Arch. Med. Res. 54(3), 272–274 (2023)CrossRef

25.

Zhu, J.-J., et al.: ChatGPT and environmental research. Environ. Sci. Technol. 57(46), 17667–17670 (2023)

26.

Surameery, N.M.S., Shakor, M.Y.: Use chat GPT to solve programming bugs. Int. J. Inf. Technol. Comput. Eng. (IJITC) 3(01), 17–22 (2023). ISSN: 2455–5290

27.

Lo, C.K.: What is the impact of ChatGPT on education? A rapid review of the literature. Educ. Sci. 13(4), 410 (2023)CrossRef

28.

Kohnke, L., Moorhouse, B.L., Zou, D.: ChatGPT for language teaching and learning. RELC J. 54(2), 537–550 (2023). https://doi.org/10.1177/00336882231162868

29.

Frieder, S., et al.: Mathematical capabilities of ChatGPT (2023). arXiv arXiv:2301.13867

30.

Pan, L., et al.: Semantic graphs for generating deep questions (2020). arXiv preprint arXiv:2004.12704

Title: Transformer Models in Natural Language Processing
Authors: László Kovács
László Csépányi-Fürjes
Walelign Tewabe
Publisher: Springer Nature Switzerland
Book: The 17th International Conference Interdisciplinarity in Engineering
Print ISBN: 978-3-031-54673-0

Electronic ISBN: 978-3-031-54674-7

Copyright Year: 2024
DOI: https://doi.org/10.1007/978-3-031-54674-7_14

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partners