TRAINING OF RECURRENT NEURAL NETWORKS IN A LIMITED TRAINING DATASET

Author(s): Tshub V.S., Tsvetkova O.L.

Rubric: Information technology

DOI: 10.21777/2500-2112-2023-4-79-85

Release: 2023-4 (45)

Pages: 79-85

Keywords: recurrent neural networks, transfer learning, limited data set, natural language processing

Annotation: Currently, transformer-based transfer learning methods are used to solve natural language processing problems, but they can be demanding on computing resources and memory. An alternative approach proposed in this article is related to the use of a pre-trained language model based on the recurrent neural network LSTM (Long Short- Term Memory), which allows natural language processing (tonality analysis) to be performed on a set containing text data. As a reference test of the proposed model for solving the problem of recognizing the “hate speech” in the text, other models based on transformers are considered. The degradation test, which evaluates the resilience of models to performance degradation with a decrease in the amount of training data, confirms the effectiveness of the model for a limited training dataset. The proposed LSTM model based on a recurrent neural network allows us to use a pre-trained language model from scratch and provide faster pre-learning in transformer-based models.

Bibliography: Tshub V.S., Tsvetkova O.L. TRAINING OF RECURRENT NEURAL NETWORKS IN A LIMITED TRAINING DATASET // Education Resources and Technologies. – 2023. – № 4 (45). – С. 79-85. doi: 10.21777/2500-2112-2023-4-79-85