学位論文

The Study on Knowledge Distillation Methodology Design for Natural Language Processing

自然言語処理における知識蒸留手法の設計に関する研究

eng

https://doi.org/10.15002/00031065

1

113

2024-09-15

Not Applicable (or Unknown)

32675甲第620号

2024-09-15

博士(工学)

Natural language processing (NLP) offers significant potential for analyzing massive amounts of information in public sectors. However, in real-world applications, generalization and robustness remain major concerns. These issues are exacerbated when effective datasets are limited due to confidentiality or privacy concerns that are common in public sectors. Additionally, as large language models (LLMs) become more prevalent, their resource-intensive nature and substantial training demands pose significant challenges. To tackle these challenges, this study introduces innovative methodologies centered around Knowledge Distillation (KD), a promising technique that transfers knowledge from larger, more powerful models (teachers) to lighter, simpler networks (students). In the first proposal (Chapter 2), Domain-Adapted Distillation and Reinforcement Ensemble (DARE) aims to improve generalization and robustness in data-constrained domains. Instead of relying on large-scale labeled training data from the target domain, DARE makes full use of only a small amount of unlabeled data and effectively selects informative samples from the synthetic data generated by a part-of-speech (POS) generator. Our approach is based on the ideas of reinforcement learning (RL) and knowledge distillation. Specifically, the distillation loss is used as the reward signal within the RL module. The RL selector identifies the most informative samples for the KD module, which in turn provides the loss used as the criterion for the RL selector. We evaluate our approach across various experimental settings, and the extensive results demonstrate that DARE outperforms other baseline methods in most tasks. Additionally, due to its robust training scheme, DARE proves particularly effective in low-resource scenarios. In the second proposal (Chapter 3), the Weakly Supervised Hybrid Attention Masking Capsule Neural Network (HAMCap) is proposed to address the fine-grained multiple entity-aspect sentiment classification task, a particularly challenging task in NLP. HAMCap employs a hybrid attention mechanism to capture both the internal semantic structure and the feature relationships between aspects and their context. Different from current local context focus (LCF) methods, we introduce a novel Partial Context Window (PCW) to pinpoint the aspect-related local context and utilize a Partial Context Mask (PCM) mechanism to model the strong associations within the context. Additionally, we explore the optimal architecture of teacher and student models within a KD framework in this research. We examine whether HAMCap can serve as a robust teacher model by comparing it with larger teacher models to assess the impact of selecting an appropriate teacher model without introducing additional distillation strategies. For a rigorous comparison and evaluation, we select 11 baseline models, categorized into four groups, to analyze the specific contributions and enhancements of our proposed methodology. Experimental results demonstrate that HAMCap consistently outperforms other baseline methods across most tasks. Moreover, we find that selecting an appropriate teacher model is significantly more crucial than opting for a larger one in the KD process. In addition, we provide a novel fine-grained domain dataset and establish annotation criteria for sentiment polarities in this work. In the third proposal (Chapter 4), building upon previous work, we summarize two remaining key challenges in KD. First, when there is a significant gap between the teacher model and the student model, knowledge transfer can be hindered, resulting in suboptimal learning outcomes for the student model. We refer to this issue as the capacity gap in this study. Second, we aim to implement a progressive distillation process to facilitate better knowledge transfer from the teacher to the student model. To address these challenges, we propose a novel approach called the Adversarial Training- based Dynamic Tutorial Knowledge Distillation Algorithm (TutorKD). This method regulates the progressive distillation process using a variable, trainable temperature tutor, which is implemented through a Gradient Reversal Layer (GRL). To further enhance robustness and generalization in the knowledge transfer process, we introduce an adversarial learning-based black box generator for data augmentation. Constructing such a generator in NLP is challenging due to the discrete nature of sentence space and the tendency of adversarial examples to produce non-fluent sentences. To tackle this, we propose an adversarial generator based on the Metropolis-Hastings Sampling Algorithm (MHA). Experimental results demonstrate that, compared to conventional KD methods, our approach consistently outperforms baseline methods and shows effectiveness in handling out-of-domain (OOD) tests and adversarial attacks. In conclusion, this study aims to optimize and integrate knowledge distillation to address various NLP challenges across different real-world scenarios and applications. This study focuses on designing KD methodologies to obtain models that exhibit strong generalization and robustness while also being lighter and more efficient.

Thesis