Abbas Saliimi, Lokman (2024) Modified word representation vector based scalar weight for contextual text classification. PhD thesis, Universti Malaysia Pahang Al-Sultan Abdullah (Contributors, Thesis advisor: Mohamed Ariff, Ameedeen).
|
Pdf
Modified word representation vector based scalar weight for contextual text classification.pdf - Accepted Version Download (5MB) | Preview |
Abstract
This thesis investigates contextual text classification, which is the process of categorising textual data into different classes or categories based on its meaning within a given context. Central to this process is the representation of words through vectors for computational interpretation. Current practices employ Large Language Models (LLMs) to generate contextualised word representation vectors, achieved through pre-training the LLM on vast corpora that enables it to grasp intricate language patterns and context. For contextual text classification, the pre-trained LLM is further train on classificationspecific labeled data in a process called fine-tuning. Although this approach is currently considered the most optimal in the field, it poses a notable challenge due to the substantial demand for computing resources stemming from the vast number of trainable parameters in LLMs. Furthermore, although pre-trained LLMs can generate contextualised word representation vectors, they lack the flexibility to modify the semantic significance of these vectors outside of the LLM, necessitating fine-tuning for the modification of word vectors. To bridge this gap, a five-phase research methodology is structured to propose and evaluate an algorithm enabling the external modification of LLM-generated word vectors using scalar values as the focus weightage. To validate this algorithm, the modified word vectors are compared with original LLM-generated word vectors to evaluate their reflection of the intended context. In addition, a contextual text classification experiment is conducted using benchmarked datasets to assess the performance of the modified word vectors in the targeted classification task. For this experiment, the modified word vectors serve as input to train a Machine Learning (ML) model for the text classification process, aiming for the developed ML model to have a significantly smaller parameter count. This experiment aims to determine the effectiveness of the modified word vectors in contextual text classification tasks, utilizing a more computationally efficient approach. Based on the acquired results, the experiments reveal that the modified word vectors algorithm can effectively alter original LLM-generated word vectors to reflect intended contexts and can outperform baseline scores in contextual text classification tasks. Evaluation metrics including Accuracy, Precision, Recall, and F1 score are employed in the evaluation process, with Accuracy and F1 score serving as primary metrics. The evaluation showcases significant improvements, with the test ML model achieving a best accuracy score of 0.571, a 46% increase from the baseline, and a best F1 score of 0.727, a 30% increment from the baseline. Overall, this thesis presents five contributions: the proposed modified word vectors algorithm, the new contextual classification dataset named QCoC, the efficient question-type classifier based on the feed-forward neural network algorithm, the potential transferability of the presented work to other domains, and the practical implications of the presented work towards cases where computational resources are limited or costly.
| Item Type: | Thesis (PhD) |
|---|---|
| Additional Information: | Thesis (Doctor of Philosophy) -- Universiti Malaysia Pahang – 2024, SV: Associate Professor Dr. Mohamed Ariff Ameedeen, NO. CD: 13668 |
| Uncontrolled Keywords: | Large Language Models (LLMs) |
| Subjects: | Q Science > QA Mathematics > QA75 Electronic computers. Computer science |
| Faculty/Division: | Institute of Postgraduate Studies Faculty of Computing |
| Depositing User: | Mr. Mohd Fakhrurrazi Adnan |
| Date Deposited: | 30 May 2025 02:36 |
| Last Modified: | 30 May 2025 02:36 |
| URI: | http://umpir.ump.edu.my/id/eprint/44632 |
| Download Statistic: | View Download Statistics |
Actions (login required)
 |
View Item |