Min-GPT: A Small Dataset Approach to Language Modeling

Abstract

The paper aims to develop a compact version of GPT that can process and generate coherent sentences using a limited dataset. This miniature model demonstrates language model adaptability and efficiency, showcasing how a reduced architecture can still handle essential language generation tasks effectively. The Min-GPT model investigates the capabilities of a scaled-down GPT model in Natural Language Processing (NLP) tasks using minimal data and computational resources. By reducing the number of parameters and dataset size, this paper evaluates how core elements of the GPT architecture function in a compact model, making sophisticated language models more accessible and practical for smaller-scale applications.
The implemented model utilizes a token-based character encoding scheme, uses a transformer architecture with self-attention mechanisms and multi-head attention. It incorporates a vocabulary derived from the dataset, token embeddings, and position embeddings to capture contextual relationships. The model was trained with a batch size of 64 and a block size of 258, utilizing 6 attention heads and 6 transformer layers with a dropout rate of 0.2. Optimization was performed using the AdamW optimizer with a learning rate of 3e-4. The evaluation was conducted using perplexity as a metric to measure the effectiveness of text generation.
The model was tested by generating text sequences given an initial prompt. The results indicate that despite the reduced scale, Min-GPT retains fundamental NLP capabilities, generating coherent sentences while adhering to learned patterns. However, limitations were observed in diversity and long-term dependencies due to the restricted dataset size. Additionally, visualization techniques such as loss vs. batch number plots were utilized to monitor training progress and assess convergence.
This study highlights the viability of compact transformer-based models in resource-limited settings, demonstrating trade-offs between computational efficiency and linguistic expressiveness. Future enhancements could include fine-tuning on pre-trained language models and employing data augmentation techniques to improve text diversity and fluency.