Neural Probabilistic Language Modeling: Overcoming the Curse of Dimensionality with Distributed Representations

Abstract

Statistical language modeling is the task of predicting the probability of word sequences in a given language. This task is complicated by the "curse of dimensionality," which arises when previously unseen word sequences appear during testing, making it difficult to estimate probabilities since these sequences differ significantly from those encountered during training. Traditional n-gram models address this challenge by leveraging overlapping word sequences. We propose an alternative approach that mitigates this issue by utilizing distributed word representations. These representations allow the model to generalize better by capturing semantic similarities, thus enabling it to predict word sequences more effectively. In this framework, the model learns both (1) distributed representations for individual words and (2) a probability function to predict word sequences based on these representations. This approach assigns high probability to unseen sequences if they contain words that are semantically similar to those encountered in the training data. To tackle the complexity of training models with millions of parameters, we employ neural networks to model the probability function. Our experimental results on two large text corpora demonstrate that this method significantly outperforms traditional n-gram models by using longer contexts for more accurate predictions.