Supervised learning
The primary purpose of supervised learning is to train a model using input/output pairs (e.g. documents labeled with their respective categories) as opposed to an unsupervised model which is trained on unlabeled data. The quality of output produced by supervised models can often be improved through data preprocessing (cleaning), feature engineering/selection and normalization; each of these steps will typically increase the amount of time spent training the model.
Self-attention
Self-attention mechanisms are used to enable a model (such as BERT or GPT) to learn about relationships between words. Models such as BERT and GPT utilize both supervised learning (when labels are provided), and self-attention mechanisms to enable them to identify how relationships exist between words in a given sentence.
BERT and GPT were initially trained via self-supervised learning. This means that instead of using human labeling to create prediction targets, prediction targets are created based upon the text itself. Both models do however rely on well-prepared training data. For example, BERT was trained by having it predict missing words. The ability to find relationships between words that are far away from one another, and the potential for the simultaneous processing of all tokens in a sequence during training, allow models utilizing self-attention to leverage large scale pre-training into multiple other areas of NLP.
Limitations
Each method has its own limitations. While a great deal of cost savings can be achieved by training a supervised model, costly labeling must occur prior to training. Additionally, there is no guarantee that the real world data you wish to process will resemble your training set. There are numerous drawbacks associated with large transformer-based models. They can take extensive amounts of computational power to run, and while they are capable of producing excellent results, it can be challenging to determine why certain predictions were made. When focusing on smaller, simpler, tasks, less computationally intensive, and easier to interpret supervised models are available.
In practice
Document classification in my organization requires reliable category labels and high-quality samples of representative examples. We reduce our burden of training our models by leveraging pre-trained models, however, we must still clean/preprocess our input data, assess the accuracy of our outputs, safeguard confidential/sensitive information and consider costs.