How to Use Pre-Trained Language Models for Regression | by Aden Haussmann | January, 2025

Why and how to convert mT5 into a regression metric for numerical prediction

My honors thesis was a Natural Language Processing (NLP) research project. It focuses on the production of multilingual texts in underrepresented languages. Because existing metrics performed very poorly in evaluating the results of models trained on the dataset I was using, I needed to train a learned regression metric.

Regression can be useful for many text tasks, such as:

• Sentiment analysis: Predict the strength of a positive or negative feeling instead of simple binary categorization.
• Average writing quality: Estimate how high the quality of a piece of text is.

In my use case, I needed a model to find out how good another model's prediction was for a given task. The rows of my data set contain a text input and a label, 0 (bad prediction) or 1 (good prediction).

• Input: Text
• Label: 0 or 1
• Job: Guess the probability of numbers between 0 and 1

But transformer-based models are generally used for production operations. Why use pre-trained LM…