The appearance of Chatgpt in 2022 has completely changed how to understand artificial intelligence. The amazing operation of Chatgpt has led to immediate development on other powerful llms.
We can probably say ChatGPT is a promised version of GPT-3. But compared to previous GPT versions, this time the Opelaai developers are many or just complex of model buildings. Instead, they designed a wonderful process that allows success.
In this article, we will talk about RLHF – a basic algorithm used in Chatgpt than the LLMS population restrictions. Although algorithm is based on the functionality of the Proximal (PPO), we will keep the meaning of the simplest, without consulting learning information, which is not a focus on this document.
NLP Development Before ChatGpt
To better fall into the context, let us remind how ILMS is built in the past, before the time period. In many cases, the development of the LLM contained two categories:
Primary training includes a language model – a function where the model is trying to predict a hidden token in context. The possuction may be produced by a hidden token model compared to the distribution of the world's true losing and repetition. In this way, the model reads the formation of the language of the language and the meaning of the words in the rear.
If you want to learn more about Present training and good frameworkSee, check my article about Bert.
Thereafter, the model is well organized in the low work, which can include different purposes, text summarizing, text translation, text storage, requiring a person's best response to allow its model properly and avoid extremes.
This is where the boundaries of good planning appears. Data annotation is usually a job that is taking time for people. Let's take a job to answer questions, for example. To create training samples, we will need manually installed data and answers. To all the question, we will need a direct response to a person. For example:
In fact, to train a LLM, we will need millions or even billions of that (question, answer) in pairs. This annotation process takes time and is not equal.
Rlhf
As he understood the main issue, now it is good time to enter the RLHF details.
If you have already used ChatGPT, you may have experienced the situation when Chatgpt asks you to choose a response that best fits your faster:
This information is actually used to further develop Chatgt. Let's understand how.
First, it is important to note that choosing the best answer between two options is a very simple one made of a person rather than to give a direct response to open question. The idea that will look directly from that: we just look for a person choose the answer from two possible options to create the data described.
A generation of response
In the LLMS, there are several possible methods of extracting feedback from the spread of opportunities foretold:
- Having a distribution of issuance kind In addition to the tokens, the model always decides token at the highest opportunity.
- Having a distribution of issuance kind In addition to the tokens, the model of the Token is at a time of the key to the opportunity to be allocated.
This second-sample method results in random behavior, which allows the production of various text. In the meantime, let's imagine that we produce many pairs of such a chronological order. Dataasset catches in pairs are written by people: All pairs, a person is asked what one of the output is fits the better installation. Defensive data is used in the next step.
In the case of RLHF, the specified dataset is made this way is called “Person's Answer”.
Reward model
After the dataset described, we use it to train the “reward”, its goal is to learn more numbers that appropriate or how much the temporary is provided. Well, We are looking for a reward model to produce good prices for good answers and bad prices for bad answers.
Speaking of the reward model, its construction is exactly like the first llm, without the last layer, when instead of taking the sequence of text, a floating model.
Necessary to pass the speedy immediately and the reply produced as input in a reward model.
Work of loss
You may also ask how the reward model will learn how to postpone it if not the data rates described. This is the right question. To look at it, we will use a lovely strategy: We will exceed a good and bad answer in a reward model, which will save two different measurements (rewards).
Then we will build the work of the loss that will be compared to them.
Let's connect to certain arguments to work properly and analyze their conduct. Below a connected toilet table:
Soon we can view two exciting information:
- If the difference between R₊ and R₋ it is incorrectThat is the best response to the lowest reward than worse, then the amount of loss will be a big difference, which means the model needs to be very corrected.
- If the difference between R₊ and R˚ is straightforwardThat is the best response to a high reward than worse, then the loss will be included within the lowest rates (0, 669), indicating that the model is well done good and bad answers.
The pleasant thing through the work that the model reads the appropriate Rewards of the Scriptural text, and we (people) don't have to exploit all the answer: just give the amount of the 6 binary: answer provided better or worse.
Training original llm
The trained Reward model is used to train the original llm. In that case, we can provide a series of new suggestions in the LLM, which will produce outgoing sequence. Then, that is motivated by installation, as well as a sequence of exit, a reward model is given to measure how beautiful those answers are.
After producing price ratings, that information is used as an original llm response, then makes weight updates. The easiest but beautiful way!
Most of the time, in the final step to repair the model, using algorithm to be reinforced by the Proximal Policy – PPO).
Even if it is not good for technology, if you are not familiar with learning right or PPO, you can imagine it as a backpropithration, as in normal machine learning.
Miss
During compliance, only the original model is used. At the same time, the model can be continuously developed after collecting user motives and applies to time to measure which two better answers.
Store
In this article, we have learned RLHF – a very efficient and corrupted process to train modern llms. A good link of llm with a reward model allows us to facilitate human explanation, which requires great efforts in the past when made by good functions.
RLHF is used in the spine of many popular models such as ChatGPT, Claude, Gemini, or illegal.
Resources
All photos unless noted in another way is a writer
