in production, actively responding to users' tests. However, now you want to improve your model to treat a large fraction of customer applications successfully. How do you talk to this?
In this article, I discuss the situation when you already have a LLM and you want to analyze and expand its functionality. I will discuss the ways I use to reveal where the LLM is also valid when it requires improvement. In addition, I will discuss the tools I use to improve my LLM performance, with tools such as anthropic adtizer.
In short, I follow the three-step process to quickly improve the operation of my llm:
- Analyze the llm exit
- It also improves the most powerful areas for the demand
- Analyze and Indere
Content
Motive
My motivation for this article is that I often find myself in the situation described in Intro. I already have my llm up and run; However, he does not act as expected or to achieve customer expectations. For many experiences of analyzing my llms, I have created this simple three-step process I always use to improve llms.
Step 1: Analyzing of the llm exit
The first step to improve your llms should always be analyzing their effect. To be able to see high on your stage, I strongly recommend using the trace of the trace manager, such as Langfus or PressLayer. These tools make it easy to collect all your LLM supplier in one place, ready for analysis.
Now I will discuss some ways I use to analyze my llm exit.
A handful examination of an unripe effect
The easiest way to analyze your llm effect to manage your lelling money. You must collect your finalization of the LLM, read the entire crucial context in the model, and which issuing a given model. I find this wonderful way in problems reveal. For example, I get:
- Containing context (part of my context is doubled due to planning error)
- Missed Context (I didn't satisfy all the information I expect in my llm)
- etc..
Manual assessment of data should not be monitored. Looking properly the data hand gives you the understanding of the data that makes, which is hard to find in any other way. In addition, I also find that I have to examine many database points than before I want to spend time.
For example, Suppose it takes 5 minutes to manually to manage one example for installation. My intuition often tells me that I may have spent 20-30 minutes in this, and thus check out the 4-6 data points. However, I find that you should usually spend a long time in this part of the process. I recommend at least at this time, so instead of spending 30 minutes I checked by hand, using 2.5 hours. At first, you will think this is a lot of time to use the hand test, but you will find you to save you more time later. In addition, compared with the rest of the 3 weeks of 3 weeks, 2.5 hours is an amount of the time.
Group questions According to Taxonomy
Sometimes, you will not find all your answers in simple analysis of your data. In those cases, I would move on to a lot of analysis of my data. This is compared to the first way, which I look as eligible from checking for each data point.
Accumulate users' Questions According to Teponomity is a successful way to better understand what users expect in your llm. I will give an example of doing this easier to understand:
Imagine you are Amazon, and you have a LLM customer service for managing income customers' questions. In this case, taxomy will look something like this:
- Refund requests
- Talk to one's requests
- Questions about individual products
- …
I will check the last time user questions and put it manually on this Tax. This will tell you what questions are most common, and what it should focus on responding well. He often detects that the submission of each stage will follow the distribution of pasta, many of the many specific sections.
Additionally, he informs you that customer application was successfully answered. With this information, you can now find out what kind of questions is exposed and what your ellom is right. Perhaps the llm transfers customer questions to people when requested; However, it will struggle at the information about the product. In this case, you should focus on your efforts to develop a group of questions that fights too well.
Llm as judge in gold dataset
Another measuring method I use to analyze my llm exit to create gold data for examples installation and use the LLM as a judge. This will help you make changes to your llm.
Continuing to an example of customer support from previous, you can create a list of 50 (real) questionnaire and the answer you are looking for from each of them. Whenever you make changes to your LLM (modify the extra version version, add the additional context, … This will save you the biggest time testing the llm out whenever you update your llm.
If you want to learn more about the llm as a judge, you can read my TDS article on topic here.
Step 2: Developing Your LLM
Finished with the first step, and now you want to use those ways to improve your llm. At this stage, I discuss how I approach this step to improve my performance of my llm.
When I get important news, for example, when I test the data by hand, I always fix those start. This, for example, can get unnecessary noise added to the City of the LLM, or Typos at my institutions. When I'm done, I continue to use certain tools.
One tool I use is in Prompt Optimizers, such as anthropic with these tools, usually quickly include examples of output installation. To do, for example, you enter a certificate that uses your customer service agents, as for examples of customer cooperation when the Failed LLM fails. The fastest Optimizer will analyze the Prompt and your examples and return the advanced version of your Prompt. You will see the progress similar to:
- Advanced building in your tingling, for example, uses Markdown
- Hostility on the edge. For example, treating cases where the user reads a customer support agent about the subjects that are completely unrelated, such as asking “what is the weather in New York today?”. The fastest Optimizer can add something like “If the question is not related to Amazon, tell the user that you are designed to answer questions about Amazon”.
If I have a lot of data, such as questions that collect user questions or gold data, I am analyzing with this data, and I created the amount of value. The graph of the value of the value highlights different development available, such as:
- Advanced Case Management in System System
- Use a Better RAG Mode for upgraded RAG
You have organized these data points in 2D grid, as low. You should prioritize things in the surface area because they give a lot of value and require little effort. Usually, anyway, things are found in diagonal, where the enhanced value links tightly with the highest effort.
I put all my suggestions to improve in the work graph of the amount, and I choose gradually as high as possible. This is the most effective way to solve the issues that are very stressful for your llm, has a positive impact on the most significant number of customers available for the amount of effort.
Step 3: Analyze and enter
The last step in my three-step process is to measure my llm and Itater. There are plethora strategies for the strategies you can use to examine your llm, many I associate with in my article on the subject.
Complete, you create certain llMS functionality Metrics, and make sure those metrics are developed from the changes that you run on stage 2. After using your model adequately or if you should continue to improve the model. I usually apply to 80% law, which says 80% work% are best enough for all cases. This is not 80% real accurately. Rather highlight the point that you do not need to create a complete model, but rather to create a model which is Good enough enough.
Store
In this article, I have negotiated this scenario when you already have a llm in production, and you want to analyze and improve your llm. I approach this situation by the first commentary of model and effects, perhaps with full text testing. After I certify that I really understand the dataset and how the model treats, I have returned many metrics, as questions in the collection group and using the LLM as a judge. Following this, I use the development based on my previous floor of the stage, and finally, I check that my improvement has worked as it was intended.
👉 I have found in the community:
🧑💻 Contact your
🔗 LickDin
🐦 x / Twitter
✍️ Medium
Or read some of my articles:
