Photo by writer | Ideogram.aiObvious Introduction
With the surgery of the major languages of Language (LLMS) in recent years, many of the most powerful LLM programs appear. The implementation of the LLM is introduced in the features that were not yet.
As time goes on, many llm models and products are available, each has its own benefits. Unfortunately, there is no regular way to access all these models, as each company can improve its outline. That's why being a tool for an open source as Grass It is helpful when you need regular access to your LLM applications without additional costs.
In this article, we will examine why Lidellm is beneficial to build a LLM programs.
Let's get into it.
Obvious Benefits 1: Consolidated access
The greatest benefit of Lilolm is its compliance with various model providers. The tool supports more than 100 different llm services through dry gaps, allowing us to find them any model supplier. It is especially helpful when your apps use many different models that need to work in exchange.
A few examples of senior models supported by DelllllM include:
- Open and the Azure Outrai, such as GPT-4.
- Anthropic, like Claude.
- The AWS Bedrock & Sagemaker, supporting models such as Amazon Titan and Claude.
- Google Vertex AI, like Gemini.
- Backing HUB of face and open model for open models like Llama and Misttral.
The prescribed format follows the outline of Openai, using its schema / completion. This means we can change models easily without requiring understanding the actual model SCHAME.
For example, here is the Python code to use Gemino's model gemini with Delight.
from litellm import completion
prompt = "YOUR-PROMPT-FOR-LITELLM"
api_key = "YOUR-API-KEY-FOR-LLM"
response = completion(
model="gemini/gemini-1.5-flash-latest",
messages=[{"content": prompt, "role": "user"}],
api_key=api_key)
response['choices'][0]['message']['content']You only need to get a model name and the appropriate API keywords from the model provider to achieve. This flexibility makes Layollm ready for apps that use multiple models or make model comparisons.
Obvious Profit 2: Following the cost and doing well
When working on the LLM programs, it is important to track the use of token and spend on each model you use and for all joint providers, especially in real conditions.
Litellm enables users to save the detailed log of the model in the Model API Call, providing all the required information to control the cost effectively. For example, the above-shape will be information about the use of Token, as shown below.
usage=Usage(completion_tokens=10, prompt_tokens=8, total_tokens=18, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=8, image_tokens=None))To access the hidden parameters of answering will also provide detailed information, including costs.
By releasing similar:
{'custom_llm_provider': 'gemini',
'region_name': None,
'vertex_ai_grounding_metadata': [],
'vertex_ai_url_context_metadata': [],
'vertex_ai_safety_results': [],
'vertex_ai_citation_metadata': [],
'optional_params': {},
'litellm_call_id': '558e4b42-95c3-46de-beb7-9086d6a954c1',
'api_base': '
'model_id': None,
'response_cost': 4.8e-06,
'additional_headers': {},
'litellm_model_name': 'gemini/gemini-1.5-flash-latest'}There is a lot of information, but the most important piece of `Answer_cost`, as the real trip will find during that call, although it is no longer released if the model provider provides free access. Users can also describe the customs's custom prices (for each token or a second) to calculate costs accurately.
The implementation of advanced expenses will allow users to cover the budget and limit, while coordinating the details of the cost of costs in analytics dashboard for easily integrated information. It is also possible to provide custom label tags to provide costs for access or departments.
By providing information for the use of the detailed cost, the Laelllm helps users and organizations use their cost of the BLM and Budget effectively.
Obvious Profit 3: Easy tackle for shipment
Lilollm is designed for easy shipment, whether you use local development or production area. With modest moderately the Python's library, we can use the laptop on our local laptop or handle the Docker's delegation without the need for additional complex.
Speaking about configuration, we can set a lilellm well through the Yaml Config file to write all the required information, such as the model name, API keys, and any settings for your Customizing. You can also use a backend database such as SQLITE or postgresql for the last of its status.
For data privacy, you are responsible for your privacy as a privacy of sending latllms yourself, but this method is too safer because data has not left your unique environment when sent to the LLM providers. One feature Lidellm provides for business users that Single-on Sign-on 1
Overall, Liellm offers the shipment options and configuration while storing data secure.
Obvious Benefits 4: Features of Fitness
Fitness is essential when building LLM applications, as we seek our application to serve even when faced with unexpected problems. To promote stability, a litellm offers many factors that are useful in the development of the application.
One aspect of the built-in lidellm, where users cannot be cache llm encourages answers for the same answers not entering the same cost or latency. It is an active feature if our app usually finds the same questions. The renting system is in flexibility, supports the maintenance of a remote and distance, such as the final data.
One of the KenWwolm is automatically returned, allows users to prepare a way where applications failed due to errors such as the expiry or default measures to re-activate application automatically. It is also possible to set additional fall methods, such as using another model if the application has already hit a resuming limit.
Finally, we may set the limit to the limitation of specified minutes (RPM) or tokens per minute (TPM) to limit the quality of use. It is a good way to combine a combination of some model to prevent failure and respect the needs of the app infrastructure.
Obvious Store
In the expedition of the LLM production, it has become the most convenient to build llm apps. However, of many models of models there, it is difficult to establish a LLM operations, especially in the form of a multi-modeling system. That is why Lidellm can help us create LLM applications well.
I hope this has helped!
Cornellius Yudha Wijaya It is a scientific science manager and the database author. While working full-time in Allianz Indonesia, she likes to share the python and data advice with social media and media writing. Cornellius writes to a variety of AI and a study machine.
