Construction of Constrention of LLM can see, imagine, and include: Using O3 for multiple installation and formalization

Normal “Text In, Outgoing text” Paradigm will take you until now.

The actual requests that bring real value should be able to test the visual test, the reason for complex problems, and display results used by programs.

In this post, we will build this stack by combining three powerful skills: Multimodal Installation, Reasoning, including A formal release.

I ess this, we'll travel by hand-in-depth: to build a Time-Series Anomaly Detection System For the details of e-commerce order using Oveaii's Oveaii model. Specifically, we will show how we can compare the O3 consultation capacity with photos and emit by Emit confirmed JSON, so that the minimum system can easily get rid of.

At the end, our app is:

• Intend: Analyze e-commerce charts order the volume time
• Suppose: Find unusual patterns
• Integrate: Displaying Anomaly reported

You will go with an active code You can re-use the various use charges passing across the surface of Anomaly.

Let's get in.

You're interested in reading a wider location of the llms used in an oranaly's getting? Check my previous post: To enhance your Anomaly acquisition with llms, When I summarize the 7ths of gains that you should not miss.

---

1. State study

In this sentence, we aim to create a solution of anomaly to get the e-commerce order series.

In this study, we generated three sets of and the bolt daily order data. Datassets represent three different days of daily order over one month period. Doing it is obvious, we have a shader the weekends. Ix-axis shows the day of the week.

Each figure contains some form of anomaly (Can you find them?). We will later use those figures to test our solution to Anomaly and see if it can return those analimalies.

2. Our solution

2.1 Views all

Unlike traditional machine learning methods that require a feature of the feature and engineering, our current method is very simple. Works with the following steps:

1. We prepare for the number of e-commerce order data.
2. It moves the O3 consultation model, asking for a photo of the Time Series, and determines whether an unfamiliar pattern is available.
3. The O3 model will be issuing the findings in the JSON format specified before.

And that's it. Simple.

Of course, to bring this solution, we need to enable the O3 model to take photos inputs and remove the organized result. We will see how we can do that soon.

2.2 to set up a model model

As mentioned earlier, we will use O3 model, which is the O3 consultation model from Openai that can face difficult problems with many steps through the Kingdom work. Specifically, we will use the end of the Azure Openai to call the model.

Make sure to put the end, the API key, and the name of the .env The file, we can continue to set the llm client:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

from openai import AzureOpenAI
from dotenv import load_dotenv
import os

load_dotenv()

# Setup LLM client
endpoint = os.getenv("api_base")
api_key = os.getenv("o3_API_KEY")
api_version = "2025-04-01-preview"
model_name = "o3"
deployment = os.getenv("deployment_name")

LLM_client = AzureOpenAI(
    api_key=api_key,  
    api_version=api_version,
    azure_endpoint=endpoint
)

We use the following commands as an O3 model program system (organized by GPT-5):

instruction = f"""

[Role]
You are a meticulous data analyst.

[Task]
You will be given a line chart image related to daily e-commerce orders. 
Your task is to identify prominent anomalies in the data.

[Rules]
The anomaly kinds can be spike, drop, level_shift, or seasonal_outlier.
A level_shift is a sustained baseline change (≥ 5 consecutive days), not a single point.
A seasonal_outlier happens if a weekend/weekday behaves unlike peers in its category. 
For example, weekend orders are usually lower than the weekdays'.
Read dates/values from axes; if you can’t read exactly, snap to the nearest tick and note uncertainty in explanation.
The weekends are shaded in the figure.
"""

In the Mage above, we best described the role of llm, the work that the llm should complete, and the rules of the LLM should follow.

To limit the severity of our study study, we defined only four different types of Anomaly that needed to see. We also gave clear descriptions of those types of Anomaly to remove Ambiguity.

Finally, we have entered some form of domain about e-commerce patterns, IE, low orders for the week expected compared to the days of the church. Installing the Domain Know – How is it viewed as a good way to direct the model analysis process.

Now that we are set up our model, let us consider how to prepare a picture in the O3 model to eat.

2.3 To prepare a picture

To empower the Multimodal skills of O3, we need to give prices in a specific format, that is, either in the Web URLs are available in the public or as the data URL installed on Base64. As our statistics are produced in the area, we will use the second method.

What is to install Base64 Base64 is a way to represent the binary data (such as our image files) using only the text letters safe to transfer online transfer. It changes the binary image data into the letters, numbers, and few symbols.

And what about the data URL? The data URL version of the URL that stirs the file content directly to the URL cable, rather than point to the file location.

We can use the following work to manage this automatically conversion:

import io
import base64

def fig_to_data_url(fig, fmt="png"):
    """
    Converts a Matplotlib figure to a base64 data URL without saving to disk.

    Args:
    -----
    fig (matplotlib.figure.Figure): The figure to convert.
    fmt (str): The format of the image ("png", "jpeg", etc.)

    Returns:
    --------
    str: The data URL representing the figure.
    """

    buf = io.BytesIO()
    fig.savefig(buf, format=fmt, bbox_inches="tight")
    buf.seek(0)
    
    base64_encoded_data = base64.b64encode(buf.read()).decode("utf-8")
    mime_type = f"image/{fmt.lower()}"
    
    return f"data:{mime_type};base64,{base64_encoded_data}"

In fact, our first job that keeps the number of matplotlib in the memory puffer. Includes binary data information such as Base64 text and forcing it in the URL format you want.

To think that we have access to daily order information, we can use the following work to produce a framework and convert it into the correct URL data format.

def create_fig(df):
    """
    Create a Matplotlib figure and convert it to a base64 data URL.
    Weekends (Sat–Sun) are shaded.

    Args:
    -----
    df: dataframe contains one profile of daily order time series. 
        dataframe has "date" and "orders" columns.

    Returns:
    --------
    image_url: The data URL representing the figure.
    """

    df = df.copy()
    df['date'] = pd.to_datetime(df['date'])

    fig, ax = plt.subplots(figsize=(8, 4.5))
    ax.plot(df["date"], df["orders"], linewidth=2)
    ax.set_xlabel('Date', fontsize=14)
    ax.set_ylabel('Daily Orders', fontsize=14)

    # Weekend shading
    start = df["date"].min().normalize()
    end   = df["date"].max().normalize()
    cur = start
    while cur <= end:
        if cur.weekday() == 5:  # Saturday 00:00
            span_start = cur                                      # Sat 00:00
            span_end   = cur + pd.Timedelta(days=1)               # Mon 00:00
            ax.axvspan(span_start, span_end, alpha=0.12, zorder=0)
            cur += pd.Timedelta(days=2)                           # skip Sunday 
        else:
            cur += pd.Timedelta(days=1)

    # Title
    title = f'Daily Orders: {df["date"].min():%b %d, %Y} - {df["date"].max():%b %d, %Y}'
    ax.set_title(title, fontsize=16)

    # Format x-axis dates
    ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %d')) 
    ax.xaxis.set_major_locator(mdates.WeekdayLocator(interval=1))

    plt.tight_layout()

    # Obtain url
    image_url = fig_to_data_url(fig)

    return image_url

1-3% values ​​produced by the above planning process.

2.4 Relevant releasing

In this section, let us consider it to ensure that the O3 model removes JSON's format instead of the free form text. This is known as “formal issued,” and is one of the important services to integrate the default default default.

To achieve that, we start by explaining the schema that controls the expected production. We will be using the Pydantic model:

from pydantic import BaseModel, Field
from typing import Literal
from datetime import date

AnomalyKind = Literal["spike", "drop", "level_shift", "seasonal_outlier"]

class DateWindow(BaseModel):
    start: date = Field(description="Earliest plausible date the anomaly begins (ISO YYYY-MM-DD)")
    end: date = Field(description="Latest plausible date the anomaly ends, inclusive (ISO YYYY-MM-DD)")

class AnomalyReport(BaseModel):
    when: DateWindow = Field(
        description=(
            "Minimal window that contains the anomaly. "
            "For single-point anomalies, use the interval that covers reading uncertainty, if the tick labels are unclear"
        )
    )
    y: int = Field(description="Approx value at the anomaly’s most representative day (peak/lowest), rounded")
    kind: AnomalyKind = Field(description="The type of the anomaly")
    why: str = Field(description="One-sentence reason for why this window is unusual")
    date_confidence: Literal["low","medium","high"] = Field(
        default="medium", description="Confidence that the window localization is correct"
    )

Our Pydantic Schema tries to capture both the most likely features and the relevant Anomomalies found. In each field, we clarify the nature of its data (eg. int Number values, Literal With a fixed set of selection, etc.).

Also, we use Field The function is to provide personalized definitions for each key. Those definitions are very important as they are effective as the O3 use items, to understand the Semantic description of each part.

Now, we have compiled a lot of inputs and produces formal, time to cover them together in one llm call.

2.5 O3 Model Vovel

NE3 Communication using multimodal input and systematic output, we use LLM_client.beta.chat.completions.parse() API. Some important issues include:

• model: The name of submission;
• messages: The message item sent to O3 model;
• max_completion_token: The higher number of modeling model can produce with its last answer. Note that the models are a reasonable models like O3, they will produce reasons_tokens inside to “think about” problem. Meantime max_completion_token only reduces visible tokens that are receiving users;
• response_format: Pydantic model describing the expected JSON Scon structure;
• reasoning_effort: The timeline that asks for how much O3 should use the consultation. The options available include low, center.

We can explain the work to help contact the O3 model:

def anomaly_detection(instruction, fig_path, 
                      response_format, prompt=None, 
                      deployment="o3", reasoning_effort="high"):

    # Compose messages
    messages=[
            { "role": "system", "content": instruction},
            { "role": "user", "content": [  
                { 
                    "type": "image_url",
                    "image_url": {
                        "url": fig_path,
                        "detail": "high"
                    }
                },
            ]} 
    ]

    # Add prompt if it is given
    if prompt is not None:
        messages[1]["content"].append({"type": "text", "text": prompt})

    # Invoke LLM API
    response = LLM_client.beta.chat.completions.parse(
        model=deployment,
        messages=messages,
        max_completion_tokens=4000,
        reasoning_effort=reasoning_effort,
        response_format=response_format
    )

    return response.choices[0].message.parsed.model_dump()

note that messages The item accepts both the text and the content of the image. As we will use only statistics to earn model, instant message is selected.

We put the "detail": "high" Enabling the high-image processing. Our study is currently, this is more likely as we need O3 to better learn fine information such as axis ticks, Data Point values, and subtle patterns. However, keep in mind that the detailed process of high information will receive multiple tokens and API costs.

Finally, through .parsed.model_dump()We're changing by JSS closest Python dictionary.

That is what is being done. Let's look at some of the following results.

---

3. The results

In this stage, we will install the previously generated statistics in the O3 model and ask us to identify the existing animomalies.

3.1 Spike Anomaly

# df_spike_anomaly is the dataframe of the first set of synthetic data (Figure 1)
spike_anomaly_url = create_fig(df_spike_anomaly)

# Anomaly detection
result = anomaly_detection(instruction,
                          spike_anomaly_url,
                          response_format=AnomalyReport,
                          reasoning_effort="medium")
print(result)

On the phone above, the spike_anomaly_url Is the URL of Figure Data 1. The resulting releases shown below:

{
  'when': {'start': datetime.date(2025, 8, 19), 'end': datetime.date(2025, 8, 21)}, 
  'y': 166, 
  'kind': 'spike', 
  'why': 'Single day orders jump to ~166, far above adjacent days that sit near 120–130.', 
  'date_confidence': 'medium'
}

We see that the O3 model also depends on the output in the design format. Now, we can take this effect and produce a detecture program:

# Create image
fig, ax = plt.subplots(figsize=(8, 4.5))
df_spike_anomaly['date'] = pd.to_datetime(df_spike_anomaly['date'])
ax.plot(df_spike_anomaly["date"], df_spike_anomaly["orders"], linewidth=2)
ax.set_xlabel('Date', fontsize=14)
ax.set_ylabel('Daily Orders', fontsize=14)

# Format x-axis dates
ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %d'))  
ax.xaxis.set_major_locator(mdates.WeekdayLocator(interval=1)) 

# Add anomaly overlay
start_date = pd.to_datetime(result['when']['start'])
end_date = pd.to_datetime(result['when']['end'])

# Add shaded region
ax.axvspan(start_date, end_date, alpha=0.3, color='red', label=f"Anomaly ({result['kind']})")

# Add text annotation
mid_date = start_date + (end_date - start_date) / 2  # Middle of anomaly window
ax.annotate(
    result['why'], 
    xy=(mid_date, result['y']), 
    xytext=(10, 20),  # Offset from the point
    textcoords='offset points',
    bbox=dict(boxstyle='round,pad=0.5', fc='yellow', alpha=0.7),
    arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0.1'),
    fontsize=10,
    wrap=True
)

# Add legend
ax.legend()

plt.xticks(rotation=0)
plt.tight_layout()

The production of produced looks like this:

We see that the O3 model points out the Spike Anonyaly presented in the first data data.

It's not wrong, especially if you think of the fact that we did not know the general training, by issuing the LLM.

3.2 Level Shift Anomaly

# df_level_shift_anomaly is the dataframe of the 2nd set of synthetic data (Figure 2)
level_shift_anomaly_url = create_fig(df_level_shift_anomaly)

# Anomaly detection
result = anomaly_detection(instruction,
                          level_shift_anomaly_url,
                          response_format=AnomalyReport,
                          reasoning_effort="medium")
print(result)

The result issues are shown below:

{
  'when': {'start': datetime.date(2025, 8, 26), 'end': datetime.date(2025, 9, 2)}, 
  'y': 150, 
  'kind': 'level_shift', 
  'why': 'Orders suddenly jump from the 120-135 range to ~150 on Aug 26 and remain elevated for all subsequent days, indicating a sustained baseline change.', 
  'date_confidence': 'high'
}

Also, we see that model is accurately identified that the “Level_shift” Anomaly is located on site:

3.3 Annual Disease

# df_seasonality_anomaly is the dataframe of the 3rd set of synthetic data (Figure 3)
seasonality_anomaly_url = create_fig(df_seasonality_anomaly)

# Anomaly detection
result = anomaly_detection(instruction,
                          seasonality_anomaly_url,
                          response_format=AnomalyReport,
                          reasoning_effort="medium")
print(result)

The result issues shown below:

{
  'when': {'start': datetime.date(2025, 8, 23), 'end': datetime.date(2025, 8, 24)}, 
  'y': 132, 
  'kind': 'seasonal_outlier', 
  'why': 'Weekend of Aug 23-24 shows order volumes (~130+) on par with surrounding weekdays, whereas other weekends consistently drop to ~115, making it an out-of-season spike.', 
  'date_confidence': 'high'
}

This is a difficult case. However, our O3 model managed well, with accurate distribution and clear follow-up. Impressively impressive:

---

4. Summary

Congratulations! We have successfully develop a solution for an oralaly service that is full-time data in the discretion of seeing and going.

By eating the daily order farms in the O3 consultation model and compelled output in JSON Schema, the llm has identified three different types of local Anomaly. All of this is found without training any ML model. It's amazing!

When we take a step back, we can see that the architect displays a comprehensive three skills compiling pattern:

• Intend: Multimodal Input to allow the model to eat direct statistics.
• Suppose: Power to reason for step in step by deal with complex problems.
• Integrate: Formal issues that Downstream can easily use (eg, to create visual).

Multimalal combination of multimetor + organized consultation + creates a realistic foundation for useful LLM applications.

You now have the right construction blocks. What do you want to build next?