In this lesson, we create an advanced voice agent AI using existing facets, and keeps the whole pipe easier enough to work on Google Colab. It includes ranking for speech recognition, flan-t5 consultation of the natural language, as well as the bark of speech, all connected by transformers' pipes. By doing this, we avoid strong leaning, API buttons, or complex sets, and focuses on how to change voice logical conversation. Look Full codes here.
!pip -q install "transformers>=4.42.0" accelerate torchaudio sentencepiece gradio soundfile
import os, torch, tempfile, numpy as np
import gradio as gr
from transformers import pipeline, AutoTokenizer, AutoModelForSeq2SeqLM
DEVICE = 0 if torch.cuda.is_available() else -1
asr = pipeline(
"automatic-speech-recognition",
model="openai/whisper-small.en",
device=DEVICE,
chunk_length_s=30,
return_timestamps=False
)
LLM_MODEL = "google/flan-t5-base"
tok = AutoTokenizer.from_pretrained(LLM_MODEL)
llm = AutoModelForSeq2SeqLM.from_pretrained(LLM_MODEL, device_map="auto")
tts = pipeline("text-to-speech", model="suno/bark-small")We include the required libraries and download the first three Hugging pipes set automatically to use GPU when it is available. Look Full codes here.
SYSTEM_PROMPT = (
"You are a helpful, concise voice assistant. "
"Prefer direct, structured answers. "
"If the user asks for steps or code, use short bullet points."
)
def format_dialog(history, user_text):
turns = []
for u, a in history:
if u: turns.append(f"User: {u}")
if a: turns.append(f"Assistant: {a}")
turns.append(f"User: {user_text}")
prompt = (
"Instruction:n"
f"{SYSTEM_PROMPT}nn"
"Dialog so far:n" + "n".join(turns) + "nn"
"Assistant:"
)
return promptWe quickly describe the program that guides our agent that is shorter and planned, and uses the function of the past conversation format and user's immediate effort to generate a helper. Look Full codes here.
def transcribe(filepath):
out = asr(filepath)
text = out["text"].strip()
return text
def generate_reply(history, user_text, max_new_tokens=256):
prompt = format_dialog(history, user_text)
inputs = tok(prompt, return_tensors="pt", truncation=True).to(llm.device)
with torch.no_grad():
ids = llm.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=0.7,
do_sample=True,
top_p=0.9,
repetition_penalty=1.05,
)
reply = tok.decode(ids[0], skip_special_tokens=True).strip()
return reply
def synthesize_speech(text):
out = tts(text)
audio = out["audio"]
sr = out["sampling_rate"]
audio = np.asarray(audio, dtype=np.float32)
return (sr, audio)We build three basic activities of the voice agent: Transferring changes the recorded sound using the pione, the_the variety_speech production. Look Full codes here.
def clear_history():
return [], []
def voice_to_voice(mic_file, history):
history = history or []
if not mic_file:
return history, None, "Please record something!"
try:
user_text = transcribe(mic_file)
except Exception as e:
return history, None, f"ASR error: {e}"
if not user_text:
return history, None, "Didn't catch that. Try again?"
try:
reply = generate_reply(history, user_text)
except Exception as e:
return history, None, f"LLM error: {e}"
try:
sr, wav = synthesize_speech(reply)
except Exception as e:
return history + [(user_text, reply)], None, f"TTS error: {e}"
return history + [(user_text, reply)], (sr, wav), f"User: {user_text}nAssistant: {reply}"
def text_to_voice(user_text, history):
history = history or []
user_text = (user_text or "").strip()
if not user_text:
return history, None, "Type a message first."
try:
reply = generate_reply(history, user_text)
sr, wav = synthesize_speech(reply)
except Exception as e:
return history, None, f"Error: {e}"
return history + [(user_text, reply)], (sr, wav), f"User: {user_text}nAssistant: {reply}"
def export_chat(history):
lines = []
for u, a in history or []:
lines += [f"User: {u}", f"Assistant: {a}", ""]
text = "n".join(lines).strip() or "No conversation yet."
with tempfile.NamedTemporaryFile(delete=False, suffix=".txt", mode="w") as f:
f.write(text)
path = f.name
return pathWe include associated activities about our agent: Clear_history reset the conversation, VOICE look Full codes here.
with gr.Blocks(title="Advanced Voice AI Agent (HF Pipelines)") as demo:
gr.Markdown(
"## 🎙️ Advanced Voice AI Agent (Hugging Face Pipelines Only)n"
"- **ASR**: openai/whisper-small.enn"
"- **LLM**: google/flan-t5-basen"
"- **TTS**: suno/bark-smalln"
"Speak or type; the agent replies with voice + text."
)
with gr.Row():
with gr.Column(scale=1):
mic = gr.Audio(sources=["microphone"], type="filepath", label="Record")
say_btn = gr.Button("🎤 Speak")
text_in = gr.Textbox(label="Or type instead", placeholder="Ask me anything…")
text_btn = gr.Button("💬 Send")
export_btn = gr.Button("⬇️ Export Chat (.txt)")
reset_btn = gr.Button("♻️ Reset")
with gr.Column(scale=1):
audio_out = gr.Audio(label="Assistant Voice", autoplay=True)
transcript = gr.Textbox(label="Transcript", lines=6)
chat = gr.Chatbot(height=360)
state = gr.State([])
def update_chat(history):
return [(u, a) for u, a in (history or [])]
say_btn.click(voice_to_voice, [mic, state], [state, audio_out, transcript]).then(
update_chat, inputs=state, outputs=chat
)
text_btn.click(text_to_voice, [text_in, state], [state, audio_out, transcript]).then(
update_chat, inputs=state, outputs=chat
)
reset_btn.click(clear_history, None, [chat, state])
export_btn.click(export_chat, state, gr.File(label="Download chat.txt"))
demo.launch(debug=False)We are building a clean Gradio UI that allows us to talk or type and hear the agent's response. We display our phone calls, and maintain a discussion status, and the Chatbot, text, and audio player, is all presented in one colob app.
In conclusion, we see how the suspension of the pipeline can cause us to create an agent conducted by the listening word, thinking, and responding. Now we have a working shortfall that pulls the audio, it produces clever answers, and returns the expression of the talk, everything is inside Colob. Through this basis, we can try large models, add multiple-language support, or extend the system with custom logic. Nevertheless, the basic idea remains like: We can bring together Asr, LLM, and TTS to one smooth transaction of the active Ai Ai.
Look Full codes here. Feel free to look our GITHUB page for tutorials, codes and letters of writing. Also, feel free to follow it Sane and don't forget to join ours 100K + ml subreddit Then sign up for Our newspaper.
Asphazzaq is a Markteach Media Inc. According to a View Business and Developer, Asifi is committed to integrating a good social intelligence. His latest attempt is launched by the launch of the chemistrylife plan for an intelligence, MarktechPost, a devastating intimate practice of a machine learning and deep learning issues that are clearly and easily understood. The platform is adhering to more than two million moon visits, indicating its popularity between the audience.
🔥[Recommended Read] NVIDIA AI Open-Spaces Vipe (Video Video Engine): A Powerful and Powerful Tool to Enter the 3D Reference for 3D for Spatial Ai
