Photo for Author | Kanele
Obvious Introduction
Open-Source Ai is responsible for an important moment. In development with large language models, a regular machine learning, and now technologies quickly reduced the gap in relation. One of the most exciting entries in this space is Microsoft's Open-Source Voice Stack, Vinevuines. This model family is designed for a natural, bright, illustration, causing the quality of higher commerce contribution.
In this article, we will check the Viveroice, Download model, and run to Google Colab using GPU Runtime. Additionally, we will talk about problems solving regular problems that may arise while using model.
Obvious Introduction of Vineviice
Bivice The next generation framework – to-talk (TTS) to create Expressive, a long form, a multi ceremonial such as podcasts and discussions. Unlike traditional TTS, it exceeds SCalability, speaker to synchronize, and to take a natural turn.
The basic establishment is found in continuously continuous topenizers working at 7.5 hz, paired in a large language model (QWEN2.5-1.5b) and the head of the Diffenusion. The project allows 90 minutes to talk about 4 different speakers, the front systems pass.
Vineviines is available as a model of open source Kisses faceand Code stored in a society of easy attempts and use.
Pictures from VineviiceObvious Starting with vinice-1.5B
In this guide, we will learn how we can combine the vinevioice repository and use the shortfare by providing us with text file to produce a natural speaker of a variety. It takes only 5 minutes from the setup to produce the sound.
// 1. Clone the Community Repository & Enter
First, describe the Vinecice Reportary Transportation (VineCoice-Community / Vineviice), enter the required Python, and enter HUB face wreck The download library using Python API.
Note: Before starting Colob time, make sure your type of operation is set on T4 GPU.
!git clone -q --depth 1 /content/VibeVoice
%pip install -q -e /content/VibeVoice
%pip install -q -U huggingface_hub// 2. Download Model Summary from the Food Summary
Download the last model using Hugging Face Snapshot API. This will download all files from the microsoft/VibeVoice-1.5B the warehouse.
from huggingface_hub import snapshot_download
snapshot_download(
"microsoft/VibeVoice-1.5B",
local_dir="/content/models/VibeVoice-1.5B",
local_dir_use_symlinks=False
)// 3. Create text on speakers
We will create a text file to Google Colab. In that, we will use the magic work %%writefile to provide content. Below is a sample conversation between two speakers about Kdnuggets.
%%writefile /content/my_transcript.txt
Speaker 1: Have you read the latest article on KDnuggets?
Speaker 2: Yes, it's one of the best resources for data science and AI.
Speaker 1: I like how KDnuggets always keeps up with the latest trends.
Speaker 2: Absolutely, it's a go-to platform for anyone in the AI community.// 4
Now, we will run the Demo Text in the VineViones Reportory. The script requires a model method, text file, and speaker words.
Run # 1: Map Map 1 → Alice, Madamalo 2 → Fram
!python /content/VibeVoice/demo/inference_from_file.py
--model_path /content/models/VibeVoice-1.5B
--txt_path /content/my_transcript.txt
--speaker_names Alice FrankAs a result, you will see the next result. The model will use a Cuda to produce sound, Frank and Alice as two speakers. Will also give a summary to use analysis.
Using device: cuda
Found 9 voice files in /content/VibeVoice/demo/voices
Available voices: en-Alice_woman, en-Carter_man, en-Frank_man, en-Mary_woman_bgm, en-Maya_woman, in-Samuel_man, zh-Anchen_man_bgm, zh-Bowen_man, zh-Xinran_woman
Reading script from: /content/my_transcript.txt
Found 4 speaker segments:
1. Speaker 1
Text preview: Speaker 1: Have you read the latest article on KDnuggets?...
2. Speaker 2
Text preview: Speaker 2: Yes, it's one of the best resources for data science and AI....
3. Speaker 1
Text preview: Speaker 1: I like how KDnuggets always keeps up with the latest trends....
4. Speaker 2
Text preview: Speaker 2: Absolutely, it's a go-to platform for anyone in the AI community....
Speaker mapping:
Speaker 2 -> Frank
Speaker 1 -> Alice
Speaker 1 ('Alice') -> Voice: en-Alice_woman.wav
Speaker 2 ('Frank') -> Voice: en-Frank_man.wav
Loading processor & model from /content/models/VibeVoice-1.5B
==================================================
GENERATION SUMMARY
==================================================
Input file: /content/my_transcript.txt
Output file: ./outputs/my_transcript_generated.wav
Speaker names: ['Alice', 'Frank']
Number of unique speakers: 2
Number of segments: 4
Prefilling tokens: 368
Generated tokens: 118
Total tokens: 486
Generation time: 28.27 seconds
Audio duration: 15.47 seconds
RTF (Real Time Factor): 1.83x
==================================================Play audio in the registry:
Now we will use the Ippython work to listen to the sound produced within Cocaw.
from IPython.display import Audio, display
out_path = "/content/outputs/my_transcript_generated.wav"
display(Audio(out_path))
It took 28 seconds to produce sound, and it sounds clear, natural, and smooth. I love it!
Also try with different voice players.
Run # 2: Try different words (Mary with Speaker 1, Carther Speaker 2)
!python /content/VibeVoice/demo/inference_from_file.py
--model_path /content/models/VibeVoice-1.5B
--txt_path /content/my_transcript.txt
--speaker_names Mary CarterAudio produced was the best, with the background music at the beginning and the smooth transformation between the speakers.
Found 9 voice files in /content/VibeVoice/demo/voices
Available voices: en-Alice_woman, en-Carter_man, en-Frank_man, en-Mary_woman_bgm, en-Maya_woman, in-Samuel_man, zh-Anchen_man_bgm, zh-Bowen_man, zh-Xinran_woman
Reading script from: /content/my_transcript.txt
Found 4 speaker segments:
1. Speaker 1
Text preview: Speaker 1: Have you read the latest article on KDnuggets?...
2. Speaker 2
Text preview: Speaker 2: Yes, it's one of the best resources for data science and AI....
3. Speaker 1
Text preview: Speaker 1: I like how KDnuggets always keeps up with the latest trends....
4. Speaker 2
Text preview: Speaker 2: Absolutely, it's a go-to platform for anyone in the AI community....
Speaker mapping:
Speaker 2 -> Carter
Speaker 1 -> Mary
Speaker 1 ('Mary') -> Voice: en-Mary_woman_bgm.wav
Speaker 2 ('Carter') -> Voice: en-Carter_man.wav
Loading processor & model from /content/models/VibeVoice-1.5BTip: If you are not sure which words are available, printed prints “Voices Available:” At first.
Normal includes:
en-Alice_woman, en-Carter_man, en-Frank_man, en-Mary_woman_bgm, en-Maya_woman, in-Samuel_man, zh-Anchen_man_bgm, zh-Bowen_man, zh-Xinran_womanObvious Troubleshooting
// 1. NOPO doesn't have demo documents?
Microsoft Vinevoice Repository is deducted and reset. Social reports indicate that certain code and demons are removed or not available in the first place. If you find that official official has lost rest examples, please check the Social Example or Record Examinatory Design and instructions:
// 2. A Fast Generation or Cuffle Mistakes in Colob
Make sure you are up to date of GPU: Workout time
// 3. Cuda Oom (Out of Memory)
To reduce load, you can take several steps. Start by reducing the installation text and reduces the length of generation. Consider reducing the average sample and / or modifying chunk size if the text allows. Set batch size in 1 and select a unique little model.
// 4. No Failing Failing or Lost
The script is usually prints the last way out of console; Scroll until you find the exact location
find /content -name "*generated.wav"// 5. Word of voice words are not available?
Copy the exact words listed under the vocal vocal. Use Alias words (Alice, Frank, Mary, Carter) was indicated in the demo. Correspond to .wav goods.
Obvious The last thoughts
For many projects, I would choose an open-line stack as a vinevoice with the apis paid for several compulsory reasons. First and very important, it is easy to integrate and give a customary transaction, making it easy to get many applications. Additionally, he amazingly lightens the needs of the GPU, which can benefit from the oppressed resources.
Vineviice is an open source, which means that in the future, you can expect better structures that empower the immediate generation and even CPUS.
Abid Awa (@ 1abidaswan) is a certified scientist for a scientist who likes the machine reading models. Currently, focus on the creation of the content and writing technical blogs in a machine learning and data scientific technology. Avid holds a Master degree in technical management and Bachelor degree in Telecommunication Engineering. His viewpoint builds AI product uses a Graph Neural network for students who strive to be ill.
