- Perl 55.8%
- Python 43.1%
- Makefile 0.6%
- C++ 0.5%
| docs | ||
| nemo_msdd_configs | ||
| src | ||
| .gitignore | ||
| CHANGELOG.txt | ||
| LICENSE-apache.txt | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
| setup.py | ||
MultiVoice
With audio or video file input MultiVoice does:
- Automatic Speech Recognition (ASR).
- Speech-to-Text (STT).
- Diarization.
- Speaker Recognition (SR).
It outputs .txt and .srt (subtitle) files.
Source Code
Installation
Dependencies
Install these dependencies on Debian Bookworm (12/Stable).
sudo apt install python3-pip python3-venv python-is-python3 ffmpeg
Python
Thusly, suit to taste:
git clone https://spacecruft.org/deepcrayon/multivoice
cd multivoice/
python -m venv venv
source venv/bin/activate
pip install -U setuptools pip wheel
pip install -e .
If it complains about cuDNN version like this:
RuntimeError: cuDNN error: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH
Then run this to just use the system/Nvidia cuDNN:
pip uninstall --yes nvidia-cudnn-cu12
Usage
Main scripts:
- multivoice
- multivoice-dir
- multivoice-identify
- multivoice-register
multivoice
multivoice takes audio file input, mono 16000 Hz, and does automatic
speech recognition, outputting a text file and SRT subtitles file.
If a voice has been registered, it will also do speaker recognition.
$ multivoice --help
usage: multivoice [-h] -a AUDIO [--batch-size BATCH_SIZE] [-D] [--device DEVICE]
[--language {af,am,ar,as,az,ba,be,bg,bn,bo,br,bs,ca,cs,cy,da,de,el,en,es,et,eu,fa,fi,fo,fr,gl,gu,ha,haw,he,hi,hr,ht,hu,hy,id,is,it,ja,jw,ka,kk,km,kn,ko,la,lb,ln,lo,lt,lv,mg,mi,mk,ml,mn,mr,ms,mt,my,ne,nl,nn,no,oc,pa,pl,ps,pt,ro,ru,sa,sd,si,sk,sl,sn,so,sq,sr,su,sv,sw,ta,te,tg,th,tk,tl,tr,tt,uk,ur,uz,vi,yi,yo,yue,zh,Afrikaans,Albanian,Amharic,Arabic,Armenian,Assamese,Azerbaijani,Bashkir,Basque,Belarusian,Bengali,Bosnian,Breton,Bulgarian,Burmese,Cantonese,Castilian,Catalan,Chinese,Croatian,Czech,Danish,Dutch,English,Estonian,Faroese,Finnish,Flemish,French,Galician,Georgian,German,Greek,Gujarati,Haitian,Haitian Creole,Hausa,Hawaiian,Hebrew,Hindi,Hungarian,Icelandic,Indonesian,Italian,Japanese,Javanese,Kannada,Kazakh,Khmer,Korean,Lao,Latin,Latvian,Letzeburgesch,Lingala,Lithuanian,Luxembourgish,Macedonian,Malagasy,Malay,Malayalam,Maltese,Maori,Marathi,Moldavian,Moldovan,Mongolian,Myanmar,Nepali,Norwegian,Nynorsk,Occitan,Panjabi,Pashto,Persian,Polish,Portuguese,Punjabi,Pushto,Romanian,Russian,Sanskrit,Serbian,Shona,Sindhi,Sinhala,Sinhalese,Slovak,Slovenian,Somali,Spanish,Sundanese,Swahili,Swedish,Tagalog,Tajik,Tamil,Tatar,Telugu,Thai,Tibetan,Turkish,Turkmen,Ukrainian,Urdu,Uzbek,Valencian,Vietnamese,Welsh,Yiddish,Yoruba}]
[--model MODEL_NAME] [-o OUTPUT_DIR] [--no-stem] [--suppress_numerals] [-v] [-V]
options:
-h, --help show this help message and exit
-a AUDIO, --audio AUDIO
Name of the source audio file
--batch-size BATCH_SIZE
Batch size for batched inference, reduce if you run out of memory, set to 0 for original whisper longform inference (default: 8)
-D, --debug Debugging
--device DEVICE if you have a GPU use 'cuda', otherwise 'cpu' (default: cuda, if available)
--language {af,am,ar,as,az,ba,be,bg,bn,bo,br,bs,ca,cs,cy,da,de,el,en,es,et,eu,fa,fi,fo,fr,gl,gu,ha,haw,he,hi,hr,ht,hu,hy,id,is,it,ja,jw,ka,kk,km,kn,ko,la,lb,ln,lo,lt,lv,mg,mi,mk,ml,mn,mr,ms,mt,my,ne,nl,nn,no,oc,pa,pl,ps,pt,ro,ru,sa,sd,si,sk,sl,sn,so,sq,sr,su,sv,sw,ta,te,tg,th,tk,tl,tr,tt,uk,ur,uz,vi,yi,yo,yue,zh,Afrikaans,Albanian,Amharic,Arabic,Armenian,Assamese,Azerbaijani,Bashkir,Basque,Belarusian,Bengali,Bosnian,Breton,Bulgarian,Burmese,Cantonese,Castilian,Catalan,Chinese,Croatian,Czech,Danish,Dutch,English,Estonian,Faroese,Finnish,Flemish,French,Galician,Georgian,German,Greek,Gujarati,Haitian,Haitian Creole,Hausa,Hawaiian,Hebrew,Hindi,Hungarian,Icelandic,Indonesian,Italian,Japanese,Javanese,Kannada,Kazakh,Khmer,Korean,Lao,Latin,Latvian,Letzeburgesch,Lingala,Lithuanian,Luxembourgish,Macedonian,Malagasy,Malay,Malayalam,Maltese,Maori,Marathi,Moldavian,Moldovan,Mongolian,Myanmar,Nepali,Norwegian,Nynorsk,Occitan,Panjabi,Pashto,Persian,Polish,Portuguese,Punjabi,Pushto,Romanian,Russian,Sanskrit,Serbian,Shona,Sindhi,Sinhala,Sinhalese,Slovak,Slovenian,Somali,Spanish,Sundanese,Swahili,Swedish,Tagalog,Tajik,Tamil,Tatar,Telugu,Thai,Tibetan,Turkish,Turkmen,Ukrainian,Urdu,Uzbek,Valencian,Vietnamese,Welsh,Yiddish,Yoruba}
Language spoken in the audio, specify None to perform language detection (default: None)
--model MODEL_NAME Name of the Whisper model to use (default: large-v3)
-o OUTPUT_DIR, --output OUTPUT_DIR
Output directory for the final .txt and .srt files (default: audio file directory)
--no-stem Disables source separation.This helps with long files that don't contain a lot of music.
--suppress_numerals Suppresses Numerical Digits.This helps the diarization accuracy but converts all digits into written text.
-v, --verbose Increase output verbosity
-V, --version Show version
multivoice-dir
multivoice-dir is the same as multivoice, but recursively processes an
entire directory. Good for processing large jobs.
$ multivoice-dir --help
usage: multivoice-dir [-h] --dir DIR [--batch-size BATCH_SIZE] [-D] [--dest DEST_DIR] [--device DEVICE]
[--language {af,am,ar,as,az,ba,be,bg,bn,bo,br,bs,ca,cs,cy,da,de,el,en,es,et,eu,fa,fi,fo,fr,gl,gu,ha,haw,he,hi,hr,ht,hu,hy,id,is,it,ja,jw,ka,kk,km,kn,ko,la,lb,ln,lo,lt,lv,mg,mi,mk,ml,mn,mr,ms,mt,my,ne,nl,nn,no,oc,pa,pl,ps,pt,ro,ru,sa,sd,si,sk,sl,sn,so,sq,sr,su,sv,sw,ta,te,tg,th,tk,tl,tr,tt,uk,ur,uz,vi,yi,yo,yue,zh,Afrikaans,Albanian,Amharic,Arabic,Armenian,Assamese,Azerbaijani,Bashkir,Basque,Belarusian,Bengali,Bosnian,Breton,Bulgarian,Burmese,Cantonese,Castilian,Catalan,Chinese,Croatian,Czech,Danish,Dutch,English,Estonian,Faroese,Finnish,Flemish,French,Galician,Georgian,German,Greek,Gujarati,Haitian,Haitian Creole,Hausa,Hawaiian,Hebrew,Hindi,Hungarian,Icelandic,Indonesian,Italian,Japanese,Javanese,Kannada,Kazakh,Khmer,Korean,Lao,Latin,Latvian,Letzeburgesch,Lingala,Lithuanian,Luxembourgish,Macedonian,Malagasy,Malay,Malayalam,Maltese,Maori,Marathi,Moldavian,Moldovan,Mongolian,Myanmar,Nepali,Norwegian,Nynorsk,Occitan,Panjabi,Pashto,Persian,Polish,Portuguese,Punjabi,Pushto,Romanian,Russian,Sanskrit,Serbian,Shona,Sindhi,Sinhala,Sinhalese,Slovak,Slovenian,Somali,Spanish,Sundanese,Swahili,Swedish,Tagalog,Tajik,Tamil,Tatar,Telugu,Thai,Tibetan,Turkish,Turkmen,Ukrainian,Urdu,Uzbek,Valencian,Vietnamese,Welsh,Yiddish,Yoruba}]
[--model MODEL_NAME] [--no-stem] [--suppress_numerals] [-v] [-V]
Process audio files in a directory.
options:
-h, --help show this help message and exit
--dir DIR Top-level directory to process.
--batch-size BATCH_SIZE
Batch size for batched inference, reduce if you run out of memory, set to 0 for original whisper longform inference (default: 8)
-D, --debug Debugging
--dest DEST_DIR Destination output directory for the final .txt and .srt files (default: processed_output)
--device DEVICE if you have a GPU use 'cuda', otherwise 'cpu' (default: cuda, if available)
--language {af,am,ar,as,az,ba,be,bg,bn,bo,br,bs,ca,cs,cy,da,de,el,en,es,et,eu,fa,fi,fo,fr,gl,gu,ha,haw,he,hi,hr,ht,hu,hy,id,is,it,ja,jw,ka,kk,km,kn,ko,la,lb,ln,lo,lt,lv,mg,mi,mk,ml,mn,mr,ms,mt,my,ne,nl,nn,no,oc,pa,pl,ps,pt,ro,ru,sa,sd,si,sk,sl,sn,so,sq,sr,su,sv,sw,ta,te,tg,th,tk,tl,tr,tt,uk,ur,uz,vi,yi,yo,yue,zh,Afrikaans,Albanian,Amharic,Arabic,Armenian,Assamese,Azerbaijani,Bashkir,Basque,Belarusian,Bengali,Bosnian,Breton,Bulgarian,Burmese,Cantonese,Castilian,Catalan,Chinese,Croatian,Czech,Danish,Dutch,English,Estonian,Faroese,Finnish,Flemish,French,Galician,Georgian,German,Greek,Gujarati,Haitian,Haitian Creole,Hausa,Hawaiian,Hebrew,Hindi,Hungarian,Icelandic,Indonesian,Italian,Japanese,Javanese,Kannada,Kazakh,Khmer,Korean,Lao,Latin,Latvian,Letzeburgesch,Lingala,Lithuanian,Luxembourgish,Macedonian,Malagasy,Malay,Malayalam,Maltese,Maori,Marathi,Moldavian,Moldovan,Mongolian,Myanmar,Nepali,Norwegian,Nynorsk,Occitan,Panjabi,Pashto,Persian,Polish,Portuguese,Punjabi,Pushto,Romanian,Russian,Sanskrit,Serbian,Shona,Sindhi,Sinhala,Sinhalese,Slovak,Slovenian,Somali,Spanish,Sundanese,Swahili,Swedish,Tagalog,Tajik,Tamil,Tatar,Telugu,Thai,Tibetan,Turkish,Turkmen,Ukrainian,Urdu,Uzbek,Valencian,Vietnamese,Welsh,Yiddish,Yoruba}
Language spoken in the audio, specify None to perform language detection (default: None)
--model MODEL_NAME Name of the Whisper model to use (default: large-v3)
--no-stem Disables source separation. This helps with long files that don't contain a lot of music.
--suppress_numerals Suppresses Numerical Digits. This helps the diarization accuracy but converts all digits into written text.
-v, --verbose Increase output verbosity
-V, --version Show version
multivoice-identify
multivoice-identify does speaker recognition of an audio file, recognizing
the names of the speakers based on data created by multivoice-register.
Note, the project doesn't ship with any voices already recognized.
This is optionally used, if the speaker_vectors.pkl is available.
$ multivoice-identify --help
usage: multivoice-identify [-h] FILE_PATH
Identify a Speaker from a File
positional arguments:
FILE_PATH Path to the audio file for identification
options:
-h, --help show this help message and exit
multivoice-register
multivoice-register takes an audio file input, and the name of a speaker
to create a speaker_vectors.pkl file which can be used for speaker
recognition by the other scripts.
$ multivoice-register --help
usage: multivoice-register [-h] FILE_PATH USER_ID
Register a New Voice/File
positional arguments:
FILE_PATH Path to the audio file for registration
USER_ID User ID for the speaker being registered
options:
-h, --help show this help message and exit
Development
Notes on development.
Python
Thusly, suit to taste:
git clone https://spacecruft.org/deepcrayon/multivoice
cd multivoice/
python -m venv venv
source venv/bin/activate
pip install -U setuptools pip wheel
pip install -e .
pip install -e .[dev]
Formatting
Format with black.
black src/multivoice/*.py src/multivoice/lib/*.py
Linting
Check code with ruff.
ruff check src/multivoice/
Build Wheel
Before building the wheel, build after creating a git tag, unless you want a "dev" build. Then install it locally, for the latest version.
pip install -e .
python -m build
Documentation
This documentation uses Sphinx.
pip install -e .[docs]
make html
rsync -ultav docs/_build/html/ multivoice:/var/www/html/multivoice-org/
Status
- Beta.
- All main features implemented.
- Very inefficient.
- Needs cleanup.
- Needs docs.
Upstreams
-
STT Whisper scripts based on BSD 2-Clause licensed files from https://github.com/MahmoudAshraf97/whisper-diarization
-
CTC forced alignment based on BSD licensed files from https://github.com/MahmoudAshraf97/ctc-forced-aligner
-
By default CTC forced alignment uses CC-BY-NC 4.0 licensed model from https://huggingface.co/MahmoudAshraf/mms-300m-1130-forced-aligner
-
CTC forced alignment uses MIT licensed uroman by Ulf Hermjakob, USC Information Sciences Institute.
-
Demucs based on MIT licensed files from https://github.com/MahmoudAshraf97/demucs which is based on https://github.com/adefossez/demucs which is based on https://github.com/facebookresearch/demucs
-
Identify and Register voice scripts based on MIT licensed file from https://github.com/Mrkomiljon/Speaker-Verification-GUI
-
Deep Multilingual Punctuation Prediction based on MIT licensed https://github.com/oliverguhr/deepmultilingualpunctuation
-
Uses over 200 Python projects from the Python Package Index (PyPI). https://pypi.org/
License
Apache 2.0.
Unofficial project, not related to upstream projects.
Upstream sources under their respective copyrights.
Copyright © 2025 Jeff Moe.