AniMUL

Experimental interspecies communication model.

Status: Under development.

This repo is quite messy right now, as I figure out all the dependencies, optimize the scripts, etc. So expect some borkenness for now.

Install

Thusly, suit to taste.

Instead of just a nice clean pyproject.toml, the deps are a bit "custom" since Qwen3-Omni is so new and some upstream bits don't support it yet. Plus there is a lot of dependency conflicts with latest/greatest versions that need to be hammered out... And the NatureLM dataset isn't natively supported, so some small patches were worked up for that.

Requires Python 3.11.

mkdir -p ~/devel/deepcrayon
cd ~/devel/deepcrayon
git clone https://spacecruft.org/deepcrayon/AniMUL
cd AniMUL/
python -m venv .venv
source .venv/bin/activate
pip install -U setuptools pip wheel

More Install...

Cruft for installing latest versions to actually work with current Qwen3-Omni models...

sudo apt install ninja-build
sudo apt install cudnn9-cuda-12-9 libcudnn9-headers-cuda-12

# Find TORCH_CUDA_ARCH_LIST thusly:
nvidia-smi --query-gpu=name,compute_cap --format=csv

# A6000
export TORCH_CUDA_ARCH_LIST="8.6"
# RTX A6000 Pro / RTX 6000 Ada (96GB Ada Lovelace)
#export TORCH_CUDA_ARCH_LIST="8.9"
# Actually, the NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0 is:
export TORCH_CUDA_ARCH_LIST="12.0"
# H100 / H200
export TORCH_CUDA_ARCH_LIST="9.0"
# B200
export TORCH_CUDA_ARCH_LIST="10.0"

#pyenv local 3.11
pip install torch==2.7.1 torchvision --index-url https://download.pytorch.org/whl/cu128

# Bunch of repos that need latest version. Instead of installing
# with pip install git+https, just download them because they
# may need to be re-installed a bunch of times to get everything going.
cd ~/devel/deepcrayon
git clone --recursive https://spacecruft.org/deepcrayon/ms-swift
cd ~/devel/deepcrayon/AniMUL
pip install -e ~/devel/deepcrayon/ms-swift

pip uninstall transformer-engine transformer_engine_cu12 transformer_engine_torch -y
mkdir -p ~/devel/NVIDIA/
cd ~/devel/NVIDIA/
git clone https://github.com/NVIDIA/TransformerEngine.git
cd TransformerEngine/
git checkout v2.7
git submodule update --init --recursive
export NVTE_FRAMEWORK=pytorch
export CUDA_HOME=/usr/local/cuda-12.9
export PATH=/usr/local/cuda/bin:$PATH
export NVTE_FRAMEWORK=pytorch
export TORCH_CUDA_ARCH_LIST="10.0"
export PIP_NO_BUILD_ISOLATION=1
rm -rf build/ dist/ *.egg-info
pip install ninja cmake pybind11
pip uninstall transformer_engine
pip install -v . --no-deps --no-build-isolation
pip install onnxscript==0.3.1
cd ~/devel/deepcrayon/AniMUL

mkdir -p ~/devel/huggingface/
cd ~/devel/huggingface/
git clone --recursive https://github.com/huggingface/transformers
cd transformers/
git checkout v4.57.3
cd ~/devel/deepcrayon/AniMUL
pip install -e ~/devel/huggingface/transformers/

pip install accelerate tensorboard-plugin-profile qwen-omni-utils decord einops packaging ninja pybind11 "fsspec[http]<=2025.3.0,>=2023.1.0" "numpy<2.0.0"

pip uninstall flash-attn -y
MAX_JOBS=48 pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.8.1 --no-deps --no-build-isolation

mkdir -p ~/devel/NVIDIA/
cd ~/devel/NVIDIA/
git clone --recursive https://github.com/NVIDIA/apex
cd ~/devel/NVIDIA/apex/
# Perhaps large cruft: In setup.py comment out this check:
# if bare_metal_version != torch_binary_version
python setup.py install --cpp_ext --cuda_ext
cd ~/devel/deepcrayon/AniMUL

pip install -e .
pip install -e .[dev]

Download

Get the Qwen3-Omni model:

animul-dl-qwen-model

Download the 17 terabyte dataset:

# animul-dl-esp-audio-data
# or
hf download --repo-type dataset --local-dir /srv/hf/datasets/EarthSpeciesProject/NatureLM-audio-training/ EarthSpeciesProject/NatureLM-audio-training

Model conversion

Takes less than 10 minutes to run on 2x A6000 GPUs. Takes less than 4 minutes to run on 8x B200 GPUs.

./scripts/qwen3-to-mcore
# Copy over json:
cp -p /srv/hf/Qwen/Qwen3-Omni-30B-A3B-Instruct/*.json /srv/devel/deepcrayon/AniMUL/model
cp -p /srv/hf/Qwen/Qwen3-Omni-30B-A3B-Instruct/merges.txt /srv/devel/deepcrayon/AniMUL/model

Test run

This will run in under and hour or so on 2x A6000 GPUs:

./scripts/moe.sh

LoRA

Test with for a LoRA with ESP dataset on 2x A6000:

./scripts/moe-a6000.sh

Full Model

Needs to be run on B200 or other large GPUs.

To process just 1% of the dataset. Takes ~14 hours on 8x B200s.

./scripts/moe-b200-1percent.sh

Full dataset:

./scripts/moe-b200-fp8-production-next.sh

Convert

When done, convert model (needs ~120G RAM, 78G VRAM):

./scripts/convert_to_hf.sh

Test

Extract a sample to test:

python scripts/extract_dataset_audio_answer.py

Example:

Question: What is the taxonomic name of the focal species in the audio?
Expected Answer: Chordata Aves Passeriformes Passerellidae Atlapetes fuscoolivaceus
Audio saved to: test_audio.wav

Create a test dataset with one sample:

./scripts/create_test_dataset.py

Run a single test:

./scripts/test_tuned_model.sh

Compare it with Qwen3-Omni results:

./scripts/test_qwen3-omni.sh

Create a test dataset with 100 samples:

./scripts/create_test_dataset_100.py

Run test for 100 samples with new model and Qwen3-Omni:

./scripts/test_tuned_model-100.sh
./scripts/test_qwen3-omni-100.sh

Then compare the results between the correct answer, the new model, and original Qwen3-Omni:

./scripts/compare_results.py

Upstream

This project is unofficial and not related to the upstream projects.

Earth Species Project

• https://www.earthspecies.org/
• https://huggingface.co/datasets/EarthSpeciesProject/NatureLM-audio-training
• https://github.com/earthspecies

Qwen

• https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct
• https://github.com/QwenLM/Qwen3-Omni

License

Apache 2.0.

Copyright © 2025 Jeff Moe.