Semantic Audio Generative Encoder · ×64 compression

Audio compressed into
meaningful latents.

SAGE is a SwinV2 variational autoencoder that compresses music ×64 into a compact latent that is both faithful to reconstruct and semantically structured: it matches or surpasses SOTA autoencoders up to 9× larger, runs 2-10× faster, and leads all 15 semantic probes.

Listen to samples Start the test
SCROLL TO DISCOVER⌄

A/B comparison

Hear the difference

Same clip, every model. Switch instantly, playback never stops.

Benchmarks

Objective benchmarks

SAGE against state-of-the-art audio VAEs: reconstruction fidelity and how well its latent space supports downstream music understanding.

Axis 1 · Reconstruction fidelity

Keep the signal

Compress stereo 44.1 kHz music at ×64 and rebuild it with state-of-the-art perceptual quality. Whatever is lost here caps every downstream result.

Axis 2 · Semantic structure

Organize the signal

Distances in the code must reflect genre, artist, instrumentation: that geometry is what makes the latent navigable for diffusion and useful as a feature space.

Model & efficiency

Capacity, training data and inference cost. SAGE is the lightest to run here: fastest per file and lowest real-time factor.

Reconstruction & perceptual metrics

Reconstruction fidelity and perceptual quality against state-of-the-art audio VAEs, measured on two datasets.

Semantic probing metrics

How well the frozen latent space supports downstream semantic tasks (MAEB), measured on two datasets.

At a glance

The two axes, one picture

Axis 1 · Reconstruction fidelity

Best fidelity at the lowest runtime

Fidelity vs speed on FMA - bubble area ∝ parameters. SAGE sits in the winning corner: best FAD at the lowest RTF, 47.7 ms per 10 s clip.

0.005 0.01 0.02 0.05 RTF (log) · ← faster 0.20 0.25 0.30 0.35 0.40 FAD-MERT ↓ · better fidelity SAME SAO CoDiCodec Music2Latent SAGE
Axis 2 · Semantic structure

Stems cluster by timbre

UMAP of frozen SAGE latents on MoisesDB isolated stems: no instrument label ever reaches the encoder, yet the code organizes by timbre.

UMAP projection of frozen SAGE latents for isolated MoisesDB stems: points cluster by instrument
bass drums percussion guitar piano other keys vocals

Under the hood

Commit the latent, then polish the decoder

Phase 1 · pre-training
❄ CLAP e 512-d target φ pool + 64→512 ê ℒsem 1 − cos(ê, e) audio E z D reconstruction ℒrec + ℒGAN ❄ frozen loss dataflow

Everything trains: reconstruction, KL, adversarial loss, and CLAP distillation shaping the latent. The encoder and the latent geometry are committed here.

Phase 2 · decoder fine-tune
audio ❄ E ❄ z D post-net zero-init residual reconstruction ℒrec + ℒGAN ❄ frozen loss dataflow

Encoder frozen, latent pinned. A residual post-net polishes perceptual detail: the code downstream models consume never moves.

Subjective evaluation

Take the listening test

Your ears are the final benchmark: a blind MUSHRA test (ITU-R BS.1534-3), 10 trials, about 12 minutes, SAGE against the state of the art.

Start the test