Is the model trainable?
Lora, Lokrs, etc
Important, important thing for making the model relevant on actual use.
Thankss
Until they release the Encoder, it's not.
yes: https://docs.simpletuner.io/quickstart/MINIMAX_MUSIC/ better answer: not really in a useful manner just yet. don't waste your compute budgets on this; try ACE-Step or HeartMuLa instead.
LyCORIS and PEFT LoRA trainable via SimpleTuner, with lyrics + caption validation libraries too. though that's not easily configured via the webui just yet (the validation library), the example prompts and captions from their website are included as a built in validation prompt library if you're doing eg. a broad finetune and want to validate a number of separate concepts.
it works with distillation too, so, the next project i'm on with MM Music is to throw AnyFlow at it and bring it down to a decent 4-step generation pipeline in the flow transformer.
what we're limited by is the lack of RVQ encoder's semantic audio tokens from raw audio. i think we could reverse engineer one though if so inclined, i'm not sure whether i'd be going against the license to do so.
Nice.. Its something.
Would it make any sense to opensource a model to then not share a fundamental piece? A model that is not trainable/conditionable is pretty much a toy.
The RVQ encoder is required for properly fine-tuning the model. Without it, you can maybe adjust some tone and pitches, though you would be doing so 100% without proper conditioning. It is like training an image model but instead of captions, you are just typing pure nonsense random text in the encoder for the conditioning. I mean sure, you CAN train it that way. But you will not get good results. The RVQ encoder is absolutely needed.
that's not true, look at how the pipeline actually works. the depth decoder gives the model its tokens from text. the encoder just does it from raw audio.
the inference path mirrors training. you're not required to supply RVQ encoder output (audio references) at inference time. why would you think it'd be mandatory at training?
our training target for the flow matching objective is the VAE audio latents, not the RVQ encoder output.
we wouldn't even be able to run the model for inference if the encoder were required for training.
@bghira You are attempting to train on non aligned tokens. During inference, they are aligned because they were trained to be that way, with the RVQ tokenizer, something you will be breaking with your "training" method.. You are just swapping out a non aligned target and breaking the alignment and saying you trained it. It is like training an LLM without a tokenizer. You have your output tokens, but you are feeding complete garbage on the input.
Think of it this way. RVQ hidden state token are like a MIDI track. A super high detail, frame by frame, MIDI track that contains lyrics for each time step, drums, the beat, when the guitar strums are, to the millisecond, when the chorus starts, everything. It is play by play, perfectly time aligned sheet music and detailed composition for each frame of the song. Without the RVQ tokenizer, you are generating this precise midi track, and throwing a completely different song at it where none of the timing aligns , the lyrics are completely different, this one does not even have drums, the "E note piano key" is now a "F note flute". It is in a different key, pitch, everything is 100% misaligned. And you are teaching the model,"make this sound like that" You are just completely breaking down the model doing this. 100%. And maybe, just maybe, before the model is completely broken, it may sound a little bit like the 100% misaligned song you fed into it. But that does not mean you trained it, you just broke it pretty.
i don't get where you're getting any of that from. sorry, it's just not the case.
- RVQ encoder gives target-audio-derived discrete tokens.
- RVQ depth decoder gives text-derived residual-code hidden states.
- the transformer consumes condition-encoder outputs, not raw RVQ encoder states.
- the model supports text-to-audio inference, which is conditioning generated from text via the feedback loop with the LM and depth decoder, not from the RVQ encoder
- text-to-audio LoRAs train against the same conditioning used at inference
- RVQ encoder is ONLY needed for target-token supervision, reference-audio conditioning
we're not training on "garbage text", but sure, if you have really bad captions and lyrics inputs that don't align at all with the latents the audio latent encoder creates, you'd get bad results.
have you actually tried training it, or just repeating yourself hoping that people believe it?
No point arguing with arrogance.
agreed
I don't understand enough of the tech details to get those intricacies but Ostris and Bghira arguing on my question thread is an honor I guess.. 😁
agreed
@bghira you've implemented training modules for it on SimpleTuner, do you have any examples we can test? Have you produced any LoRA/LyCORIS/finetune with your method that we can try? I can provide a small dataset with some relatively obscure genres/styles, and evaluate the results if you want.
That's the easiest way to find out whether it works or not without the RVQ encoder.
yes, but it was using internal datasets and i'm contractually barred from sharing those results. i've put together a public dataset after identifying that Jamaican patois / reggaeton is really badly generated by the model (its' actually kind of funny how bad it is), you can check the data i'm using under RareConcepts/suno-reggae-test-dataset
edit: and the test LoRA using internal data wasn't different enough from what the model could already produce, it doesn't work when trying something the model doesn't already know
Isn't their whole documentation about the LMs (both Global and Local) giving structure that the DiT then turns into decodable latents?
Think ACE-Step 1.5, i actually wrote Side-Step (which to be fair, compared to SimpleTuner and ai-toolkit, not even God himself knows), they do use an LM as well for "planning" and pre-conditioning their generations. On ACE-Step 1.5 it is not mandatory to use, and text encoding is provided by another Qwen model separate from the causal LM providing their so-called audio codes, but Minimax Music 3 apparently uses its hidden states which are already text-conditioned, which for training is a gut punch.
You said "the inference path mirrors training... we wouldn't even be able to run the model for inference if the encoder were required for training."
But at inference, condition and latent are aligned automatically, no? The DiT generates the latent from the condition, so the alignment is just given, there's no external target it has to align to.
At training there is an external target, and something has to supply that alignment between the condition and the target. MiniMax supplied it with their RVQ tokenizer/encoder, which is basically what we're asking for here.
I don't think @ostris or anyone else is saying that the RVQ output was the target during training. It's what makes the conditioning actually mean the target in the first place. /shrug
All that said, I'll gladly be educated by anyone on this thread. if misaligned rollout conditioning really is sufficient for adapter training and/or If the LMs are an optional path as it is on ACE-Step 1.5 during inference, it is a win for everyone training adapters, arguably even people just generating without them.
In any case @bghira i was personally also in the side-quest of reverse engineering the tokenizer, but for a solo dev this is high risk, no reward, not to mention im broke. If you guys ever decide to go that path, i will do everything on my power to help out, i figured what is the most time-consuming is just having a sheer volume of generations to save intermediary artifacts; that alone (given how slow a generation is to, well, generate) on consumer hardware at batch 1 already talked me out of doing it solo.
so does this work or do i have to keep banging my head with the RVQ encoder. i guess we need it for real reference, but training? and i agree, imma give it 9 hours to finish this run and then done if it doesn't work lol
why didn't they release the encoder? would it reveal they trained on copyrighted music? isn't there a good enough open RVQ encoder that could be trained a little with an adapter to work with this?
why didn't they release the encoder? would it reveal they trained on copyrighted music? isn't there a good enough open RVQ encoder that could be trained a little with an adapter to work with this?
most people just don't because they don't want to be liable for cloning and stuff.
Isn't their whole documentation about the LMs (both Global and Local) giving structure that the DiT then turns into decodable latents?
...
At training there is an external target, and something has to supply that alignment between the condition and the target. MiniMax supplied it with their RVQ tokenizer/encoder, which is basically what we're asking for here.
firstly, the LM is indeed generating structured states from the text. because the encoder and decoder were trained together. the depth decoder's knowledge on how to structure lyrics embeds comes from being co-trained against the encoders' outputs from raw audio latents. it's trained to match the text to audio. like how CLIP is trained on image-text pairs, the text embeds end up containing a tiny squished image.
secondly, it's the DiT that's generating the alignment between the conditioning and target. they're not "automatically aligned" based on just text input unless the model has actually seen only text inputs at training time as well.
here's the first LoRA I can share, it's 4000 steps but i was too conservative with learning rate.
https://huggingface.co/bghira/minimax-music-suno-reggae-rank128
i'm putting together a 2nd round, and since i've got the embeds cached it won't take as long. i'm not super happy with the dataset, but it's clearly shifting the distribution toward the inputs.
where that LoRA will load and inference other than SimpleTuner's codebase i'm not sure, it'll likely require changes to ComfyUI or even Diffusers since there's no PEFT LoRA loader mixin for it over there yet.
it's indeed a real struggle to train this model, but also it was pointed out to me that using too-short of a track duration results in broken outputs / poor adherence.
still, as everyone who was doubting me was hoping would be the case, it's likely i'm wrong about the RVQ encoder being required. but the model isn't degrading like i'd expect for a mismatched objective. in any case, i'm sorry if i gave anyone unwarranted hope that we could train it. i hoped it would be a repeat of flux.1 [dev] "distilled models can't be finetuned" myth.
a few thoughts;
- if it does work, it's probably still not worth the cost. the precaching of LM outputs for 299 tracks consumed 35gb of disk space. yikes.
- if we don't precache to save disk space, you're adding the time to run the LM to each step. for a batch size of 8, it's taken 33 seconds.
- it's possible that providing the audio tokens so the LM is conditioned on the real track timings would speed it up a little? but we don't have any code examples to even know whether that's true.
- if it doesn't work, congratulations to minimax music team for achieving what no image/video model creator has done yet, produce an untrainable artifact that is frozen forever with no further liability to the creators. even flux.1 [schnell] was finetunable given enough effort and compute burnt.
for more context, it took 8 hours to pre-cache the 299 embeds on H200 node and i reuploaded them to huggingface for reuse so that others don't have to spend much to try and experiment with the model.
it's been about $500 in GPU time spent on the experiments, so, even if we discover we can't train shit into the weights of the flow transformer without the RVQ encoder's raw audio handling, well, at least it was just me who burnt that budget.
running the 4k step lora at 0.25 strength in ComfyUI does seem to work. 🤷🏽♂️
i dont know anymore lol
as everyone who was doubting me was hoping would be the case, it's likely i'm wrong about the RVQ encoder being required.
@bghira I don't think anyone was hoping that you were wrong. We all want the model to be trainable, and it is with the RVQ tokenizer. We were just trying to save you that $500 by trying to explain why it is not trainable without it.
thanks for trying, i appreciate the attempts to explain - my questions earlier weren't rhetorical either, i genuinely was looking for more answers and information to help me understand.
the $500 of GPU time is contracted, i didn't spend anything that wouldn't have been spent anyway. i did learn things along the way - this architecture i really didn't know anything about.
i'm moving on to deploying a cluster of H100s to generate data from the Music 3 model that we can use for creating a compatible encoder. i know we cannot recover the original one, but maybe we can build something suitable anyway.
if anyone would like to pitch in on the efforts, the simpletuner discord server is here and i can make the hidden-states/codebook extraction code public too.
samples would be real handy, i had a oracle harness churning but i mean, i hit the point we all knew was going to happen, mass compute or tying up local for 20 hours at a time lol
sweet, samples samples samples, the code isn't hard. now we can all get on it and try stuff
i'm adding a bit of varying temperature and CFG as iRedsneth on Discord suggested, and will update the previous shards to include the fixed values they used as well as new shards including the dynamic values.
here's a little set of wav outputs with codes in case you feel like doing reruns on different stuff. if someone is dipping their toes in, it'll get you to the mass training part lol.
https://huggingface.co/marduk191/Minmax_music3_experiments/tree/main/corpus