Feature Tracker and FAQ

[0xShug0]

Use this thread to:

Request new features
Pick up open features TODOs
Ask or find answers to common questions

PRs for these features are greatly appreciated! If you want to work on a TODO, please comment first to avoid duplicated effort.

Features

• Support mp3 and other inputs/output (PR ready)
• HTTPS support (PR ready)
• Name based backend selection
• Other server metrics
• Multi-GPU support for pooling VRAM Multi-GPU support for pooling VRAM (e.g. 2× GTX 1070) like llama.cpp? #505. May need a careful model-by-model rollout.
• Better support for Intel GPU and Adreno GPU (Vulkan: maxComputeWorkGroupCount error (Intel GPU & Adreno GPU) #489)

UI:

• Adding a History of music generations (currently any new generation removes the previous one in the right panel.)
• Possibility to attach metadata (prompt, lyrics etc) to the audio file, or
• Exporting the metadata as *.json file, full for reproducibility.
• In HeartMuLa having a default 'pop' in tags would serve better than throwing error: tags needed.
• In ACE-step having a default for prompt 'upbeat ...' would also serve better than throwing error: prompt needed.
• Any new run stops the play, it would be better if it leaves play alone and only stops it if the new generation done and the new song ready.
• The newly added -1 = auto label could move next to Duration seconds similar to existing Seed -1 = random.

FAQ

Docker can't run

This is usually a CUDA driver compatibility issue. See https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-toolkit-and-corresponding-driver-versions

Vulkan

• Useful build tips (from @Dev0Lab1) : Tips for Vulkan Build #475
• Use shared VRAM on Windows (from @Dev0Lab1): set "GGML_VK_ALLOW_SYSMEM_FALLBACK=1" HeartMuLa 3B: failed to allocate HeartCodec scalar decoder graph -- Can't use Windows shared VRAM. Workaround: set "GGML_VK_ALLOW_SYSMEM_FALLBACK=1" #434

Nvidia Jetson Orin

• Check this thread: Nvidia Jetson Orin related questions #12

Placeholder...

[blox11]

https://github.com/woongzip1/UniverSR
https://github.com/jusperlee/apollo

Are these two that you would consider implementing? They are more like audio restoration tools.
Also, I appreciate your work on audio.cpp. Juggling a bunch of different music gen/TTS models between their own implementation or comfy-ui was annoying and difficult to get working, but this has been a very easy experience.

I have some other recommendations too (no MuScriptor Large?), but thought it might be a bit rude to just dump a wall of them without any context 😂

[gitchat1]

Is there any chance we could get a whisper check like it is implemented in this project. https://github.com/zeropointnine/tts-audiobook-tool That would improve quality significently and since audio.cpp runs on both AMD as well as Nvidia it would make the comunity less dependent on Nvidia. It would also be pretty important to be able to offload the whisper model to the cpu for those of us with lower end hardware.

[0xShug0]

@blox11 Thanks for the pointer! I’ve added them to the model candidates list. For variants like MuScriptor Large, they’re usually lower priority for me, so I’m hoping the community can pick them up. Most of these extensions should be pretty straightforward since the architecture stays the same and only the model parameters change. My focus is still on extending support for unique model families.

@gitchat1 I don’t quite understand how this feature works. Could you elaborate a bit?

[gitchat1]

I'm not a programmer so you would have to look at the code to know how it's really done. What the feature does in practice is it retranscribes generated segments of audio and checks the transcript against the original text. If the segment passes the check it is processed further if it does not the segment is discarded and regenerated. The programme runs through this cycle up to five times if it can't resolve the error it accepts a flawed segment. Of course the Whisper model in question does also need VRAM when run together with the model on the graphics card, therefore it would be great if you could reallocate that model in this case whisper-turbo to the CPU for memory constraint environments. What would make it even better is if after the programme fails five times it didn't just pick the latest generation but the best of them. This programme has such a feature. https://github.com/petermg/Chatterbox-TTS-Extended I do not know how difficult something like that would be to implement though. Oh yes and in the case of audiobooktool there are five possible acuracy levels the segments need to pass in order to be processed further. The higher the acuracy level the closer the segment has to be to the real text in order to pass.

[0xShug0]

What the feature does in practice is it retranscribes generated segments of audio and checks the transcript against the original text. If the segment passes the check it is processed further if it does not the segment is discarded and regenerated.

I see. This kind of feature should live in applications built on top of audio.cpp. The core engine won’t implement features like this.

[blox11]

Two more that would potentially be quite useful, especially for using the new YuE2 model with stem separation/lyrics transcription respectively.

https://github.com/facebookresearch/sam-audio
https://github.com/tencent-ailab/songprep