A local AI music generator that is trainable!

[BluGenes]

Even cooler news, well for me, lol. I'll now be able to test better and such. I decided I really like this AI stuff and already use it daily to write code, have been for a long time. So, I invested in new hardware, specifically set up to run AI. Why not, ya know. Anyways, all it is, is a i7 with a NPU and a GTX 5080.

This means I can now run RC with Ollama!! I tested this so far using Ollama and Google's Gemma4:e12b model, which is a large model. The entire model fits on the GPU!

But, in the mean time, I did have an issue arise with Suno and my account. It is now a humorous situation, in a way, but has not been resolved.

Basically, because of technicality with Yahoo, and the fact that I authorized Suno via Google, I lost an email account used for that. So, all they need to do is move my stuff to a new account, which I wholly proved I own. Anyways, the issue is Suno only uses AI for support. So, it has to assign the problem to a "team", but, the users that need support is in the millions, so, there is a very long wait for resolution. From what I have heard, things do eventually get resolved. (about 7 million songs a produced daily!)

So, in the spirit of all that, I wrote a song about it. Maybe the Suno people will hear it. lol
This video was generated with Codex, same app I program with..

[BluGenes]

New machine really is amazing. So, yes, you will need good hardware to do anything worth while

I did the graphics for this on it:

[BluGenes]

One more for now

On this one, I used the new GPT 5.6 models, actually all 3, Sol, Terra and Luna. Sol did the heavy work and Terra and Luna did the rendering. What I did was launch ComfyUI with the Image2Video LTX model 2.3 loaded and while it was running, GPT generated 14 images from the lyrics as scenes, then each scene was rendered as video in ComfyUI. The new GPT Desk Top app can now generate images! I used Codex to run the rendering and Codex auto switched the models from Luna to Terra and back a couple of times. That was interesting..

This one took two prompts, one for the initial run, and then the second to clean up some problems. For example, on the first try, a couple of scenes needed improvement, but, instead of generating an animation, it generated stills. So, I had to re-run it with corrections.

I mean, it worked, but, I dunno, kind of lame really. I suppose it's kid a friendly video.

But, here it is:

[starflakeprj]

I don't mean to be rude, but there are still quite a few inconsistencies in the video

Cool though, is the music still from ACE Step?

Edit: I see you are mentioning Suno in at least one of your more recent posts.

[BluGenes]

no rudeness there nor implied.. it helps..

Yea I know.. why I declared it kind of lame.. I could improve it by teaching the AI the BPM the song uses. For example, if loaded into DaVinci Resolve, there is a detect beats and it draws the beats on the audio (something some DAWS do and some don't too). Giving the AI a screen shot of that would tell the AI how to calculate the motions better. I don't have a way for it to analyze audio other than that. I mean yea, I could write Python for it, but dunno..

The big thing I am taking away from this run, is the prompt wasn't overly complicated.

I am now able to run ACE Step, and it was really bad in the output. I would say it's not a total loss, it probably could be useful, like maybe by converting stems to midi with another app, but, given it has a lot of artifacts in the audio, even that would be questionable. Like Suno, it's just a tool.

But yea, all of those songs so far are from the account I am locked out of. I still can get to the songs via my phone by downloading them and emailing them back or use iTunes.

I watched a release video by OpenAI on the new "work" mode, as a demo they used a similar dataset of support issues. It was huge. So, I have that to look forward to, a long wait. But, watching it work was impressive what they did.

Maybe soon, I'll re-visit this song and make it beat friendly.. It does kill the mood of it..

thanks for the feed back and I guess the take away from this it still needs manual editing!

[BluGenes]

Well, helps to actually research stuff. Found some cool stuff and it isn't that hard to use with ComfyUI. Using the same model I used to generate the animations, it can do lip syncing, for example. I am really interested in syncing to the music, not just lip syncing, but, dancing and just anything that could drive an image.

So, this is going to be a really powerful setup, once I get everything in place. The power comes from using Codex too.

[BluGenes]

So, took a while for this, but, finally got a better video! I am still exploring this as a technology and just found a ComfyUI workflow called LTX Director. It's pretty much designed to do videos with multiple scenes. It's a custom setup and I had to use a custom node which wasn't part of the install version of director. I used a new "relay prompt" thing a ma bob. I had Codex do all the work too.

I also installed the Chrome browser extension for ChatGPT/Codex which allows Codex to run ComfyUI from the browser. You have to first save the work flow as an API and then Codex can do it's magic. lol Even though Codex could run the desktop version, this one works a lot better, I think.

In this video, granted there are a couple discrepancies, but, ahh well. For example, the lyrics say the cat has no collar, but, in the video it does. Some of that cats will morph and disappear, but, I see it as a pleasingly aesthetic. After all, it's magical. lol. What surprised me the most, was a couple of the scenes, the cat is singing in lip sync!

Also, as with another video I did, there is an easter egg in it.. hehe.

so, let me know what ya think.. it is far from perfect, but, ya know, going in the right direction.

[BluGenes]

And just a quick update, the video took about 6 hours to render. The way it works is, it generates an image as a start frame, if there needs to have an end frame, then it generates an image for that, for example the dance scenes needs those.

Each scene is about 5 seconds each, with one about 7 seconds. It generated 40 mp4 files, then quality checked each one. If one failed, it regenerated it. After I approved, it generated the video. All this was automated with Codex, ComfyUI, and Chrome (browser). Image generation was done using ChatGPT's Image Gen 2 from inside Codex.

I have Codex on my phone, so, I could watch it work while out and about too.