Rendered at 07:58:18 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
boustrophedon 9 hours ago [-]
I might have missed it in the readme, but what hardware did the training take place on? This is really neat.
sheepseb 9 hours ago [-]
For the world model training I've used an RTX 4060, but for the agent playing the game that was directly on the pico using stream-ac
dannyw 11 hours ago [-]
This is incredibly cool! Fascinating to see looped transformers be necessary for mini-Doom, but not for simpler games.
andai 11 hours ago [-]
Very cool. I didn't know NNs were capable of such determinism, that's very interesting. Is this a similar effect to overfitting? Also, it's the same model for all the games, right?
svantana 10 hours ago [-]
> Also, it's the same model for all the games, right
It's a bit unclear but it looks like different models, since they all have different parameter and MAC counts. It would be much more impressive with a single world model that could maybe dream up new games inbetween the games in the training set.
sheepseb 10 hours ago [-]
It's a self contained net for each game but the data used for training doesn't overlap with the validation. I think it would be easy to merge them into a single model
xpct 7 hours ago [-]
Is the actor trained on world model's mistakes as well? If so, it should learn to force hallucinations that force it into an easier state.
sheepseb 20 minutes ago [-]
Yes, there were some hallucinations, e.g: in breakout hitting a corner would destroy more blocks than expected, in doom-lite it could shoot from miles away and get a kill. The fixed was add some noise to the input frame and keep the target clean.
11 hours ago [-]
cchance 4 hours ago [-]
If this runs on a PI... can it run on webgpu ?
sheepseb 31 minutes ago [-]
The core of it uses rust with no_std and on the pico we use embassy-rs with no os. At the same time, the core can be build for webassembly but it doesn't use webgpu.
jbarberu 3 hours ago [-]
Unlikely, Raspberry Pi Pico 2 W is a microcontroller, not a computer. It runs either bare metal or an RTOS.
nater5000 11 hours ago [-]
Honestly fascinating. I almost feel like you'd be better off avoiding any discussion of the fact that this runs on a Raspberry Pi Pico 2 W because it takes away from the crazy world model capabilities (of course, the fact that this does run on a Pico 2 makes the whole thing much more impressive).
There's gotta be a good way to incorporate this logic into a larger, more "traditional" game in a meaningful way. Seems like an approach for procedural generation that could solve some of the issues usually found with actual procedural generation.
It's a bit unclear but it looks like different models, since they all have different parameter and MAC counts. It would be much more impressive with a single world model that could maybe dream up new games inbetween the games in the training set.
There's gotta be a good way to incorporate this logic into a larger, more "traditional" game in a meaningful way. Seems like an approach for procedural generation that could solve some of the issues usually found with actual procedural generation.