Testing AI and LLM on Rockchip RK3588 utilizing Mixtile Blade 3 SBC with 32GB RAM

That is nice information for my very own Rockchip chipset exploration, which nonetheless has a methods to go–there now appears to be working Mali GPU acceleration for LLMs, and having extra individuals doing this type of testing on ARM is each informative and an indication there’s curiosity within the small mannequin, edge AI eventualities I’ve been toying with.

Serendipitously, I did have a look at llm-rk3588 after I bought my Orange Pi 5+ (it was really developed on one) however discarded it as a result of the NPU can’t actually be used for LLMs and the required firmware blob didn’t load underneath my Armbian construct (I assume the repo proprietor was utilizing the Orange Pi linux distro).

I would like that be baked into ollama to have a baseline that’s comparable with Apple Silicon and Intel chips–i.e., very same mannequin weights, a minimum of–however its nice to see one thing might be made to work with the Mali GPU (though I’m probably not clear on what mannequin layers will profit, the way it offers with quantization, and so forth.).

However I’ll take a look at reproducing (and, ideally, constructing domestically and tweaking) what was used right here and attempt to give each CPU and GPU inference a go on an analogous (however extra compact) 32GB RAM board that I’ve sitting on my in tray.

So watch this house (I’ve been busy with lots of work and different, extra bodily testing…)