The power usage isn’t even that much on an individual basis once it’s trained, it’s that they have to build these massive data centers to train and serve millions of users.

I’m not sure it’s much worse than if nvidia had millions of customers using their game streaming service running on 4080s or 4090s for hours on end vs less than an hour of AI compute a day.

It’d be better if we could all just run these things locally and distribute the power and cooling needs, but the hardware is still to expensive.

You have apple with their shared GPU memory starting to give people enough graphics memory to load larger more useful models for inference, in a few more generations with better memory bandwidth and improvements, the need for these data centers for consumer inference can hopefully go down. These are low power as well.

They don’t use CUDA though so aren’t great at training, inference only.