The concept still works with almost no users. If you’re the only one watching a video, the original host is more than capable of serving it to you and so no need for extra peers ;).

Being able to handle a small amount of users isn’t the problem, it’s once a lot of users suddenly join in that the system would collapse without the peer functionality.

Also, peer tube is not fully decentralized. All videos need to be stored on an instance (similar to torrent seedboxes) so there will always be at least one direct source available. So I don’t think that standalone app as you describe is needed. (That would be interesting though in a fully decentralized model, without any instances but all videos just floating between peers. But that would have the danger of creating dead videos similar to dead torrents)