You'd normally have a home partition somewhere. There are some Linux distros specifically geared for such set ups. Knoppix, I think is one example.
I used to work with a system like this before, where the company's system admin objected to a Linux installation on their machines. It broke the CD drive after sometime of using it continuously as a read-only HD, at which point the admin saw the light...
I mean - yes. But I really want a thin client that has enough to bootstrap wifi+wpa/ssh/a few local tools
Do as much externally as possible, use it for forensics and dealing with weird systems, bring it across borders, and generally use it for strange circumstances. Use external AIDE to make sure nothing ever changes, ever.