I had a brother-in-law that didn't "believe in" full coverage car insurance, because he said he'd pay more to the insurance company than he'd ever get back, even if the car were totaled. He also only bought newer used cars (never new) and paid cash rather than financing them.
Financially all his advice was good, but my brother-in-law had money -- if he needed a new car, he could just go buy one. At the time, I was not in the same financial position.
The reason for that story is to say that unless $50,000 is no big deal to you, I can't image not having insurance on it. Of course I'm financially conservative and believe in "better safe than sorry." But since the insurance cost is based on the home's value, plus you can pare the poilcy down to the bare necessities and keep a high deductible, the cost should be pretty reasonable.