There's a reason why bears and other beasts hole up in caves: They're ready-made shelters that provide immediate protection from rain, snow, wind, or brutal sun. No need to work at erecting a hut—just move in and set up housekeeping.
Watch Out for Water
Make sure the cavern is high enough to be out of danger from flash floods, incoming tides, and storm surges.
Put Up a Fence
Erect a low stone wall across the opening to help keep dirt from blowing around.
Start a Fire
The stone walls make good reflectors for the campfire, and there's no worry of the fire spreading to nearby vegetation and getting out of control. To keep campfire smoke from becoming a problem, build the fire near the cave entrance.
Beat the Draft
Because they are made of rock, caves generally retain the cold. They're good places to escape the heat of a hot desert, but not so desirable in the dead of winter. Unless you can get a good fire going or partition a section of the cavern into a small room, cold air will always surround you.
Stay Safe in a Cave
Like any neighborhood, a cave has its advantages and disadvantages. Before you can move in, you need to make sure it isn't already occupied. Depending on where you are, caves might be favored dwellings for venomous snakes, bats, wolves, bears, cougars—you get the picture. Even if the only cave dwellers are rats, mice, or squirrels, you might become ill from contact with hantavirus in their urine or feces, so look for a clean cave floor. One other caution: Caves exist because the ceiling "caved" in from some type of erosion. If you see evidence of instability overhead or fresh rockfalls on the floor, or if there's water flowing through the cave, it's probably not structurally sound. I'd move on.