I keep a single layer 30 degree bag ready to go. I figure if it gets colder than that then I'm going with some sort of shelter. For me an extra layer would just be extra weight. But the coldest it gets here is single digits and that's very rare.
Keep in mind that you can use a pile of dry leaves for extra insulation in a pinch (in and completely around your bag), but be careful with the camp fire.