On May 18, 1863, as the Confederate rear guard fell back into the Vicksburg defenses, soldiers were ordered to burn all the houses in front of their works. The Shirley barns and outbuildings were quickly burned to the ground, but the soldier assigned to destroy the house was shot before he could apply the torch.
Mrs Shirley, her 15-year-old son Quincy, and several servants, were caught in the cross-fire as Union soldiers approached Vicksburg. Fearing for their lives, they remained in the house huddled in a chimney corner for three days before Mrs Shirley tied a sheet to a broom handle and had it placed on the upper front porch. The frightened occupants of the "white house" were finally removed by Union soldiers and given shelter in a cave.
When the siege ended, the Shirley house was badly damaged and abandoned. The house and sixty acres were sold to the United States government in 1900 by the couple's daughter Alice who insisted her parents be buried behind the house.
Information from signage and National Park Service