Levi was married three times. During his marriage to Martha Angeline Cooper, he and his family lived in Choctaw County, Mississippi. The Spencer-Cooper union produced three children. The first child died in infancy, the second child died when it was two or three years old. The third child, Sarah Angeline Spencer, survived.

After Martha's death, Levi returned to Lowndes County, Alabama. He had lived in the county with his parents as a youth. In 1861 he moved from Lowndes county to Jasper County, Mississippi and settle three and a quarter miles north of Paulding and some seventeen miles from Enterprise. He lived there for over 30 years.

He served in Company "B", 5th Regiment, Mississippi Infantry, CSA, during the Civil War. During the war, grain crops were grown on his farm. Some of the harvest was distributed to the government and needy neighbors.

About 1884, Levi joined the Phalti Baptist Church bear Paulding and became an ordained deacon shortly after his baptism. His pastor said that Levi died from an injury suffered from a fall caused by vertigo.