When Federal troops arrived in Paducah, their headquarters were directly across from the Woolfolk home. Woolfolk was pro-Southern and flew a Confederate flag in response, sparking a riot in December 1861 that included Union soldiers, particularly those of the 11th Indiana Regiment. The incident would begin Brigadier General Charles Ferguson Smith's decline as he saw his subordinate, Ulysses S. Grant, raised above him almost immediately.

Woolfolk was banished from Paducah and the United States to Canada on August 1, 1864 by Union Brigadier General Eleazer A. Paine. Two weeks later his wife and family were also banished to Canada, which resulted in eight of Woolfolk's household, four others from Paducah, and eleven from Columbus, Kentucky following Woolfolk into exile. These acts infuriated the Kentucky governor, and Paine was removed from command in Paducah after only 90 days.


“TROUBLE AT PADUCAH
New York Times
Published: December 8, 1861

An unfortunate collision occurred between Gens. SMITH and WALLACE, of the National froces at Paducah, KY., a few days since, the circumstance's of which are thus stated in a Cairo letter of Nov. 25, in the St. Louis Democrat:
"On Tuesday last, a Secessionist in Paducah, by the name of WOOLFOLK, hung a secession flag out of his window as some of our troops were passing by, and hurrahed for JEFF. DAVIS. The man had done the same thing before on several occasions, and the matter was reported to Brig.-Gen. C.F. SMITH, but he refused to interfere. This refusal of Gen. SMITH caused great indignation among the troops, and doubts of his loyalty were freely expressed in Paducah.

The matter having been reported to Gen. WALLACE, he sent his Aid-de-Camp with a squad of men, to order the traitorous flag to be taken in, and, if WOOLFOLK refused, then to take it in, and erect the Stars and Stripes over his house. WOOLFOLK, knowing that Gen. SMITH was senior officer, refused to obey Gen. WALLACE's order, whereupon WALLACE's aid forcibly took down the rebel flag, and hoisted the Stars and Stripes in its stead.

In the meantime, WOOLFOLK having appealed to Gen. SMITH, the latter sent his aid, Lieut. PRICE, to order Gen. WALLACE to have the Stars and Stripes taken down from WOOLFOLK's house. WALLACE refused to obey the order, and sent word to SMITH that the flag should not be taken down while there was a live man in his brigade. WALLACE's aid said that WOOLFOLK should sleep under a loyal flag one night, anyhow. SMITH's aid replied that he did not consider that any great honor. Whereupon WALLACE's aid knocked down SMITH's aid. Gen. PAINE sent WALLACE assurances of his cooperation.

As Gen. SMITH had nobody but his discomfited Lieutenant to enforce his order "the old flag still waves."

The next day Gen. SMITH issued a general order complaining of this breach of military discipline, remonstrating against like occurrences in the future, and reminding the troops that they were sent into Kentucky as protectors of a loyal State, and that moderation and forbearance should be exercised toward unarmed enemies, and that they should ever show themselves the champions of law and order.

Another account says that Gen. WALLACE went with his whole regiment to tear down the obnoxious flag, and that he made a congratulatory speech to his soldiers after the affair was over. Also, that a Missouri regiment started with a fire-engine to wash out Gen. SMITH's headquarters, which they were dissuaded from doing by Gen. WALLACE. There was much feeling against Gen. SMITH."