Despite being under Confederate Army authority, the people of eastern Tennessee, where the Johnsons lived, were largely loyal to the Union. With Senator Johnson speaking vigorously against the Confederacy and seeking Union protection of his region, Eliza Johnson became a target. Without warning, her Greeneville home was confiscated for use as sleeping quarters for Confederate Army troops. Forced from there, Eliza Johnson and her young son Frank and adult son Charles, had to seek shelter at the nearby Carter County home of her daughter and son-in-law, Mary and Daniel Stover, and three young children. The Stover home, however, was also located in the area controlled by the Confederate government.
In April of 1862, Eliza Johnson, along with other prominent Union families in that jurisdiction were given short notice to vacate by Confederate General Kirby Smith, who oversaw it. In one of her only remaining letters, Eliza Johnson responded formally but honestly that “in my present state of health, I know I can not undergo the fatigues of such a journey; my health is quite feeble, a greater portion of the time being unable to leave my bed.” Five months later, she wrote him again, this time declaring herself able to travel and requesting the necessary permits for movement within the Confederate-held regions and to cross into Union territory when necessary.
Starting in mid-September 1862, the privations endured by Eliza Johnson essentially made her a wartime refugee. For several nights, she and her daughter Mary Stover also prepared and smuggled food into nearby mountain caves where her son-in-law and his fellow Union military sought shelter and eluded detection by Confederates. In late September, she was detained for two days in Murfreesboro by Confederate General, Nathan B. Forrest which proved to be a degrading and harrowing episode. Having had no warning that the family would be detained in Murfreesboro, Eliza Johnson was literally forced to go door to door to seek to the homes of strangers and beg for shelter that night for herself and her family. Only begrudgingly was one home of Confederate sympathizers made available to them but denied the next night. On the second night, Eliza Johnson and her family were able to find shelter only in an abandoned restaurant, with no place to sleep, no food for sustenance, and no light. Eliza Johnson had apparently considered such a possibility, for she had brought candles from home and kept sandwich remnants from the previous day, which she gave her grandchildren to eat. Once permission from the Confederate capital in Richmond was wired to officials in Murfreesboro, Eliza Johnson and her family proceeded by train to Nashville, during which they were violently harassed and her sons threatened with death by fellow passengers who were Confederate sympathizers.
Although she and her family were given safe refuge in Nashville, arriving there on 13 October 1862, Eliza Johnson was soon notified that the alcoholism of her adult son Robert Johnson had deteriorated his condition and threatened his Union Army appointment as a Colonel. Stationed with his military unit in Cincinnati, Ohio, determination of his case was delayed, due to the status of his father as a U.S. Senator. With Andrew Johnson seeking to coordinate matter from Washington, Eliza Johnson and her family members left Nashville for Cincinnati in November of 1862, to personally intercede on Robert Johnson’s behalf. From there, with her son Frank, Mary Stover, and her three Stover grandchildren, Eliza Johnson sought a health treatment at a sulfuric spa in Vevay, Indiana. Joined there by her son-in-law Daniel Stover in early 1863, the party proceeded to Louisville, Kentucky. The anxious months and exposure to the elements had worsened Eliza Johnson’s breathing problems and she decided to then proceed to Nashville in May, rather than unite with her husband in Washington, where the weather would further deteriorate her condition.
Appointed by President Lincoln as Military Governor of Tennessee (1862-1865), Johnson made several dramatic references in public speeches to the treatment of his wife by the Confederate Army. This prompted deeper resentment of him and increased threats against his life. Andrew and Eliza Johnson had an emotional but brief reunion in Nashville when she arrived there with her family in May of 1863. He separated from them again weeks later, removing himself to Kentucky’s Union territory for his safety. Ongoing threats against him and their renewed separation nonetheless perpetuated anxiety for Eliza Johnson.
This was intensified in April of 1863 by the sudden death of their son Charles, killed instantly when his skull fractured after being thrown from a horse. Having moved successfully from publishing a small-Union newspaper to earning a medical degree and then being appointed a Union Army surgeon with the rank of Colonel, 33 year old Charles Johnson had become an especial point of pride for Eliza Johnson and his unexpected death was one from which she was said to have never emotionally recovered, forever sensitive even to the mention of his name. The one consolation during this period was that daughter Martha Patterson, her husband and two children rejoined Eliza Johnson and lived in the same home with her.
In early June 1864, Andrew Johnson was nominated as the National Union Party’s vice presidential candidate, on the ticket with President Lincoln who was seeking re-election. Eliza Johnson played no role in his campaign, a fact which stemmed not from disinterest but her role in handling a family crisis at the time. Robert Johnson’s alcoholism had so worsened that he was forced to resign as a Union Colonel. In August of 1864, Eliza Johnson brought him to the Lewis Sanitarium in Lexington, Massachusetts for recovery treatment, and simultaneous treatment for her tuberculosis and the first signs of it in her younger son Frank. After making the arduous wartime journey from Nashville to Boston, they first rested at a resort, “Pigeon Cove,” outside the city. Another tragedy soon hit the family when Mary Stover was widowed by the sudden death of her husband in December of 1864.Eliza Johnson remained in Nashville, rather than attend the Washington, D.C. swearing-in ceremony of Andrew Johnson as Vice President in March 1865. A month later, upon learning that Lincoln had been killed and of the conspiracy to kill members of his Administration, her daughter wrote to her father that, “Poor mother, she is almost deranged fearing that you will be assassinated.