Margaret Taylor is the only First Lady we have no extant picture of today. If you see a picture of Margaret Taylor, chances are it is actually Betty Bliss, her daughter. Margaret Taylor did not act as the White House hostess and so little was known about her, although she actually was a well-educated Southern lady.
From the White House site:
After the election of 1848, a passenger on a Mississippi riverboat struck up a conversation with easy-mannered Gen. Zachary Taylor, not knowing his identity. The passenger remarked that he didn't think the general qualified for the Presidency--was the stranger "a Taylor man"? "Not much of one," came the reply. The general went on to say that he hadn't voted for Taylor, partly because his wife was opposed to sending "Old Zack" to Washington, "where she would be obliged to go with him!" It was a truthful answer.
Moreover, the story goes that Margaret Taylor had taken a vow during the Mexican War: If her husband returned safely, she would never go into society again. In fact she never did, though prepared for it by genteel upbringing.
"Peggy" Smith was born in Calvert County, Maryland, daughter of Ann Mackall and Walter Smith, a major in the Revolutionary War according to family tradition. In 1809, visiting a sister in Kentucky, she met young Lieutenant Taylor. They were married the following June, and for a while the young wife stayed on the farm given them as a wedding present by Zachary's father. She bore her first baby there, but cheerfully followed her husband from one remote garrison to another along the western frontier of civilization. An admiring civilian official cited her as one of the "delicate females...reared in tenderness" who had to educate "worthy and most interesting" children at a fort in Indian country.
Two small girls died in 1820 of what Taylor called "a violent bilious fever," which left their mother's health impaired; three girls and a boy grew up. Knowing the hardships of a military wife, Taylor opposed his daughters' marrying career soldiers--but each eventually married into the Army.
The second daughter, Knox, married Lt. Jefferson Davis in gentle defiance of her parents. In a loving letter home, she imagined her mother skimming milk in the cellar or going out to feed the chickens. Within three months of her wedding, Knox died of malaria. Taylor was not reconciled to Davis until they fought together in Mexico; in Washington the second Mrs. Davis became a good friend of Mrs. Taylor's, often calling on her at the White House.
Though Peggy Taylor welcomed friends and kinfolk in her upstairs sitting room, presided at the family table, met special groups at her husband's side, and worshiped regularly at St. John's Episcopal Church, she took no part in formal social functions. She relegated all the duties of official hostess to her youngest daughter, Mary Elizabeth, then 25 and recent bride of Lt. Col. William W.S. Bliss, adjutant and secretary to the President. Betty Bliss filled her role admirably. One observer thought that her manner blended "the artlessness of a rustic belle and the grace of a duchess."