The White House historical society has a memoir by Paul Jennings, a former slave, on it's site so you can read this rare account of James Madison by an African American. The page also includes a scholarly article on the memoir. I pulled out the part where Paul Jennings talks about each of the Madisons in turn:
Mrs. Madison was a remarkably fine woman. She was beloved by every body in Washington, white and colored. Whenever soldiers marched by, during the war, she always sent out and invited them in to wine and refreshments, giving them liberally of the best in the house. Madeira wine was better in those days than now, and more freely drank. In the last days of her life, before Congress purchased her husband’s papers, she was in a state of absolute poverty, and I think sometimes suffered for the necessaries of life. While I was a servant for Mr. Webster, he often sent me to her with a market-basket full of provisions, and told me whenever I saw anything in the house I thought she was in need of, to take it to her. I often did this, and occasionally gave her small sums from my own pocket, though I had years before bought my freedom of her.
Mr. Madison, I think, was one of the best men that ever lived. I never saw him in a passion, and never knew him to strike a slave, although he had over one hundred; neither would he allow an overseer to do it. Whenever any slaves were reported to him as stealing or "cutting up" badly, he would send for then and admonish them privately, and never mortify them by doing it before others. They generally served him very faithfully. He was temperate in his habits. I don’t think he drank a quart of brandy in his whole life. He ate light breakfasts and no suppers, but rather a hearty dinner, with which he took invariably but one glass of wine. When he had hard drinkers at his table, who had put away his choice Madeira pretty freely, in response to their numerous toasts, he would just touch the glass to his lips, or dilute it with water, as they pushed about for the decanters. For the last fifteen years of his life he drank no wine at all.
You can go enjoy this entire piece and accompanying commentary.