Slave labor was used to build many of the original buildings in DC, including the White House (which shouldn't be surprising given that many of our early leaders were slave owners and DC is so close to many slave states):
....the President's House, as the White House was first known, was constructed with significant help from slave labor, as well as free blacks and whites. Slaves lived in huts amid a cacophony of brick kilns and sawing operations, probably on the site of what is now Lafayette Park. One slave, George, was owned by James Claggett and leased to the federal government for five months, according to a pay stub recently put on display by the National Archives. The document, in elegant script, says that "the commissioners of the Federal District" paid Claggett "for hire of Negro George," for "working at the President's House."
The construction of the President's House began in 1792, with slaves often toiling "seven days a week during the high construction summer months alongside white workers and artisans," according to a history compiled by the White House Historical Association. An estimated 120 slaves helped dig the foundation of the White House and brought stonework to the site. Some of the stonework can still be seen in the exterior of the original, central portion of the building.
I also posted awhile ago an excerpt from A Colored Man's Reminisences of James Madison by Paul Jennings (who was a former slave) if you are interested as well.