This article drew me in because Jefferson died horribly in debt and was not well known for his ability to keep a personal budget, but this is about how he helped to reduce the American deficit while in office.
Here is some information on his work to reduce it by a third:
Jefferson understood that debt was necessary to pay for war and to invest in the public good, but he believed that “neither the representatives of a nation, nor the whole nation itself, assembled can validly engage debts beyond what they may pay in their own time....” That was a generation, according to Jefferson, and his debt reduction plan, devised by his Secretary of Treasury Albert Gallatin, was to eliminate the debt he inherited in sixteen years.
It was a formidable task. When Jefferson came into the presidency in 1800, the country was $83 million in debt, most of it from the Revolutionary War. He started his attack by doing what most Republican deficit-hawks today suggest: he went after the federal civilian bureaucracy. “We are hunting out and abolishing multitudes of useless offices,” he proudly wrote his son-in-law, “striking off jobs, lopping them down silently.”
The problem was that the civilian government was more muscle than lard, including only 130 employees. Gallatin explained to Jefferson that while cutting civilian jobs saved thousands of dollars, they could save hundreds of thousands more if they followed federal expenditures, which mostly went to the military.