James Callender was a reporter who was once in the employ of Jefferson. It was Callender who first published the name "Sally" and told the world that she was Jefferson's concubine and bore his children. How did Callender get on the wrong side of Jefferson?
While Jefferson was John Adams' Vice President and rival for the presidency, he secretly hired Callender to attack Adams. Callender wrote that Adams was a “hideous hermaphroditical character, which has neither the force and firmness of a man, nor the gentleness and sensibility of a woman." Now that's the way to destroy your opponent. It worked. John Adams was a one-term president and Jefferson succeded him.
After the election, Callender and Jefferson's genial relationship soured, and Callender decided to retaliate. Thus the "dusky Sally" stories began and never stopped. While Jefferson never publicly responded to the accusations, plenty of visitors to Monticello confirmed that the house slaves looked very like their master.