Troutbeck was settled in 1765 by an English family named Benton. Because the hills reminded them of England, and the streams were full of trout, they named it Troutbeck, after their home in the Lake District, Troutbeck, England. The Bentons were literary-minded farmers who established relations with Emerson and Thoreau, wrote books and started the local literary and historical societies.

In the Teens and Twenties, the house leaped into prominence with its second owners, Joel and Amy Spingarn, who had great influence among the literati and liberals of the country. Sinclair Lewis was a frequent guest, as were Hemingway, Lewis Mumford, poets and scholars from around the world, and Theodore Roosevelt when he was President. Mr Spingarn, considered to be one of America's five best minds of the period, was a literary critic, and the first Jewish professor at Columbia University.

He was also an influential liberal who brought the brilliant W.E.B. DuBois, whom he'd known at Harvard, and helped him settle the dispute with followers of Booker T. Washington, and ultimately realize his concept of a unified black movement through the founding of the N.A.A.C.P., the National Association for the Advancement of Colored People. In later years, Col. Spingarn (his rank during W.W. I), became the 2nd president of the N.A.A.C.P. And it was Spingarn who originally spoke the rallying words of a later charismatic African-American leader, "I have a dream...of a unified Negro population."