"I didn't say there was a racial component. I was very careful not to say that," Holder told The Huffington Post on Friday, when asked about his comments before the Rev. Al Sharpton’s National Action Network.
In his speech last Wednesday, Holder said that he and President Barack Obama have faced "unwarranted, ugly and divisive adversity."
"What attorney general has ever had to deal with that kind of treatment? What president has ever had to deal with that kind of treatment?" Holder asked during his National Action Network speech.
Although much of the media coverage of Holder's comments interpreted them as a reference to racial divisions, he told HuffPost that he was referring to a lack of civility in Washington.
"I think what we have seen is kind of a breakdown in civility in Washington, D.C., and that becomes important because I think it has substantive impact," Holder said. "We are celebrating the 50th anniversary passage of the 1964 Civil Rights Act. If we had a Congress or an executive branch-legislative branch relationship in the way that we now have one, where there's this lack of civility, I wonder whether or not you could have forged the necessary compromises, things that involved personal relationships, in order to get such a landmark piece of legislation passed.
"And that's essentially what I was decrying, the fact that we can't somehow separate whatever our personal feelings are and focus on our functions as members of the executive branch or as legislators. I think that I've done a pretty good job in doing that, but it's frustrating at times," Holder said.