The couple allowed to attend a White House dinner they weren't invited to failed to show up for a hearing, Thursday, to discuss why they were at the event.
Secret Service director Mark Sullivan testified before the House Homeland Security Committee, acknowledging mistakes were made that allowed Tareq and Michaele Salahi into the White House state dinner.
Three Secret Service agents are now on administrative leave because of the mistake.
"This is our fault and our fault alone," Sullivan said. "There's no other people to blame for this. Look at me, and blame me. It's our fault."
White House Social Secretary Desiree Rogers also declined to appear at the hearing.
The committee is threatening to subpoena them.
Meanwhile, the White House has urged staff to take more precaution.
A staff memo said, "The White House did not do everything we could have done to assist the United States Secret Service in ensuring that only invited guests enter the complex."