Retired Lt. Gen. William "Jerry" Boykin has decided not to speak at a West Point prayer breakfast after a veterans' advocacy group asked the Army chief of staff to rescind Boykin's invitation.

VoteVets.com officials complained that allowing Boykin to speak at the U.S. military academy would be contrary to Army values and disrespectful to Muslim cadets.

Boykin has issued frequent warnings about the threat of radical Islam and Sharia law, a position the left-wing political action committee views as "Islamophobic."

"My views on Islam have been grossly misrepresented," Boykin told CBN News.  "I do fear Sharia. I do believe we need to stop the encroachment of Sharia."

"But I have also said consistently that there are many Muslims in America that reject Sharia as well, and those are the people we need to be reaching out to," he added.

Click play for Boykin's comments on the decision, and criticism he's received for his views on radical Islam.

Meanwhile, the Council on American-Islamic Relations welcomed Boykin's decision not to appear at the event. 

CAIR National Executive Director Nihad Awad said he hoped Boykin's replacement would "offer cadets a spiritual message that promotes tolerance and mutual understanding,"