JOEL - First of all you need to read the entire quote carefully:

"Some of the witnesses of the Book of Mormon, who handled the plates and conversed with the angels of God, were afterwards left to doubt and to disbelieve that they had ever seen an angel. One of the Quorum of the Twelve—a young man full of faith and good works, prayed, and the vision of his mind was opened, and the angel of God came and laid the plates before him, and he saw and handled them, and saw the angel, and conversed with him as he would with one of his friends; but after all this, he was left to doubt, and plunged into apostasy, and has continued to contend against this work. There are hundreds in a similar condition." (Journal of Discourses Vol 7 page 164)

This quote is not talking about the three or eight official witnesses; it is talking about other witnesses(hundreds) who had similar visions(or at least thought they did), who later doubted. None of the original eleven witnesses were ever a member of the Quorum of the Twelve Apostles as the quote above refers to.
Also regarding the reliability of things quoted in the Journal of Discourses, please see this page.
Your initial information was correct; none of the original eleven witnesses doubted the expereince they had with the Book of Mormon plates and angel.