In "The Minister's Black Veil," why do the villagers bury Mr. Hooper without removing the veil?

When the Reverend Mr.
Clark sits by Father Hooper's deathbed, Mr. Clark asks Father Hooper to allow him to remove the
veil from Father Hooper's face so that he can meet eternity without it.  Father Hooper, however,
is horrified by this idea, and he shrieks, "'Never! [...].  On earth, never!'"  It is
clear that Father Hooper wishes never to have the veil removed, and this may be one reason that
his parishioners do not remove it.

Further, everyone gathered at Father
Hooper's deathbed still seems to fear the mysterious veil.  Mr. Clark even suggests that it
signifies some terrible sin Father Hooper had committed.  However, Father Hooper
asks, 

"Why do you tremble at me alone? [....]
Tremble also at each other!  Have men avoided me, and women shown no pity, and children screamed
and fled, only for my black veil?  What, but the mystery which it obscurely typifies, has made
this piece of crape so awful?  When the friend shows his inmost heart to his friend; the lover
to his best beloved; when man does not vainly shrink from the eye of his Creator, loathsomely
treasuring up the secret of his sin; then deem me a monster, for the symbol beneath which I have
lived, and die!  I look around me, and, lo! on every visage a Black Veil!"

It seems to be, in part, this "mystery" that Father
Hooper describes that so terrifies the people.  They seem always to have had an obscure idea of
what the veil symbolizes -- that each of us has secret sins that we attempt to hide from each
other, ourselves, and even God -- but no one wants to admit that they might understand because
that would be tantamount to admitting that one has these secret sins on one's soul, and what
everyone wants the most is to hide this very fact.  If, in truth, these people really do have
even a vague understanding of the veil's meaning, then they would not want to remove it
after Father Hooper has died because it is a meaningful and accurate symbol; if, on the other
hand, they really don't have a concept of the veil's meaning, then they would not want to remove
it because it is such a mystery, and we fear mysteries.