As long as the vest is in the boat and free to access (not clipped to the cross-brace in the canoe), I have not had a problem.
I do know a group of long islanders in the camp site next to use got popped one year, but they were also standing in the canoe and funneling for a while.