This is a popular misconception, which crops up often on this board, as a result, I believe, of poor advice from NALC. In fact, as you suspect, it is the Parish council which 'holds' the information and decides who needs to view it and not the clerk. This will be evidenced by the fact that it is the Parish Council, and not the clerk, which is registered as data controller with the ICO.
Therefore, if the Parish Council wishes the allotment working group to have the names and addresses, the clerk has no authority to refuse. Nor do you need consent from those on the waiting list, as you have a legitimate interest in using their information to contact them.
If you receive further challenge I suggest that you call the ICO's helpline, or use the webchat which offers you the facility to have a copy of the advice sent to you at the end of the chat.