Broadly speaking, the clerk is right. A councillor who is not a member of a committee has no rights other than as a citizen.
It can be argued that the restriction on public comments to a period at the start of a meeting is nowadays too constraining. Standing orders can be changed, and a possible change is to allow public comments at any time in a meeting, subject to the chairman's discretion. Clearly the chairman has to exercise sensible control to allow business to be transacted effectively, but usually public contributions are helpful and give people more involvement in council affairs.
Many councils have been in the practice of adjourning the meeting or "suspending" standing orders if they wish to allow non-members to speak. But this leaves a situation where there are no rules. Having standing orders that remain in force but allow discretion to the chairman avoids that issue.
This approach may be impractical for a very large council, but for many smaller councils it is perfectly feasible.
A possibly related matter is that under typical standing orders, a council can appoint new members to a committee or dissolve a committee without prior notice and without needing an agenda item.