I think this may be another of those questions where the devil is in the detail. All meetings of the council or any committee of the council must be open to the public and have an agenda which specifies the business to be transacted which is issued no later than three clear days before the meeting. There is absolutely no necessity for the clerk to be in attendance but surely that's what the clerk is paid for? However, if someone else is able to take the minutes then I'd suggest there are no issues. Assuming you're using your own premises for the meeting, I presume venue availability is no issue either although if you're not, I can see that village halls etc. don't always have availability for the day/date/time you might have resolved to have a meeting.
I do have a question about your "informal" meetings. Frankly, if you're having a chat at the pub, clearly that's not a meeting but if you're having meetings to discuss items ultimately coming to council for agreement, there is an argument that you are making decisions outside of the formal processes for doing so. Working groups or task and finish groups might be set up to discuss items as part of a project for example, and their meetings can be informal, do not need a clerk or anyone else, but it is important that these have terms of reference to set out the parameters and that they do not make decisions which are binding on the council.
As I said, lots of ifs, buts and maybe to answer your question. More information might help refine any response