The chairman is in a strong position, since they do not have to take any notice of a vote of no confidence. There is no way to remove the council chairman during the council year, except by their resignation. Faced with a confidence motion, the chairman has to decide whether they've had enough and want to get out, or whether they want to tough it out and look for support where they can find it.
If you feel that you should pursue your concerns about councillor behaviour, it would make sense to talk to your local Monitoring Officer (usually the legal officer of the district council). He will be knowledgeable about all the rules relating to councillor behaviour and should be willing to discuss the matter. The Clerk's behaviour can only be governed by the council.