It does sound as if you're up against it on this issue, without support from other councillors. It is lawful for a council to make an allowance to the chairman and it can be fixed as a lump sum regardless of actual expenditure. The authorisation cannot be assumed to roll over year after year, although if other councillors are prepared to support it, then forcing the council to vote on the matter would not alter the outcome. There could be a tax liability, but the chairman is not an employee, so any tax is the responsibility of the chairman, not the council.
Ten or fifteen years ago it was quite common for councils to have this kind of arrangement. It was often used for things like a civic dinner, with chairs of other councils in the area invited. My impression is that, in these straitened times, attitudes have changed. The council of which I am a member has moved away from providing a lump sum, and simply authorises the chairman to be reimbursed for expenses incurred on official duties. There hasn't been a civic dinner for some time.
Although you may not make much immediate progress, it does sound an issue that could be part of a "value for money" election platform when council elections come round!