The answer that you don't have a choice seems likely to be right, at least technically. Although since you can't be compelled to attend, it seems pointless for a council to insist on appointing unwilling councillors to a committee. Too many non-attending members will prevent the committee even being quorate, let alone effective. So I guess if you want to press the issue, then make it clear when the committee is being set up that you have no interest and will not attend if appointed. Provided you are contributing in other areas, I don't think anyone should criticise you for that.