I believe that any councillor not on the committee is attending in the same way as a member of the public would and therefore could be asked to leave if the meeting is closed to the public.
Closing the meeting to the public is only legitimate on a few grounds and you don't mention what the committee is. If it was a personnel committee, the councillor not on the committee would not be allowed to remain if the Clerk's wages, performance or personal matters were being discussed for example.
If the meeting was by the finance committee who were examining a lease for a property with delegated powers from the council to do so, then it would seem appropriate for the councillor not on the committee to be allowed to remain (but it is not their right to remain).
The argument could be made that if the item is contentious for the whole council, then the matter should be discussed at Full Council but in private when all councillors would remain.
More details required to get a finite answer, Im afraid!