It is a common misconception but it is a misconception. By virtue of the Local Government Act 1972 s100 and the Public Bodies (Admission to Meetings) Act 1960, members of the public and press have right to attend and observe town and parish council meetings and meetings of committees of town and parish councils. The press or public may be excluded from a meeting for one or more agenda items if publicity during the discussion of that item would be prejudicial to the public interest. A resolution to exclude the press and public during those items is required. Of all types of discussion, it is common for staffing matters to be discussed in confidential session but note that the meeting itself must be open to the public at the start, notice must be given (i.e. the notice of meeting must be in a conspicuous place within the parish (and your website) and the meeting must allow the public to attend, even if it is only for the initial part of the meeting such as the declaration of interests, to receive apologies or whilst the minutes of the previous meeting are approved. Frankly, your committee terms of reference are legally non-compliant. As with all minutes of meetings of the council and its committees, minutes are required to be available to the public and should be published on your website. As I have said, the discussion can be held without members of the public in attendance but the resultant resolution must be minuted and be available to the public.
The only type of meeting that can be held without due notice or the public in attendance is that of a working party or task and finish group but such meetings cannot make binding decisions. They can only make recommendations to council.