That depends what is meant by a subgroup. Councils need to take care to consider the impression they are giving to their citizens. But subject to that, there is nothing to stop a council from creating a group that is delegated to look into a particular matter, with a view to coming back to the council with recommendations. An informal group of this kind can meet in private and only produce sufficient documents to enable it to operate effectively.
The recommendations to the council should be sufficiently detailed to ensure that the final decisions are plainly being made by the council, with opportunities for councillors who were not part of the group to query the proposals and consider alternatives.
If the intention is that decisions should be made, then that can only be done by a committee of the council, which is then required to meet all the same rules as for the council in relation to meeting in public, giving notice of meetings, publishing agendas and minutes, etc.