A custodian trustee has no legal power to take back an asset. A custodian trustee does not own a charity's assets. They remain the property of the charity (or a third party). The role of custodian trustee is to provide a legal entity to hold the title. That is all. The custodian trustee must, at all times, act in accordance with the wishes of the managing trustees, in your case, the management committee.
If the managing trustees are unable or unwilling to continue managing the asset in accordance with their governing document, they must follow the dissolution clause in their governing document to determine the future of the asset. They cannot just walk away. If the dissolution clause requires them to identify another organisation to continue to operate the facility, they may choose their custodian trustee, subject to their agreement. The impetus for all of this must come from the management committee, as managing trustees.
Reading between the lines, is it possible that your parish council has misunderstood its role here and tried to take a more hands-on role than it should? If so, you might be the problem, not the solution.
I'm happy to provide more specific guidance if required.