The role of a parish council is to spend public money for public benefit. From what you have said, the public benefit of this item of expenditure is questionable. It is also incumbent upon us all to demonstrate the highest standards of probity and integrity in our conduct and not to embroil ourselves in petty squabbles. The Code of Conduct and Nolan Principles are there to be used in situations such as this.
On a more practical level, the 20 years could be of great significance, as this is the period prescribed by law to create a permanent right of access. The legislation is complicated and I'm not a solicitor, but if you Google "easement by prescription" you'll find more detail. Ultimately, only a solicitor specialising in this area can determine whether such a right exists for you, which may simply boil down to the date on which the parishioner first used your land to access his. If a right of access exists, erecting a fence would be against the law and might cost you far more than the cost of the fence.
You mention three councillors who, I presume, are a minority group, so a simple vote against their proposal would prevent it from going any further. In view of the history here, I believe that you are within your rights to ask to see proof that the request was made by a third party, however this is largely immaterial. You should be able to rely upon the support of your Clerk in resolving this situation, however that will depend upon his or her knowledge, experience and willingness to confront members.
I hope that you are able to resolve this amicably.