Free speech isn’t just about speech it is about context. We are free to say anything we want in the right context. But we are not free to say anything we want in any context.
Anti-war protestors picketing a dead soldiers funeral would technically be legal, but is it just? No.
A university with a significant transgender community is not the right context for a speaker that disagrees with trans rights. There may be another university where the student body is not as diverse that would be a more appropriate setting for that speaker.
Each university should be free to offer a specific context, and only accept speakers that express opinions fitted to that context.
The government actually acknowledges this in other spaces. Its 2014 Public Spaces Protection Order (PSPOs) allows local authorities to penalise individuals if their conduct has had “a detrimental effect on the quality of life of those in the locality”.
It is this order that would allow a council to prevent a far-right, anti-immigration group marching through an area with a significant immigrant population.