The Local SDA Conferenced typically has the authority to ban specific people from speaking in Churches. That is due in part to the fact that typic ally the Local Conference has ownership of the actual property in which the congregation meets. There are few situations in which the congregation is meeting on/in property that is not owned by the Local Conference. That raises issues that are not easily answered.
In the end, the authority exists for a congregation to be expelled from the SDA Church. That is seldom done today outside of it being done for a lack of membership. I have personally been involved is such when the adults attending numbered 6 or 7, as I recall.