Three or four people is plenty to start. You do not need a crowd. The very first SCA UK meet up, in a London pub in February 2015, had thirteen people and it changed the course of all of them. Many regional meet ups since have started with far fewer.
If you can find two or three others in your area who want to come, you have everything you need. Post in our community to gauge interest, check the member map to see who is near you, and go from there. Numbers often grow naturally once people see that it actually happened.