It is rude of people to attach themselves to a lunch party without being invited. Lunch is not work time, and people can spend it how they want. On the other hand you have identified an issue, that other people in the office want to do lunch with colleagues, and they aren't getting the opportunity, and it would be good for the workplace if you do something about it. AndAlso constantly avoiding colleagues who would like to socialize with you is not very friendly, and by letting this continue you have established it as "OK" that anyone can tag along with anyone going to lunch.
I suggest a compromise. Agree with your regular lunch crew that you will take a break from it for a week or so, and maybe not lunch together every day. Then you establish a new norm of how people do lunch if they want to.
Next time start to get ready for lunch, and then say to the office "anyone want to go to lunch?". The people who want to lunch will say yes, and you are establishing the principle of invitation, as well as your own credentials as a friendly person. Do lunch with anyone who wants to; have fun; be nice. Keep doing this for several days until it becomes normal.
When it's normal, it's time to do occasional lunches with your old crew. Arrange to meet them at a place you are unlikely to encounter the other people. Start to get ready for lunch, but don't issue the invitation, and if people try to join you say "I'm sorry, I'm lunching with someone else today. Go without me. I'll join you tomorrow." If more than one person has tried to join, point out that they can do lunch together and don't actually need you. Hopefully they will do that. Now you've established a norm, which is that when you want to lunch with everybody you ask, and when you don't ask it's because you are doing something else. And that they can lunch with their colleagues without needing you. With any luck other people will start initiating - getting ready for lunch and inviting other people.
Keep this up for a bit. You lunch sometimes with your team, sometimes with your buddies. Your team doesn't feel excluded, and hopefully the team sets a precedent of lunching together without needing you to enable it. If the team starts to do lunch even when you are not there, you can increase the number of times you take to lunch with your buddies, maybe even back to doing it most of the time.