Why are friends joining me in Fortnite when my Party Joinability is set to Invite only?

Anyone in your party can change the party's joinability, and everyone in the party is notified when it changes. If people are joining unexpectedly, check whether another member switched the party to Joinable.

When you change party joinability yourself, it applies to your current party and also becomes your personal default once you leave that party. When someone else in your party changes it, it applies to the current party only and does not change your default.