I'd say it a case-by-case basis if you decide that your reply is of interest to your followers. I tend to set mine to "Unlisted" (like I'm doing for this one) in which case it won't show up for followers in their timelines but will be visible if people read this thread of replies.
It's a nice way of controlling how you interrupt (or not) the reading flow of others, but without hiding the content.
I rarely use "Followers only" or "Mentioned people only".