Brian
It’s a good thing that you’re feeling worried — getting involved with someone who is already partnered certainly has its set of emotional risks. Your fear is a healthy response that is like those weird highway signs proclaiming “Danger: Falling Rocks.” (What on earth are you really supposed to do in that situation?)
On some level, you have actively chosen to connect with someone who isn’t actually available and there could be any number of reasons why you would do this. You’ll need to be honest with yourself in your exploration of how, despite stressing you out, his unavailability is in some way serving you. For example, if you’ve been agonizingly heartbroken in a previous relationship, there’s a chance that your current set-up feels safe if only because there’s not much to lose to begin with.
How you proceed should also be informed by your personal values. If your loverman is keeping your hookups (and any others he’s got going on) a secret, then you’ll need to know whether this is something your conscience can feel okay with. If they’ve agreed upon an open relationship, then this becomes less of a moral/values issue, though it still remains a potentially risky situation. One injuring outcome, for example, is that their relationship eventually improves and you find yourself discarded just as you’ve become fully attached.
Your part-time lover is also providing you with valuable information about himself: Part of how he copes with dissatisfaction in his relationship is by getting something on the side. Should you two ever end up together, there’s a strong likelihood that he could avoid addressing relationship issues head-on and act out instead. In fact, there’s an opportunity here for you to take some responsibility by speaking up and initiating a discussion about your respective intentions.
If you are both able to stay on the same page in strictly wanting occasional mind-blowing sex, then it’s pretty safe. If you’re holding out hope that he’ll ultimately toss his relationship and choose you in a romcom run-to-the-airport-to-claim-you kind of way, then you might be on a path of self-sabotage that only you can choose to intercept.