A fallacy matters if it’s central to proving the argument, otherwise it probably doesn’t. Eg Bringing up an anecdote, or a subjective experience as a way of illustrating a point could be said to be fallacious, but is not, if the argument is well supported enough that would stand without it.

I just had an argument where I ended my point with the words “this is a pure could have been:” and added a very likely scenario that may well could have come to pass it some events were different. Obviously it was speculation and not central to the previous argument, but in my estimation likely.

Then other person instead of responding to actual points took the last part and accused me of should’a, would’a, could’a.

Dude, yes! But not the point, also I was the one that pointed it out. The type of person that would explain to a comedian their own joke.