It happened to my father back in the 70's. Unsafe turn turning left into a drive with a pickup truck. Someone decided to pass him despite the turn signals. I was young, but I recall the logic being that the person turning left is entering another right of way. It's their responsibility to make sure no one is using it.

There might be a distinction depending on which part of your vehicle gets hit by the person that was passing. In Ontario, once you are in a lane, you own it, meaning someone who hits you from behind is always at fault. So if your side gets hit, then you are at fault, but if your rear gets hit, the other guy is at fault. Just guessing here. Probably depends a lot on which person the investigating officer likes better that day.