Is he breaking up with me?

That could have multiple meanings. But sounds to me like he is breaking up with you. I could be wrong though, you might want to ask him.