We know that one of the reasons that Mohammed confronted the Jews was that he coveted their wealth and lands.
Another reason why the Jews had to reject Mohammed was that Mohammed was not a Jew. All the prophets of the OT during the national old covenant form of Israel of the Mosaic covenant were Israelites. It could not be otherwise.
He knew that the Jews had to reject him because he could have no relevance for them and he had no ethnic right to be a prophet of Israel . He knew that this would provide an excuse, among others, for him to attack them. This was a calculated strategy from the get go.