A lot of those people were economic immigrants more than slaves. What use would Afro Levantines, and Jews be? Hewers of wood and drawers of water, the children of Ham, the Jews would be too busy arguing, ritually bathing, extracting denarii from the gullible and finding food to eat. Actually most of the economic immigrants were Greek speakers, people who were from Anatolia, and North African colonies, some from the Levant. They weren't exactly Jews or Black Africans or wholly Greek, they just spoke Greek.
Imperial Rome was a population sink, people went there to make money but they usually died without increasing the population, so more immigrants were needed, and they died, and more immigrants, a cycle of death. Rome wasn't a healthy place to live then, probably not today either, insulae regularly burned down, the cloaca dumped sewage into the Tiber, food had to be imported from Egypt.
Those populations from foreign lands didn't do much to increase the Roman and Italian population. Juvenal wrote that the Tiber was polluted by the Orestes, but he was a bigot. My view is that most of the non Italian population that went to Italy came from Anatolia and West Asia mostly during the late Bronze age into the Iron age, and they weren't Black Africans, Levantines or Jews. And what happened to the deported Muslims that were kept in a concentration camp in Lucera?
Look Italians are assholes, a bunch of cazzi and facce da culo, but it's their culture, not their genetics.