I think the most dangerous part of contemporary Leftist and "progressive" thought is indeed that race makes any difference pertaining to moral authority.
Frankly, the main reason it's important for me as a Jewish person to identify as non-white is because we are being currently framed as "white colonizers" in our indigenous homeland in order to justify literal rape, torture, abduction and execution of Jewish civilians (including actual babies, children and elderly). That very reductive/moronic Leftist worldview that *does* conceptualize all matters of race as a binary black/white framework of moral authority has decided that on the erroneous basis of Jews allegedly being "white" (despite the objective phenotypic realities of the Israeli populace, which literally includes swaths of actual Black people and a majority of visibly brown-skinned Jews), that we are intrinsically colonial, violent, power-hungry and bad.
But this is also garbage because it was only 80 years ago that our population was virtually cut in half from genocide precisely because we are not white, and the Nazis decisively found us to be an "inferior" race. And this same sentiment is still echoed in American neo-Nazi and KKK spheres, which have long hated Jews and seen us as a non-white, inferior race.
You are spot on that intersectionality as a concept makes "privilege" quite relative. You as a white person may indeed have significantly less privilege than someone like Beyonce, who is wealthy, able-bodied, Christian, and meets all the right criteria of Western beauty standards to the point where she is literally worshipped. It's an imperfect mechanism of understanding hierarchies of power, and I really don't think it should be used to engage in oppression Olympics, nor to fully vindicate or victimize any individual person (or groups of people). It's a tool to understand a world where inequality is still rampant, but it shouldn't be the only north star.
I resent any framing of so-called "people of color" as intrinsically more ethical than white people. To make such a characterization is to engage in the very tenets of historical white supremacy (e.g. referring to Native people as "savages") that got us into this mess in America to begin with.
Thanks for reading and commenting!