Many white people feel that white people are stereotyped as racists and are victims of racial prejudice. Consequently, many whites assume that non-white people will be racially prejudiced against whites on the basis of their skin colour, assuming that they are racist because they are white.
Of course, most people of colour who were raised in white-majority societies are aware of this, and some of us make extraordinary efforts to make white friends and parade them around to make ourselves appear “not racist”, or at least not racially prejudiced against ‘whitey’. What makes the stereotype that non-whites are racially prejudiced against whites so resilient is that it fits nicely into the stereotype that non-white people always stick together and self-segregate.