Sometimes a white person may declare that she knows how it feels to be a person of colour, because she has spent a lot of time amongst people of colour. Other people of colour may then declare that she does not know how it feels, because she is white. The white person may then think that she is being discriminated against because of the colour of her skin, that those who accuse her of not empathizing are just making assumptions based on stereotypes about white people.
However, the argument that a person would know how it feels to be of group X because she spent a lot of time with people of group X is fallacious. If you are not of group X, then spending time with people of group X does not change the fact that you are not of group X.
For example, most people would agree that if a visible person of colour spent a lot of time with white people, it does not follow that the person of colour must know how it feels to be white. Some white people find the reverse idea more plausible, because they believe themselves to be raceless or even racially disembodied.