I don't understand why people keep dissing my accent?

When I was younger I was taken to an international school where most of the students and teachers were virtually white. I had trouble communicating initially cause it was British system and they'd only vividly hear me if I had a British accent. I'm older now, I can speak my native tongue but I tend to slip and sound a bit British cause it stuck, especially when I speaking English. Where I come from, since I'm black, people are racist and tribalist in the sense that they'd only buy such accents from white foreigners. Many insult me because of my diction and English vocabulary, just cause they wouldn't ordinarly be familiar with those words. But when I converse with my British mates they think I'm intelligent, and we communicate quite fine. It's not that I'm putting it on, it's just that the teachers always taught me to speak with certain diction. Everyone seems to have a problem with the school I went to as well. They just go on and on about how people who go there are snobs and such, but I'd like to think I turned out quite fine. Now I'm moving to Scotland next month cause people just don't seem to get me and my acquintances think I'll be worse when I get back. Your thoughts?