You miss the point. To oneself common sense is a self-evident abstract. Therefore a person or group can believe they are inherently right whilst another group maybe as self-assured whilst holding a diamterically opposing set of ideas that are right in accordance with their "common sense".
People are unequal physically, socio-economically etc etc...therefore the idea that the powerful can subject the weak and take advantage of them seems to be a sensible idea..it makes perfect sense..in fact most of the great, ancient philosophers agreed with this. The manifestations of this has changed over the years from the ancient Egyptians through to the Nazis.. .the fact that a "self evident abstract" (common sense) is subject to change dependent on time, location, culture, etc etc has resulted in these eras of great injustice.
This is where religion comes in..it mandates a set divine morals and rights that are unchanging and passed on from generation to generation. In fact the moral underpinings of secular societies, put forward as ideals of governance by athiests, are all derived from religion.