At the height of European colonialism there were numerous experiments undertaken to prove different levels of intelligence between races, such as filling skulls of different races with grapeshot. Inferences about the intelligence of different races were made by measuring the capacity each races' skull could hold. Of course, the more grapeshot meant the more space for brains...this was seen as totally valid science and was performed by esteemed scientists.
The problem is that while we now know that brain mass does not necessarily correlate to greater or less intelligence, in those days it was an acknowledged 'fact'. What happened then (and still happens today), is that scientists make their evidence fit what is currently accepted as truth. It was simply obvious to people in the early 20th Century that black people were less intelligent, and scientists were not looking to see whether or not this was true, but rather looking for new ways to confirm this.
'Facts' and 'truth' are incredibly relative, and subject to constant change in society as we gain new knowledge, and as our contexts change. I think this is why you're on shaky ground. The IQ test has lost a lot of credibility in modern scientific circles, and is rarely used anymore. If you are going to argue this point, you might want to find yourself some newer science.