Preventing bias by not providing irrelevant data

Question

This seems like such a simple idea, but I've never heard anyone that has addressed it, and a quick Google revealed nothing, so here it goes.

The way I learned about machine learning is that it recognizes patterns in data, and not necessarily ones that exist -- which can lead to bias. One such example is hiring AIs: If an AI is trained to hire employees based on previous examples, it might recreate previous, human, biases towards, let's say, women.

Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?

Again, this seems like such an easy topic, so I apologize if I'm just being ignorant. But I have learned a bit about AI and machine learning specifically for some time now, and I'm just surprised this hasn't ever been mentioned, not even as a "here's-what-won't-work" example.

naive · Answer 1 · 2019-08-10T06:31:46.387

4

Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?

Yes. The AI/ model still can learn those prejudiced connections. Consider that you have a third variable which is a confounding variable or has spurious relationship that is correlated with the bias variable (BV) and the dependent variable (DV). And, the analyst removed the BV but failed to remove the third variable from the data that is fed to the model. Then the model will learn the relationships the analyst didn't want it to learn.

But, at the same time the removal of the variables could lead to omitted variable bias, which occurs when a relevant variable is left out.

Ex:

Suppose that the goal is prediction of salary ($S$) of an individual and the independent variables are age ($A$) and experience ($E$) of the individual. The analyst wants to remove the bias that could come in because of age. So, she removes age from one of the models and comes up with two competing linear models:

$S = \beta_0 + \beta_1E + \varepsilon$

$S = \beta_0 + \beta_1^*E + \beta_2A + \varepsilon$

Since, experience is highly correlated with age, in presence of age in the model, it is very likely that $\beta_1^* < \beta_1$. $\beta_1$ will be a bogus estimate of a person's experience on salary as the first model suffers from the omitted variable bias.

At the same time the predictions from the first model would be reasonably good although the second model is very likely to beat the first model. So, if the analyst wants to remove any 'bias' that might come in because of age i.e. $A$ she must also remove $E$ from the model.

edited Aug 10 '19 at 06:31

answered Aug 09 '19 at 11:57

naive

699
6
13

I think that a specific example would help to visualize the concept that you describe that involves the confounding variable or the omission of variables. – nbro Aug 09 '19 at 14:55
Yes, that would help. Because my thinking is that of there IS some relevant correlation between the chances of being selected and a seemingly-biased attribute, then maybe that attribute is relevant. An example off the top of my head would be physical ability. If AI is able to identify men and women based on that, and if physical ability is really relevant to being selected, then is it really an unfair bias? And I would say that someone omitting a relevant variable would be more human error than bias. – Levi Lesches Aug 09 '19 at 17:38
Its not an unfair bias that being more able, physically, lands a man a job rather than a woman. It might be the case that more physical ability leads to more productivity. But, if physical ability is not a significant factor at job then the bias towards men, if it exists, is unfair. – naive Aug 09 '19 at 19:17
It is considered omitted variable bias because if the analyst removes one of the confounders then the relationship between the dependent and the ind. could be presented differently than if the confounding variables would have been included in the model. This biases the results i.e. the estimates of the coefficients of regression. – naive Aug 09 '19 at 19:22
But yes when the only goal is prediction then you might be okay to consider it a human error because removing the variables might decrease the predictive ability of the models. – naive Aug 09 '19 at 19:23
@LeviLesches, one thing to remember is that you don't just care about whether something is a relevant factor. You also care about how significant it is, and how much significance the data gives it. For example, suppose the school leaving qualification for Country A is legitimately a higher award than that for Country B, to the extent that all things being equal you should give the job to someone with qualification A, but if someone has qualification B and a few months experience they could overtake. – Josiah Aug 09 '19 at 21:47
However, if there was systematically racist hiring, because what country you come from affects what qualification you get, the dataset might suggest that qualification A is miles better to the point that someone with qualification B couldn't compete. Or indeed, if the racism goes against country A, the data may indicate that people with qualification A are systematically rejected and those with B even at lower grades are hired. An AI trained on this data would learn that having qualification B is a good indicator of whether someone's worth hiring. – Josiah Aug 09 '19 at 21:53
@nbro added an illustration to make things a bit more clear. – naive Aug 10 '19 at 06:32

Josiah · Answer 2 · 2019-08-10T15:52:29.207

Sometimes, the reason that this isn't an option is that you don't have that much control over what data is provided. Suppose, for example, you want a fancy AI that reads a Résumé and filters on suitability for a job. There isn't a particularly rigid formula about what people put in their Résumé, which makes it difficult to exclude things you'd rather not consider.

Where you do have more control about exactly what information you consider, it can still be thwarted by correlations. Think, for a moment, how this pans out with a human decision maker. You want to ensure that Joe Sexist gives women a fair chance at being hired, so you make sure that there isn't a gender field in the application form. You also blind out the applicant's name, since there is no good reason that a name should determine suitability for a role, and including it would reveal a lot of genders. But you don't block out the hobbies, clubs and societies entry, because it's thought to say something positive about an applicant if they were the captain of their college sports team. Joe Sexist, however, considers it a positive if an applicant captained a male dominated team such as American football, but considers negative being captain of a female dominated team! Some might say that wouldn't quite be bias against women; it's bias against players of sports that Joe Sexist considers effeminate. But really a skunk by any other name would stink as bad.

The same sort of thing can happen with AI. Now to be clear, the AI is not sexist. It is a blank sheet with no preconceptions until it gets fed data. But when it gets fed data, it will find patterns in the same way. The dataset it gets given is years of hiring decisions by Joe Sexist. As suggested, there is no entry for gender, but there are fields for all the things that might be considered slightly relevent. For example, we include whether they have a clean driving license. The AI notices that there is a positive correlation between the number of road traffic offences an applicant has and Joe's likelihood of hiring them (Because, of course, there happens to be a gender correlation between dangerous driving and gender). Again, the AI has no preconceptions. It doesn't know that traffic offences are dangerous and should be weighted against. As far as its dataset suggests, they're points! With this sort of information in a dataset, the AI can exhibit all the same sorts of biases as Joe Sexist, even though it doesn't know what a "woman" is!

To expand this with specific numbers, suppose that your dataset has 1000 male and 1000 female applicants for a total of 1000 places. Of those, 400 of the men and 100 of the women have a tarnished traffic record.

Joe Sexist was not in favour of reckless drivers: in fact a clean traffic record guaranteed you would beat an equivalent candidate with a tarnished record. But he was very in favour of men: being male made you 9 times more likely to get hired than being female.

So he gives places to 900 of the men: all 600 of the clean drivers and 300 dirty drivers. He gives places to 100 of the women: all to clean drivers.

Now, you take away any mention of gender in the dataset. There are 2000 people, 500 drive badly, 1500 drive well. Of these, 300 bad drivers get jobs, and 700 good drivers get jobs. Therefore the 25% of the population who drive badly get 30% of the jobs, which means (as far as an AI that just looks blindly at the numbers is concerned) that driving badly suggests you should get the job. That's a problem.

Further, suppose you have a new batch of 2000 applicants with the same ratios and it's the AI's turn to decide. Now often AIs actually make this even worse by exagerating the significance of subtle indicators, but let's suppose that this one does everything in strict proportionality. The AI has learned that 60% (300 / 500) of the bad drivers should get the job. It doesn't know about gender, so it at least allocates the bad driver bonus "fairly": 240 male and 60 female bad drivers get jobs. Then 280 male and 420 female good drivers get jobs. This comes to 520 male and 480 female applicants getting in. Even though the original applicant pool was balanced and if anything women were better (at least at driving) the original sexism in the training dataset still gives some advantage to the men. (as well as giving an advantage to bad drivers)

Now, don't let me completely disuade you. In the human case, it is a known fact that blinding out some information does indeed give more balanced hiring decisions. And even in my toy example, while it doesn't get to fairness it has massively reduced the scale of the sexism. So yes, it probably would make the AI somewhat less sexist if the most blatant indicators aren't provided in the dataset. But perhaps this gives some intuition about why it's not a complete solution to the problem. There is some sexism that leaks through, and it also causes the system to make very weird associations with other bits of the dataset.

score 0 · Answer 3 · answered Aug 09 '19 at 21:27

There is a wider social issue to consider here also. When we build machines, we evaluate what they do and decide if the action that they undertake is to our benefit or not. All societies do this, although you are probably more aware of obvious examples such as the Amish than you are of your own society.

When people complain about biased decision making by AI systems, they are not just evaluating if the result is accurate, but also if that decision supports the values that they wish to see instantiated in society.

You can make a human take cultural factors into account when making a decision, but not an AI that is completely unaware of them. People describe this as complaining about 'bias', but that is not always completely accurate. They are really complaining that the use of AI systems fail to take into account wider social issues that they consider to be important.

Preventing bias by not providing irrelevant data

3 Answers3