Please review further resources,
where we provide links to various materials that we have found helpful.
Also, in our user guide, we link to the papers describing
the algorithms implemented in Fairlearn.
If your goal is to train a model whose predictions are statistically
independent of the sensitive features, then it is not enough to simply ignore the
Information is often redundantly encoded across several features, and machine learning
algorithms will uncover these links (it is what they are designed to do).
For example, in the US, the ZIP code where a person lives is well correlated with their
Even if the model is not provided with race as a feature, the model will pick up on it
implicitly via the ZIP code (and other features).
Worse, without having the race available in the dataset, it is hard to assess the
model’s impact across different groups defined by race or by race intersected with other
The answer to this question depends on what is meant by ‘unfair’, ‘biased data’,
and ‘better data’ in any particular context.
Consider the example of a company seeking to build a tool for screening the resumes of
The company is planning to use their internal job evaluation data and train a
model to predict job evaluations of the applicants; those with higher
predictions will be ranked higher by the screening tool. This setup might
present several fairness issues:
If the company has historically hired few women, there will be fewer of them
in the training data set, and so a trained model may be less accurate
The choice of features also affects the accuracy of the model. The features
that are predictive for one group of applicants might not be as predictive for another
group, and so more data will not necessarily improve the accuracy.
The accuracy of a model might not mean that the model is fair. If women have received
systematically poorer reviews due to biased managers or worse workplace conditions,
then the model might appear to be accurate,
but the choice of the label (in this case, job evaluation)
does not accurately reflect the applicants’ potential.
These are just three ways how the data may be ‘biased’, and they are not mutually
exclusive. The processes for getting ‘better data’ will be different for
each. In some of these cases, obtaining ‘better data’ may not be practical, but it
might still be possible to use some mitigation algorithms.
Machine learning models often perform poorly for subgroups which are poorly
What constitutes poor representation is context specific, and may well be affected by
historical misrepresentation (consider the example above, of a company which had previously
hired few women).
For this reason, balanced sampling is generally better for ML than population sampling.
On a related point, this is why it is important to consider multiple fairness metrics,
and how they vary across different subgroups.
There are often many machine learning models that achieve similar levels of accuracy
or other performance metrics, but that dramatically differ in how they affect
Mitigation algorithms seek to improve the fairness metrics without strongly affecting
the accuracy, or more generally to navigate the trade-offs between performance and
There are many ways in which a model can be unfair. Fairlearn mitigation algorithms
only address some of them: those that can be quantified by our supported
However, to assess whether the new model is fairer, it is important to consider
not only the fairness metrics, but also the societal and technical context in which
the model is applied.
Firstly, always remember that there is more to fairness than technical details such as
metrics - fairness is a sociotechnical problem.
Even if these are all considered at training time, the ML lifecycle doesn’t end when a
model is deployed.
Models need to be monitored in production.
On a technical level, this means checking for data drift within the vulnerable
subgroups identified during the fairness analysis.
However the societal aspects need to be considered as well, for example:
Are the actual harms as expected, both in the nature of the harms and their
Have the users of the model (who may not be the subjects of the model)
adjusted their usage patterns? This is sometimes called ‘strategic
We currently focus on two kinds of harms:
These harms can occur when AI systems extend or withhold opportunities, resources,
or information. Some of the key applications are in hiring, school admissions, and lending.
Quality-of-service harms. Quality of service refers to whether a system works
as well for one person as it does for another, even if no opportunities, resources,
or information are extended or withheld.
We do not have concrete plans for this at the present time.
Right now we do not have an automated tool that would help you decide
which mitigation algorithm to use. Our focus is on expanding the documantation
and examples to highlight when each of the algorithms might be more applicable.
Note that model training is just one step in the AI development and
deployment lifecycle, and other steps, such as data gathering and curation,
or monitoring and debugging of the deployed system, may be better places
of intervention to improve the fairness of an AI system.
Please see our Mitigation section.
We have generally followed conventions of scikit-learn.
However, our mitigation algorithms can be used to augment
any ML algorithms that provide (or can be wrapped to provide) fit() and
predict() methods. Also, any classification or regression
algorithm can be evaluated using our metrics.
We have not (yet) looked at using Fairlearn on image or text data.
However, so long as the image or text classifier provide
fit() and predict() methods
as required by Fairlearn, it should be possible to use them
with Fairlearn mitigation algorithms. Also, any classification or regression
algorithm can be evaluated using our metrics (regardless of the
data it is operating on).
For the moment, we only support Python >= 3.6
Absolutely! Please see our contributor guide to see
how. We welcome all contributions!
Fairlearn has grown from a project at Microsoft Research in New York City.