Development process

Development happens against the main branch following the GitHub flow model. Contributors should use their own forks of the repository. In their fork, they create feature branches off of main, and their pull requests should target the main branch. Maintainers are responsible for prompt review of pull requests.

Pull requests against main trigger automated tests that are run through Azure DevOps, GitHub Actions, and CircleCI. Additional test suites are run periodically. When adding new code paths or features, tests are a requirement to complete a pull request. They should be added in the test directory.

Documentation should be provided with pull requests that add or change functionality. This includes comments in the code itself, docstrings, and user guides. For exceptions to this rule the pull request author should coordinate with a maintainer. For changes that fix bugs, add new features, change APIs, etc., i.e., for changes that are relevant to developers and/or users please also add an entry in CHANGES.md in the section corresponding to the next release, since that’s where your change will be included. If you’re a new contributor please also add yourself to AUTHORS.md.

Docstrings should follow numpydoc format. This is a recent decision by the community. The new policy is to update docstrings that a PR touches, as opposed to changing all the docstrings in one PR.

Advanced installation instructions

While working on Fairlearn itself you may want to install it in editable mode. This allows you to test the changed functionality. First, clone the repository locally via

git clone git@github.com:fairlearn/fairlearn.git

To install in editable mode using pip run

pip install -e .

from the repository root path.

To verify that the code works as expected run

python ./scripts/install_requirements.py --pinned False
python -m pytest -s ./test/unit

Fairlearn currently includes plotting functionality that requires the matplotlib package to be installed. Since this is for a niche use case Fairlearn comes without matplotlib by default. To install Fairlearn with its full feature set simply append customplots to the install command

pip install -e .[customplots]

Note that the Fairlearn dashboard is built using nodejs and requires additional steps. To build the Fairlearn dashboard after making changes to it, install Yarn, and then run the widget build script.

The Requirements Files

The prerequisites for Fairlearn are split between three separate files:

The requirements.txt and requirements-customplots.txt files are consumed by setup.py to specify the dependencies to be documented in the wheel files. To help simplify installation of the prerequisites, we have the install_requirements.py script which runs pip install on all three of the above files. This script will also optionally pin the requirements to any lower bound specified (by changing any occurrences of >= to == in each file).

Onboarding guide for users of version 0.2 or earlier

Up to version 0.2, Fairlearn contained only the exponentiated gradient method. The Fairlearn repository now has a more comprehensive scope and aims to incorporate other methods. The same exponentiated gradient technique is now the class fairlearn.reductions.ExponentiatedGradient. While in the past exponentiated gradient was invoked via

import numpy as np
from fairlearn.classred import expgrad
from fairlearn.moments import DP

estimator = LogisticRegression()  # or any other estimator
exponentiated_gradient_result = expgrad(X, sensitive_features, y, estimator, constraints=DP())
positive_probabilities = exponentiated_gradient_result.best_classifier(X)
randomized_predictions = (positive_probabilities >= np.random.rand(len(positive_probabilities))) * 1

the equivalent operation is now

from fairlearn.reductions import ExponentiatedGradient, DemographicParity

estimator = LogisticRegression()  # or any other estimator
exponentiated_gradient = ExponentiatedGradient(estimator, constraints=DemographicParity())
exponentiated_gradient.fit(X, y, sensitive_features=sensitive_features)
randomized_predictions = exponentiated_gradient.predict(X)

Please open a new issue if you encounter any problems.

Investigating automated test failures

For every pull request to main with automated tests, you can check the logs of the tests to find the root cause of failures. Our tests currently run through Azure Pipelines with steps for setup, testing, and teardown. The Checks tab of a pull request contains a link to the Azure Pipelines page), where you can review the logs by clicking on a specific step in the automated test sequence. If you encounter problems with this workflow, please reach out through GitHub issues.

To run the same tests locally, find the corresponding pipeline definition (a yml file) in the devops directory. It either directly contains the command to execute the tests (usually starting with python -m pytest) or it refers to a template file with the command.

Building the website

The website is built using Sphinx and some of its extensions. Specifically, the website is available for all our releases to allow users to check the documentation of the version of the package that they are using.

To be able to build the documentation you need to install all the requirements using pip install -r requirements-dev.txt.

When making changes to the documentation at least run the following command to build the website using your changes:

python -m sphinx -v -b html -n -j auto docs docs/_build/html

or use the shortcut

make doc

This will generate the website in the directory mentioned at the end of the command. Navigate to that directory and find the corresponding files where you made changes, open them in the browser and verify that your changes render properly and links are working as expected.

To fully build the website for all versions use the following script:

python scripts/build_documentation.py --documentation-path=docs --output-path=docs/_build/html

or the shortcut

make doc-multiversion

The comprehensive set of commands to build the website is in our CircleCI configuration file in the .circleci directory of the repository.