Development happens against the master branch following the GitHub flow model. Contributors should use their own forks of the repository. In their fork, they create feature branches off of master, and their pull requests should target the master branch. Maintainers are responsible for prompt review of pull requests.
master
Pull requests against master trigger automated tests that are run through Azure DevOps, GitHub Actions, and CircleCI. Additional test suites are run periodically. When adding new code paths or features, tests are a requirement to complete a pull request. They should be added in the test directory.
test
Documentation should be provided with pull requests that add or change functionality. This includes comments in the code itself, docstrings, and user guides. For exceptions to this rule the pull request author should coordinate with a maintainer. For changes that fix bugs, add new features, change APIs, etc., i.e., for changes that are relevant to developers and/or users please also add an entry in CHANGES.md in the section corresponding to the next release, since that’s where your change will be included. If you’re a new contributor please also add yourself to AUTHORS.md.
Docstrings should follow numpydoc format. This is a recent decision by the community. The new policy is to update docstrings that a PR touches, as opposed to changing all the docstrings in one PR.
All contributions require you to sign the developer certificate of origin (DCO). This is a developer’s certification in which you declare that you have the right to, and actually do, grant us the rights to use your contribution. We use the exact same one created and used by the Linux kernel developers. You can read it at https://developercertificate.org.
You sign the DCO by signing off every commit comment with your name and email address: Signed-off-by: Your Name <your.email@example.com>
When you submit a pull request, a DCO-bot will automatically determine whether you need to provide a DCO and indicate how you can decorate the PR appropriately (e.g., label, comment).
You can manually sign-off by adding a separate paragraph to your commit message:
git commit -m “Your message Signed-off-by: Your Name <your.email@example.com>
or
git commit -m “Your message" -m “Signed-off-by: Your Name <your.email@example.com>”
If this feels like a lot of typing, you can configure your name and e-mail in git to sign-off:
git config --global user.name “Your Name” git config --global user.email “your.email@example.com”
Now, you can sign off using -s or --signoff:
-s
--signoff
git commit -s -m "Your message"
If you find -s too much typing as well, you can also add an alias:
git config --global alias.c "commit --signoff"
Which allows you to commit including a signoff as git c -m "Your Message".
git c -m "Your Message"
These instructions were adapted from this blog post.
You can also fully automate signing off using git hooks, by following the instructions of this stack overflow post.
While working on Fairlearn itself you may want to install it in editable mode. This allows you to test the changed functionality. First, clone the repository locally via
git clone git@github.com:fairlearn/fairlearn.git
To install in editable mode using pip run
pip
pip install -e .
from the repository root path.
To verify that the code works as expected run
python ./scripts/install_requirements.py --pinned False python -m pytest -s ./test/unit
Fairlearn currently includes plotting functionality that requires the matplotlib package to be installed. Since this is for a niche use case Fairlearn comes without matplotlib by default. To install Fairlearn with its full feature set simply append customplots to the install command
matplotlib
customplots
pip install -e .[customplots]
Note that the Fairlearn dashboard is built using nodejs and requires additional steps. To build the Fairlearn dashboard after making changes to it, install Yarn, and then run the widget build script.
The prerequisites for Fairlearn are split between three separate files:
requirements.txt contains the prerequisites for the core Fairlearn package requirements-customplots.txt contains additional prerequisites for the [customplots] extension for Fairlearn requirements-dev.txt contains the prerequisites for Fairlearn development (such as flake8 and pytest)
requirements.txt contains the prerequisites for the core Fairlearn package
requirements-customplots.txt contains additional prerequisites for the [customplots] extension for Fairlearn
[customplots]
requirements-dev.txt contains the prerequisites for Fairlearn development (such as flake8 and pytest)
The requirements.txt and requirements-customplots.txt files are consumed by setup.py to specify the dependencies to be documented in the wheel files. To help simplify installation of the prerequisites, we have the install_requirements.py script which runs pip install on all three of the above files. This script will also optionally pin the requirements to any lower bound specified (by changing any occurrences of >= to == in each file).
pip install
>=
==
Onboarding guide for users of version 0.2 or earlier
Up to version 0.2, Fairlearn contained only the exponentiated gradient method. The Fairlearn repository now has a more comprehensive scope and aims to incorporate other methods. The same exponentiated gradient technique is now the class fairlearn.reductions.ExponentiatedGradient. While in the past exponentiated gradient was invoked via
fairlearn.reductions.ExponentiatedGradient
import numpy as np from fairlearn.classred import expgrad from fairlearn.moments import DP estimator = LogisticRegression() # or any other estimator exponentiated_gradient_result = expgrad(X, sensitive_features, y, estimator, constraints=DP()) positive_probabilities = exponentiated_gradient_result.best_classifier(X) randomized_predictions = (positive_probabilities >= np.random.rand(len(positive_probabilities))) * 1
the equivalent operation is now
from fairlearn.reductions import ExponentiatedGradient, DemographicParity estimator = LogisticRegression() # or any other estimator exponentiated_gradient = ExponentiatedGradient(estimator, constraints=DemographicParity()) exponentiated_gradient.fit(X, y, sensitive_features=sensitive_features) randomized_predictions = exponentiated_gradient.predict(X)
Please open a new issue if you encounter any problems.
For every pull request to master with automated tests, you can check the logs of the tests to find the root cause of failures. Our tests currently run through Azure Pipelines with steps for setup, testing, and teardown. The Checks tab of a pull request contains a link to the Azure Pipelines page), where you can review the logs by clicking on a specific step in the automated test sequence. If you encounter problems with this workflow, please reach out through GitHub issues.
Checks
To run the same tests locally, find the corresponding pipeline definition (a yml file) in the devops directory. It either directly contains the command to execute the tests (usually starting with python -m pytest) or it refers to a template file with the command.
yml
devops
python -m pytest
First add a description of the changes introduced in the package version you want to release to CHANGES.md.
It is also best to verify that the Fairlearn dashboard loads correctly. This is slightly involved:
Install the wheel package by running pip install wheel
wheel
pip install wheel
Create a wheel by running python setup.py sdist bdist_wheel from the repository root. This will create a dist directory which contains a .whl file.
python setup.py sdist bdist_wheel
dist
.whl
Create a new conda environment for the test
In this new environment, install this wheel by running pip install dist/<FILENAME>.whl
pip install dist/<FILENAME>.whl
Install any pip packages required for the notebooks using python ./scripts/install_requirements.py --pinned false
python ./scripts/install_requirements.py --pinned false
Check that the dashboard loads in the notebooks
We have an Azure DevOps Pipeline which takes care of building wheels and pushing to PyPI. Validations are also performed prior to any deployments, and also following the uploads to Test-PyPI and PyPI. To use it:
Ensure that fairlearn/__init__.py has the correct version set.
Put down a tag corresponding to this version but preprended with v. For example, version 0.5.0 should be tagged with v0.5.0.
v
0.5.0
v0.5.0
At queue time, select Test or Production PyPI as appropriate.
As part of the release process, the build_wheels.py script uses process_readme.py to turn all the relative links in the ReadMe file into absolute ones (this is the reason why the applied tag has be of the form v[__version__]). The process_readme.py script is slightly fragile with respect to the contents of the ReadMe, so after significant changes its output should be verified.
build_wheels.py
process_readme.py
v[__version__]