v0.5.0#

  • Adjust classes to abide by naming conventions for attributes.

  • Change ExponentiatedGradient’s signature by renaming argument T to max_iter, eta_mul to eta0, and by adding run_linprog_step.

  • API refactoring to separate out different uses of eps within ExponentiatedGradient. It is now solely responsible for setting the L1 norm bound in the optimization (which controls the excess constraint violation beyond what is allowed by the constraints object). The other usage of eps as the right-hand side of constraints is now captured directly in the moment classes as follows:

    • Classification moments: ConditionalSelectionRate renamed to fairlearn.reductions.UtilityParity and its subclasses have new arguments on the constructor:

      • difference_bound - for difference-based constraints such as demographic parity difference

      • ratio_bound_slack - for ratio-based constraints such as demographic parity ratio

      • Additionally, there’s a ratio_bound argument which represents the argument previously called ratio.

    • Regression moments: ConditionalLossMoment and its subclasses have a new argument upper_bound with the same purpose for newly enabled regression scenarios on ExponentiatedGradient.

    For a comprehensive overview of available constraints refer to the new user guide on fairness constraints for reductions methods.

  • Renamed several constraints to create a uniform naming convention according to the accepted metric harmonization proposal:

  • Added TrueNegativeRateParity to provide the opposite constraint of TruePositiveRateParity to be used with reductions techniques.

  • Add new constraints and objectives in ThresholdOptimizer

  • Add class InterpolatedThresholder to represent the fitted ThresholdOptimizer

  • Add fairlearn.datasets module.

  • Change the method to make copies of the estimator in ExponentiatedGradient from pickle.dump to sklearn.clone.

  • Add an argument sample_weight_name to GridSearch and ExponentiatedGradient to control how sample_weight is supplied to estimator.fit.

  • Large changes to the metrics API. A new class MetricFrame has been introduced, and make_group_summary() removed (along with related functions). Please see the documentation and examples for more information.

Migrating to v0.5.0 from v0.4.6#

The update from v0.4.6 to v0.5.0 of Fairlearn has brought some major changes. This section goes through the adjustments required.

Metrics#

We have substantially altered the fairlearn.metrics module. In place of calling group_summary() to produce a sklearn.utils.Bunch containing the disaggregated metrics, we have a new class, MetricFrame. The key advantages of the new API are:

  • Support for evaluating multiple metric functions at once

  • Support for multiple sensitive features

  • Support for control features

The MetricFrame class has a constructor similar to group_summary(). In v0.4.6, one would write

gs = group_summary(metric_func, y_true, y_pred, sensitive_features=A_col)

With the new API, this becomes

mf = MetricFrame(metrics=metric_func, y_true=y_true, y_pred=y_pred, sensitive_features=A_col)

The new object has MetricFrame.overall and MetricFrame.by_group properties, to access the metric evaluated on the entire dataset, and the metric evaluated on the subgroups of A_col.

In v0.4.6, we provided the following aggregator functions to compute a single scalar from the result of group_summary().

  • group_min_from_summary()

  • group_max_from_summary()

  • difference_from_summary()

  • ratio_from_summary()

With MetricFrame these become methods:

Before, one might write:

min_by_group = group_min_from_summary(gs)

Now, one can write:

min_by_group = mf.group_min()

The make_derived_metric() function has been removed, but will be reintroduced in a future release. The predefined convenience functions such as accuracy_score_group_min() and precision_score_difference() remain.

For an introduction to all the new features, see the Metrics with Multiple Features example in Example Notebooks.

Renamed object attributes#

Some of the object attributes have been renamed from _<name> to <name>_. For example in both ExponentiatedGradient and GridSearch, the _predictors attribute is now called predictors_.

Exponentiated Gradient and Moments#

In addition to the trailing underscore change mentioned above, several adjustments have been made to the constructor arguments of ExponentiatedGradient. The T argument has been renamed to max_iter, and the eta_mul argument to eta0.

Furthermore, the eps argument was previously used for two different purposes, and these two uses have now been separated. The use of eps as the righthand side of the constraints has now been moved to the Moment classes. The only remaining use of the eps argument is to control the optimality requirements for the optimization algorithm in ExponentiatedGradient.

For classification moments, ConditionalSelectionRate has been renamed to UtilityParity, and there are three new constructor arguments: difference_bound, ratio_bound (which replaces ratio) and ratio_bound_slack.

For regression moments, BoundedGroupLoss and its subclasses have gained a new argument upper_bound to serve as the righthand side of the constraints.

Several Moment objects have also been renamed in an effort to improve consistency: