4.1. Migrating to v0.5.0 from v0.4.6

The update from v0.4.6 to v0.5.0 of Fairlearn has brought some major changes. This document goes through the adjustments required.

4.1.1. Metrics

We have substantially altered the fairlearn.metrics module. In place of calling group_summary() to produce a sklearn.utils.Bunch containing the disaggregated metrics, we have a new class, MetricFrame. The key advantages of the new API are:

  • Support for evalulating multiple metric functions at once

  • Support for multiple sensitive features

  • Support for control features

The MetricFrame class has a constructor similar to group_summary(). In v0.4.6, one would write

gs = group_summary(metric_func, y_true, y_pred, sensitive_features=A_col)

With the new API, this becomes

mf = MetricFrame(metrics=metric_func, y_true=y_true, y_pred=y_pred, sensitive_features=A_col)

The new object has MetricFrame.overall and MetricFrame.by_group properties, to access the metric evaluated on the entire dataset, and the metric evaluated on the subgroups of A_col.

In v0.4.6, we provided the following aggregator functions to compute a single scalar from the result of group_summary().

  • group_min_from_summary()

  • group_max_from_summary()

  • difference_from_summary()

  • ratio_from_summary()

With MetricFrame these become methods:

Before, one might write:

min_by_group = group_min_from_summary(gs)

Now, one can write:

min_by_group = mf.group_min()

The make_derived_metric() function has been removed, but will be reintroduced in a future release. The predefined convenience functions such as accuracy_score_group_min() and precision_score_difference() remain.

For an introduction to all the new features, see the Metrics with Multiple Features example in Example Notebooks.

4.1.2. Renamed object attributes

Some of the object attributes have been renamed from _<name> to <name>_. For example in both ExponentiatedGradient and GridSearch, the _predictors attribute is now called predictors_.

4.1.3. Exponentiated Gradient and Moments

In addition to the trailing underscore change mentioned above, several adjustments have been made to the constructor arguments of ExponentiatedGradient. The T argument has been renamed to max_iter, and the eta_mul argument to eta0.

Furthermore, the eps argument was previously used for two different purposes, and these two uses have now been separated. The use of eps as the righthand side of the constraints has now been moved to the Moment classes. The only remaining use of the eps argument is to control the optimality requirements for the optimization algorithm in ExponentiatedGradient.

For classification moments, ConditionalSelectionRate has been renamed to UtilityParity, and there are three new constructor arguments: difference_bound, ratio_bound (which replaces ratio) and ratio_bound_slack.

For regression moments, BoundedGroupLoss and its subclasses have gained a new argument upper_bound to serve as the righthand side of the constraints.

Several Moment objects have also been renamed in an effort to improve consistency: