fairlearn.metrics.make_derived_metric#

fairlearn.metrics.make_derived_metric(*, metric, transform, sample_param_names=['sample_weight'])[source]#

Create a scalar returning metric function based on aggregation of a disaggregated metric.

Many higher order machine learning operations (such as hyperparameter tuning) make use of functions which return scalar metrics. We can create such a function for our disaggregated metrics with this function.

This function takes a metric function, a string to specify the desired aggregation transform (matching the methods MetricFrame.group_min(), MetricFrame.group_max(), MetricFrame.difference() and MetricFrame.ratio()), and a list of parameter names to treat as sample parameters.

The result is a callable object which has the same signature as the original function, with a sensitive_features= parameter added. If the chosen aggregation transform accepts parameters (currently only method= is supported), these can also be given when invoking the callable object. The result of this function is identical to creating a MetricFrame object, and then calling the method specified by the transform= argument (with the method= argument, if required).

See the Defining custom fairness metrics section in the User Guide for more details. A sample notebook is also available.

Parameters:

  • metric (callable) – The metric function from which the new function should be derived

  • transform (str) – Selects the transformation aggregation the resultant function should use. The list of possible options is: [‘difference’, ‘group_min’, ‘group_max’, ‘ratio’].

  • sample_param_names (List[str]) – A list of parameters names of the underlying metric which should be treated as sample parameters (i.e. the same leading dimension as the y_true and y_pred parameters). This defaults to a list with a single entry of sample_weight (as used by many SciKit-Learn metrics). If None or an empty list is supplied, then no parameters will be treated as sample parameters.

Returns:

Function with the same signature as the metric but with additional sensitive_features= and method= arguments, to enable the required computation

Return type:

callable