fairlearn.metrics.make_derived_metric#

fairlearn.metrics.make_derived_metric(*, metric, transform, sample_param_names=['sample_weight'])[source]#

Create a scalar returning metric function based on aggregation of a disaggregated metric.

Many higher order machine learning operations (such as hyperparameter tuning) make use of functions which return scalar metrics. We can create such a function for our disaggregated metrics with this function.

This function takes a metric function, a string to specify the desired aggregation transform (matching the methods MetricFrame.group_min(), MetricFrame.group_max(), MetricFrame.difference() and MetricFrame.ratio()), and a list of parameter names to treat as sample parameters.

The result is a callable object which has the same signature as the original function, with a sensitive_features= parameter added. If the chosen aggregation transform accepts parameters (currently only method= is supported), these can also be given when invoking the callable object. The result of this function is identical to creating a MetricFrame object, and then calling the method specified by the transform= argument (with the method= argument, if required).

See the Defining custom fairness metrics section in the User Guide for more details. A sample notebook is also available.

Parameters

  • metric (callable) – The metric function from which the new function should be derived

  • transform (str) – Selects the transformation aggregation the resultant function should use. The list of possible options is: [‘difference’, ‘group_min’, ‘group_max’, ‘ratio’].

  • sample_param_names (List[str]) – A list of parameters names of the underlying metric which should be treated as sample parameters (i.e. the same leading dimension as the y_true and y_pred parameters). This defaults to a list with a single entry of sample_weight (as used by many SciKit-Learn metrics). If None or an empty list is supplied, then no parameters will be treated as sample parameters.

Returns

Function with the same signature as the metric but with additional sensitive_features= and method= arguments, to enable the required computation

Return type

callable