# Defining custom fairness metrics#

Higher level machine learning algorithms (such as hyperparameter tuners) often make use of metric functions to guide their optimisations. Such algorithms generally work with scalar results, so if we want the tuning to be done on the basis of our fairness metrics, we need to perform aggregations over the `MetricFrame`.

We provide a convenience function, `fairlearn.metrics.make_derived_metric()`, to generate scalar-producing metric functions based on the aggregation methods mentioned above (`MetricFrame.group_min()`, `MetricFrame.group_max()`, `MetricFrame.difference()`, and `MetricFrame.ratio()`). This takes an underlying metric function, the name of the desired transformation, and optionally a list of parameter names which should be treated as sample aligned parameters (such as sample_weight). The result is a function which builds the `MetricFrame` internally and performs the requested aggregation. For example:

```>>> from fairlearn.metrics import make_derived_metric, MetricFrame
>>> from sklearn.metrics import recall_score
>>> y_true = [0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1]
>>> y_pred = [0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0]
>>> sf_data = ['b', 'b', 'a', 'b', 'b', 'c', 'c', 'c', 'a',
...            'a', 'c', 'a', 'b', 'c', 'c', 'b', 'c', 'c']
>>> recall_difference = make_derived_metric(metric=recall_score,
...                                        transform='difference')
>>> recall_difference(y_true, y_pred,
...                   sensitive_features=sf_data)
0.19999...
>>> MetricFrame(metrics=recall_score,
...             y_true=y_true,
...             y_pred=y_pred,
...             sensitive_features=sf_data).difference()
0.19999...
```

We use `fairlearn.metrics.make_derived_metric()` to manufacture a number of such functions which are commonly used. The table below displays the aggregations that we have created for each base metric:

Base metric

`group_min`

`group_max`

`difference`

`ratio`

`false_negative_rate()`

.

.

Y

Y

`false_positive_rate()`

.

.

Y

Y

`selection_rate()`

.

.

Y

Y

`true_negative_rate()`

.

.

Y

Y

`true_positive_rate()`

.

.

Y

Y

`sklearn.metrics.accuracy_score()`

Y

.

Y

Y

`sklearn.metrics.balanced_accuracy_score()`

Y

.

.

.

`sklearn.metrics.f1_score()`

Y

.

.

.

`sklearn.metrics.log_loss()`

.

Y

.

.

`sklearn.metrics.mean_absolute_error()`

.

Y

.

.

`sklearn.metrics.mean_squared_error()`

.

Y

.

.

`sklearn.metrics.precision_score()`

Y

.

.

.

`sklearn.metrics.r2_score()`

Y

.

.

.

`sklearn.metrics.recall_score()`

Y

.

.

.

`sklearn.metrics.roc_auc_score()`

Y

.

.

.

`sklearn.metrics.zero_one_loss()`

.

Y

Y

Y

The names of the generated functions are of the form `fairlearn.metrics.<base_metric>_<transformation>`. For example `fairlearn.metrics.accuracy_score_difference` and `fairlearn.metrics.precision_score_group_min`.