Advanced Usage of MetricFrame#
In this section, we will discuss how MetricFrame can
be used in more sophisticated scenarios.
All code examples will use the following definitions:
>>> y_true = [0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1]
>>> y_pred = [0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0]
>>> sf_data = ['b', 'b', 'a', 'b', 'b', 'c', 'c', 'c', 'a',
... 'a', 'c', 'a', 'b', 'c', 'c', 'b', 'c', 'c']
>>> from fairlearn.metrics import MetricFrame
Extra Arguments to Metric functions#
The metric functions supplied to MetricFrame might require additional
arguments.
These fall into two categories: ‘scalar’ arguments (which affect the operation
of the metric function), and ‘per-sample’ arguments (such as sample weights).
Different approaches are required to use each of these.
Scalar Arguments#
We do not directly support scalar arguments for the metric functions.
If these are required, then use functools.partial() to prebind the
required arguments to the metric function:
>>> import functools
>>> from sklearn.metrics import fbeta_score
>>> fbeta_06 = functools.partial(fbeta_score, beta=0.6)
>>> metric_beta = MetricFrame(metrics=fbeta_06,
... y_true=y_true,
... y_pred=y_pred,
... sensitive_features=sf_data)
>>> metric_beta.overall
0.56983...
>>> metric_beta.by_group
sensitive_feature_0
a 0.365591
b 0.850000
c 0.468966
Name: metric, dtype: float64
Per-Sample Arguments#
If there are per-sample arguments (such as sample weights), these can also be
provided in a dictionary via the sample_params argument.
The keys of this dictionary are the argument names, and the values are 1-D
arrays equal in length to y_true etc.:
>>> from sklearn.metrics import recall_score
>>> import pandas as pd
>>> pd.set_option('display.max_columns', 20)
>>> pd.set_option('display.width', 80)
>>> s_w = [1, 2, 1, 3, 2, 3, 1, 2, 1, 2, 3, 1, 2, 3, 2, 3, 1, 1]
>>> s_p = { 'sample_weight':s_w }
>>> weighted = MetricFrame(metrics=recall_score,
... y_true=y_true,
... y_pred=y_pred,
... sensitive_features=pd.Series(sf_data, name='SF 0'),
... sample_params=s_p)
>>> weighted.overall
0.45...
>>> weighted.by_group
SF 0
a 0.500000
b 0.583333
c 0.250000
Name: recall_score, dtype: float64
If multiple metrics are being evaluated, then sample_params becomes a
dictionary of dictionaries.
The first key to this dictionary is the name of the metric as specified
in the metrics argument.
The keys of the inner dictionary are the argument names, and the values
are the 1-D arrays of sample parameters for that metric.
For example:
>>> s_w_2 = [3, 1, 2, 3, 2, 3, 1, 4, 1, 2, 3, 1, 2, 1, 4, 2, 2, 3]
>>> metrics = {
... 'recall' : recall_score,
... 'recall_weighted' : recall_score,
... 'recall_weight_2' : recall_score
... }
>>> s_p = {
... 'recall_weighted' : { 'sample_weight':s_w },
... 'recall_weight_2' : { 'sample_weight':s_w_2 }
... }
>>> weighted = MetricFrame(metrics=metrics,
... y_true=y_true,
... y_pred=y_pred,
... sensitive_features=pd.Series(sf_data, name='SF 0'),
... sample_params=s_p)
>>> weighted.overall
recall 0.500000
recall_weighted 0.454545
recall_weight_2 0.458333
dtype: float64
>>> weighted.by_group
recall recall_weighted recall_weight_2
SF 0
a 0.5 0.500000 0.666667
b 0.6 0.583333 0.600000
c 0.4 0.250000 0.272727
Note that there is no concept of a ‘global’ sample parameter (e.g. a set of sample weights to be applied for all metric functions). In such a case, the sample parameter in question must be repeated in the nested dictionary for each metric function.
No y_true or y_pred#
In some cases, a metric may not have y_true or y_pred arguments, or even
either of them.
One example of this is the selection rate metric, which only considers
the y_pred values (selection rate is used when computing
demographic parity).
However, MetricFrame requires all supplied metric functions to
conform to the scikit-learn metric paradigm, where the first two arguments
to the metric function are the y_true and y_pred arrays.
The workaround in this case is to supply a dummy argument.
This is the approach we use in selection_rate(), which simply ignores
the supplied y_true argument.
When invoking MetricFrame, a y_true array of the appropriate length
must still be supplied.
For example:
>>> from fairlearn.metrics import selection_rate
>>> dummy_y_true = [x for x in range(len(y_pred))]
>>> sel_rate_frame = MetricFrame(metrics=selection_rate,
... y_true=dummy_y_true,
... y_pred=y_pred,
... sensitive_features=pd.Series(sf_data, name='SF 0'))
>>> sel_rate_frame.overall
0.55555...
>>> sel_rate_frame.by_group
SF 0
a 0.75
b 0.50
c 0.50
Name: selection_rate, dtype: float64
More Complex Metrics#
Metric functions often return a single scalar value based on arguments which are vectors of
scalars.
This is how MetricFrame was introduced in the Performing a Fairness Assessment
section above.
However, this need not be the case - indeed, we were rather vague about the
contents of the input vectors and the return value of the metric function.
We will now show how to use MetricFrame in cases where the result is not
a scalar, and when the inputs are not vectors of scalars.
Non-Scalar Results from Metric Functions#
Metric functions need not return a scalar value.
A straightforward example of this is the confusion matrix.
Such return values are fully supported by MetricFrame:
>>> from sklearn.metrics import confusion_matrix
>>> mf_conf = MetricFrame(
... metrics=confusion_matrix,
... y_true=y_true,
... y_pred=y_pred,
... sensitive_features=sf_data
... )
>>> mf_conf.overall
array([[2, 4],
[6, 6]]...)
>>> mf_conf.by_group
sensitive_feature_0
a [[0, 2], [1, 1]]
b [[1, 0], [2, 3]]
c [[1, 2], [3, 2]]
Name: confusion_matrix, dtype: object
Obviously for such cases, operations such as MetricFrame.difference() have no meaning.
However, if scalar-returning metrics are also present, they will still be calculated:
>>> mf_conf_recall = MetricFrame(
... metrics={ 'conf_mat':confusion_matrix, 'recall':recall_score },
... y_true=y_true,
... y_pred=y_pred,
... sensitive_features=sf_data
... )
>>> mf_conf_recall.overall
conf_mat [[2, 4], [6, 6]]
recall 0.5
dtype: object
>>> mf_conf_recall.by_group
conf_mat recall
sensitive_feature_0
a [[0, 2], [1, 1]] 0.5
b [[1, 0], [2, 3]] 0.6
c [[1, 2], [3, 2]] 0.4
>>> mf_conf_recall.difference()
conf_mat NaN
recall 0.2
dtype: float64
We see that the difference between group recall scores has been calculated, while a value of
None has been returned for the meaningless ‘maximum difference between two confusion matrices’
entry.
Inputs are Arrays of Objects#
MetricFrame can also handle cases when the \(Y_{true}\) and/or \(Y_{pred}\) vectors
are not vectors of scalars.
It is the metric function(s) which gives meaning to these values - MetricFrame itself
just slices the vectors up according to the sensitive feature(s) and the control feature(s).
As a toy example, suppose that our y values (both true and predicted) are tuples representing
the dimensions of a rectangle.
For some reason known only to our fevered imagination (although it might possibly be due to a
desire for a really simple example), we are interested in the areas of these rectangles.
In particular, we want to calculate the mean of the area ratios. That is:
>>> import numpy as np
>>> def area_metric(y_true, y_pred):
... def calc_area(a):
... return a[0] * a[1]
...
... y_ts = np.asarray([calc_area(x) for x in y_true])
... y_ps = np.asarray([calc_area(x) for x in y_pred])
...
... return np.mean(y_ts / y_ps)
This is a perfectly good metric for MetricFrame, provided we supply appropriate
inputs.
>>> y_rect_true = [(4,9), (3,8), (2,10)]
>>> y_rect_pred = [(1,12), (2,1), (5, 2)]
>>> rect_groups = { 'sf_0':['a', 'a', 'b'] }
>>>
>>> mf_non_scalar = MetricFrame(
... metrics=area_metric,
... y_true=y_rect_true,
... y_pred=y_rect_pred,
... sensitive_features=rect_groups
... )
>>> print(mf_non_scalar.overall)
5.6666...
>>> print(mf_non_scalar.by_group)
sf_0
a 7.5
b 2.0
Name: area_metric, dtype: float64
For a more concrete example, consider an image recognition algorithm which draws a bounding box around some region of interest. We will want to compare the ‘true’ bounding boxes (perhaps from human annotators) with the ones predicted by our model. A straightforward metric for this purpose is the IoU or ‘intersection over union.’ As the name implies, this metric takes two rectangles, and computes the area of their intersection and divides it by the area of their union. If the two rectangles are disjoint, then the IoU will be zero. If the two rectangles are identical, then the IoU will be one. This is presented in full in our example notebook.