Getting started with aequitas-report

With aequitas-report uncovering bias is as simple as running a single command on a csv.

Input machine learning predictions

After installing on your computer

Run aequitas-report on COMPAS data:

aequitas-report --input compas_for_aequitas.csv
score label_value race sex age_cat
0 1 African-American Male 25 - 45
1 1 Native American Female Less than 25

The input data at least has:

  • score
  • label_value
  • at least one attribute e.g. race, sex and age_cat.

Getting started with input data

Additionally, disparity is always defined in relation to a reference group. By default, Aequitas uses majority as the reference. Defining a reference group

Get out measures of bias that you can tailor to your problem

The Bias Report output

The Bias Report produces a pdf that returns descriptive interpretation of the results along with three sets of tables.

  • Fairness Measures Results
  • Bias Metrics Results
  • Group Metrics Results

Additionally, a csv is produced that contains the relevant data. More information about output here.

Commandline output

In the command line you will see The Bias Report, which returns counts for each attribute by group and then computes various fairness metrics. This is the same information that is captured in the csv output.

                    ___                    _ __
                   /   | ___  ____ ___  __(_) /_____ ______
                  / /| |/ _ \/ __ `/ / / / / __/ __ `/ ___/
                 / ___ /  __/ /_/ / /_/ / / /_/ /_/ (__  )
                /_/  |_\___/\__, /\__,_/_/\__/\__,_/____/
                              /_/



____________________________________________________________________________

                      Bias and Fairness Audit Tool
____________________________________________________________________________





Welcome to Aequitas-Audit
Fairness measures requested: Statistical Parity,Impact Parity,FDR Parity,FPR Parity,FNR Parity,FOR Parity
model_id, score_thresholds 1 {'rank_abs': [3317]}
COUNTS::: race
African-American    3696
Asian                 32
Caucasian           2454
Hispanic             637
Native American       18
Other                377
dtype: int64
COUNTS::: sex
Female    1395
Male      5819
dtype: int64
COUNTS::: age_cat
25 - 45            4109
Greater than 45    1576
Less than 25       1529
dtype: int64
audit: df shape from the crosstabs: (11, 26)
get_disparity_major_group()
number of rows after bias majority ref group: 11
Any NaN?:  False
bias_df shape: (11, 38)
Fairness Threshold: 0.8
Fairness Measures: ['Statistical Parity', 'Impact Parity', 'FDR Parity', 'FPR Parity', 'FNR Parity', 'FOR Parity']

...