The Bias and Fairness Audit Toolkit for Machine Learning

Aequitas is an open-source bias audit toolkit for machine learning developers, analysts, and policymakers to audit machine learning models for discrimination and bias, and to make informed and equitable decisions around developing and deploying predictive risk-assessment tools.

Learn more about the project.

Demo

See what Aequitas can do.

Sample Jupyter Notebook

Explore bias analysis of the COMPAS data using the Aequitas library.

• Welcome to Aequitas
  • Getting started
• Bias measures tailored to your problem
• Installation
• Understanding Input Data
  • Input data for Webapp
    • score
    • label_value
    • attributes e.g. race, sex, age,income
  • Input data for CLI
    • score
    • label_value
    • attributes e.g. race, sex, age,income
    • model_id
    • Reserved column names:
  • Input data for Python package
    • score
    • label_value
    • attributes e.g. race, sex, age,income
    • model_id
    • Reserved column names:
• Understanding Output
  • Fairness overview
  • Fairness Criteria Assessments
  • Disparity and Bias Metrics
  • Group Metrics
• Understanding the Metrics
  • Preliminary Concepts
  • Bias Metrics
• Using the CLI
  • CLI use cases
    • Use with binary scores and csv
    • Use with continuous scores or risk scores
    • Use with binary scores from a database
  • CLI flags
    • --input   < path to .csv file >
    • --config   < path to config.yaml >
    • --output-folder <path to folder>
    • --create-tables
• Configurations
  • Define reference groups
  • Score threshholds
  • Choosing Metrics
  • Fairness threshold
  • Attribute columns
  • Project information
  • Databases
• Using Aequitas with Python
• Running the webapp locally
  • Load data
  • Customize report
  • Get ‘The Bias Report’
• Aequitas API
  • src.aequitas.group
  • src.aequitas.bias
  • src.aequitas.fairness
  • src.aequitas.plotting
  • src.aequitas.preprocessing
• Examples
  • COMPAS Analysis using Aequitas
    • Background
  • Pre-Aequitas: Exploring the COMPAS Dataset
  • Putting Aequitas to the task
    • Data Formatting
  • What biases exist in my model?
    • Aequitas Group() Class
    • What is the distribution of groups, predicted scores, and labels across my dataset?
    • What are bias metrics across groups?
    • How do I interpret biases in my model?
  • How do I visualize bias in my model?
    • Visualizing a single absolute group metric across all population groups
    • View group metrics for only groups over a certain size threshold
    • Visualizing multiple user-specified absolute group metrics across all population groups
    • Visualizing default absolute group metrics across all population groups
  • What levels of disparity exist between population groups?
    • Aequitas Bias() Class
    • How do I interpret calculated disparity ratios?
    • How does the selected reference group affect disparity calculations?
  • How do I visualize disparities in my model?
    • Visualizing disparities between groups in a single user-specified attribute for a single user-specified disparity metric
    • Visualizing disparities between all groups for a single user-specified disparity metric
    • Visualizing disparities between groups in a single user-specified attribute for default metrics
    • Visualizing disparities between groups in a single user-specified attribute for all calculated disparity metrics
    • Visualizing disparity between all groups for multiple user-specified disparity metrics
  • How do I assess model fairness?
    • Aequitas Fairness() Class
    • Group Level Fairness
    • How do I interpret parities?
    • Attribute Level Fairness
    • Overall Fairness
  • How do I visualize bias metric parity?
    • Visualizing parity of a single absolute group metric across all population groups
    • Visualizing all absolute group metrics across all population groups
  • How do I visualize parity between groups in my model?
    • Visualizing parity between groups in a single user-specified attribute for all calculated disparity metrics
    • Researcher Check: Could the unfairness I am seeing be related to small group sizes in my sample?
    • Visualizing parity between groups in a single user-specified attribute for all calculated disparity metrics
    • Visualizing parity between all groups for multiple user-specified disparity metrics
    • Visualizing parity between groups in multiple user-specified attributes
  • The Aequitas Effect