Database Dependencies

Preparing Experiment Results

Note: If you are familiar with the DSaPP 'results schema', you can skip this section and head to the Using the Auditioner section.

Audition expects to be able to read experiment metadata from a relational database. This includes information about models, what we call 'model groups', and evaluations.

The full experiment schema used by DSaPP post-modeling tools such as Audition is defined in the results-schema repository, and is automatically populated after a triage.Experiment. However, even without using those tools, you can populate these tables for other experiments.

Here's an overview of Audition's direct dependencies:

• triage_metadata.model_groups - Everything unique about a classifier model, except for its train date. We define this as:
  • model_group_id - An autogenerated integer surrogate key for the model group, used as a foreign key in other tables
  • model_type - The name of the class, e.g 'sklearn.ensemble.RandomForestClassifier'
  • hyperparameters - This is a dictionary (stored in the database as JSON) that describes how the class is configured (e.g. {'criterion': 'gini', 'min_samples_split': 2})
  • feature_list - A list of feature names (stored in the database as an array).
  • model_config - A catch-all for anything else you want to uniquely define as a model group, for instance different pieces of time config. Use of this column is not strictly necessary.
• triage_metadata.models - An instantiation of a model group at a specific training time. In production, our version of this table has quite a few column, but the only columns used by Audition are below. Note that the name of this table can be customized with the models parameter in the audition config or as an argument to Auditioner. If you reproduce this table, these three should be sufficient:
  • model_id - An autogenerated integer surrogate key for the model, used as a foreign key in the 'evaluations' table.
  • model_group_id - A foreign key to the model_groups table.
  • train_end_time - A timestamp that signifies when this model was trained.
• test_results.evaluations - This is where metrics (such as precision, recall) and their values for specific models get stored.
  • model_id - A foreign key to the models table
  • evaluation_start_time - The start of the time period from which testing data was taken from.
  • evaluation_end_time - The end of the time period from which testing data was taken from.
  • metric - The name of a metric, e.g. 'precision@' for thresholded precision. Audition needs some knowledge about what direction indicates 'better' results for this metric, so it wishes that the metric is one of the options in catwalk.ModelEvaluator.available_metrics. If you use a metric that is not available here, it will assume that greater is better.
  • parameter - A string that indicates any parameters for the metric. For instance, 100_abs indicates top-100 entities, while 5_pct indicates top 5 percentile of entities. These are commonly used for metrics like precision and recall to prioritize the evaluation of models to how they affect the actions likely to be taken as a result of the model.
  • value - A float that represents the value of the metric and parameters applied to the model and evaluation time.