Skip to content

Auditioner

The Auditioner class is the main entry point for the Audition module. Users pass its constructor a database connection, information about the model groups to be evaluated, and a specification for a filter to prune the worst-performing models.

Other methods allow users to define more complex selection rules, list selected models, or plot results from the selection process.

logger #

Classes#

AuditionRunner #

__init__(self, config_dict, db_engine, directory=None) special #
run(self) #
validate(self) #

Auditioner #

Attributes#

average_regret_for_rules: dict property readonly #

Returns the average regret for each selection rule, over the specified list of train/test periods.

Returns:

Type Description
A dict with a key-value pair for each selection rule and the average regret for that rule. Structure

{'descriptive rule_name': .5}
metrics property readonly #
selection_rule_model_group_ids: dict property readonly #

Calculate the current winners for each selection rule and the most recent date

Returns:

Type Description
A dict with a key-value pair for each selection rule and the list of n model_group_ids that it selected. Structure

{'descriptive rule_name':[1,2,3]}
thresholded_model_group_ids: list property readonly #

The model group thresholder will have a varying list of model group ids depending on its current thresholding rules, this is a reference to whatever that current list is.

Returns:

Type Description
list

list of model group ids allowed by the current metric threshold rules

Methods#

__init__(self, db_engine, model_group_ids, train_end_times, initial_metric_filters, models_table=None, distance_table=None, directory=None, agg_type='worst', baseline_model_group_ids=None) special #

Filter model groups using a two-step process:

  1. Broad thresholds to filter out truly bad models
  2. A selection rule grid to find the best model groups over time for each of a variety of methods

This is achieved by creating a 'best distance' table, which functions like a denormalized 'model group/model/evaluations', storing for each model group/train end time/metric/parameter: 1. the raw evaluation value, 2. the distance of that evaluation metric from the best model group at that train time, 3. and the distance of the metric from the best model group the next train time

Each of the steps is computed based on the data in this table, and an iterative process of sending thresholding/selection configuration and viewing the results.

For step 1, the initial configuration is sent in this constructor (as 'initial_metric_filters', format detailed below), future iterations of this configuration are sent to 'update_metric_filters'.

For step 2, all configuration is sent to the object via 'register_selection_rule_grid', and its format is detailed in that method's docstring

Parameters:

Name Type Description Default
db_engine sqlalchemy.engine

A database engine with access to a results schema of a completed modeling run

 required
model_group_ids list

A large list of model groups to audition. No effort should be needed to pick 'good' model groups, but they should all be groups that could be used if they are found to perform well. They should also each have evaluations for any train end times you wish to include in analysis

 required
train_end_times list

A list of train end times that all of the given model groups contain evaluations for and that you want to be deemed important in the analysis

 required
initial_metric_filters list

A list of metrics to filter model groups on, and how to filter them. Each entry should be a dict of the format:

{
    'metric': 'string',
    'parameter': 'string',
    'max_below_best': .5,
    'threshold_value': .5
 }

metric (string): model evaluation metric, such as 'precision@'
parameter (string): model evaluation metric parameter,
    such as '300_abs'
max_below_best (float): The maximum value that the given metric
    can be below the best for a given train end time
threshold_value (float): The minimum value that the given metric can be
required
models_table string

The name of the results schema models table that you want to use. Will default to 'models', which is also the default in triage.

 None
distance_table string

The name of the 'best distance' table to use. Will default to 'best_distance', but this can be sent if you want to avoid clobbering the results from a prior analysis.

 None
agg_type string) Method for aggregating metric values (for instance, if there are multiple models at a given train_end_time with different random seeds). Can be

'mean', 'best', or 'worst' (the default)

 'worst'
baseline_model_group_ids list

An optional list of model groups for baseline models which will be included on all plots without being subject to filtering or included as candidate models from the selection process.

 None
plot_model_groups(self) #

Display model group plots, one of the below for each configured metric.

  1. A cumulative plot showing the effect of different worse-than-best thresholds for the given metric across the thresholded model groups.

  2. A performance-over-time plot showing the value for the given metric over time for each thresholded model group

plot_selection_rules(self) #

Plot data about the configured selection rules. The three plots outlined below are plotted for each metric.

We base a lot of this on the concept of the 'regret'. The regret refers to the difference in performance between a model group and the best model group for the next testing window if a selection rule is followed.

  1. A distance-next-time plot, showing the fraction of models worse then a succession of regret thresholds for each selection rule
  2. A regret-over-time plot for each selection rule
  3. A metric-over-time plot for each selection rule
register_selection_rule_grid(self, rule_grid, plot=True) #

Register a grid of selection rules

Parameters:

Name Type Description Default
rule_grid list

Groups of selection rules that share parameters. See documentation below for schema.

 required
plot

(boolean, defaults to True) Whether or not to plot the selection rules at this time.

 True

rules_grid is a list of dicts. Each dict, which defines a group, has two required keys: shared_parameters and selection_rules.

shared_parameters: A list of dicts, each with a set of parameters that are taken by all selection rules in this group.

For each of these shared parameter sets, the grid will create selection rules combining the set with all possible selection rule/parameter combinations.

This can be used to quickly combine, say, a variety of rules that all are concerned with precision at top 100 entities.

selection_rules: A list of dicts, each with:

  • A 'name' attribute that matches a selection rule in audition.selection_rules
  • Parameters and values taken by that selection rule. Values in list form are all added to the grid. If the selection rule has no parameters, or the parameters are all covered by the shared parameters in this group, none are needed here.

Each selection rule in the group must have all of its required parameters covered by the shared parameters in its group and the parameters given to it.

Refer to Selection Rules for available selection rules and their parameters. The exceptions are the first two arguments to each selection rule, 'df' and 'train_end_time'. These are contextual and thus provided internally by Audition.

Examples:

[{
    'shared_parameters': [
            {'metric': 'precision@', 'parameter': '100_abs'},
            {'metric': 'recall@', 'parameter': '100_abs'},
        ],
        'selection_rules': [
            {'name': 'most_frequent_best_dist',
                'dist_from_best_case': [0.1, 0.2, 0.3]},
            {'name': 'best_current_value'}
        ]
}]
save_result_model_group_ids(self, fname='results_model_group_ids.json') #
set_one_metric_filter(self, metric='precision@', parameter='100_abs', max_from_best=0.05, threshold_value=0.1) #

Set one thresholding metric filter If one wnats to update multiple filters, one should use update_metric_filters() instead.

Parameters:

Name Type Description Default
metric string

model evaluation metric such as 'precision@'

 'precision@'
parameter string

model evaluation parameter such as '100_abs'

 '100_abs'
max_from_best string

The maximum value that the given metric can be below the best for a given train end time

 0.05
threshold_value string

The thresold value that the given metric can be

 0.1
plot boolean, default True

Whether or not to also plot model group performance and thresholding details at this time.

 required
update_metric_filters(self, new_filters=None, plot=True) #

Update the thresholding metric filters

    !!! args
        new_filters (list): A list of metrics to filter model
            groups on, and how to filter them. This is an identical format to
            the list given to 'initial_metric_filters' in the constructor.
            Each entry should be a dict with the keys:

initial_metric_filters metric (string) -- model evaluation metric, such as 'precision@' parameter (string) -- model evaluation metric parameter, such as '300_abs' max_below_best (float) The maximum value that the given metric can be below the best for a given train end time threshold_value (float) The threshold value that the given metric can be plot (boolean, default True): Whether or not to also plot model group performance and thresholding details at this time.