ModelSelector
- class medigan.select_model.model_selector.ModelSelector(config_manager: Optional[medigan.config_manager.ConfigManager] = None)[source]
Bases:
object
ModelSelector class: Given a config dict, gets, searches, and ranks matching models.
- Parameters
config_manager (ConfigManager) – Provides the config dictionary, based on which models are selected and compared.
- config_manager
Provides the config dictionary, based on which models are selected and compared.
- Type
- model_selection_dicts
Contains a dictionary for each model id that consists of the model_id and the selection config of that model
- Type
list
Methods Summary
find_matching_models_by_values
(values[, ...])Search for values (and keys) in model configs and return a list of each matching ModelMatchCandidate.
find_models_and_rank
(values[, ...])Search for values (and keys) in model configs, rank results and return sorted list of model dicts.
get_models_by_key_value_pair
(key1, value1[, ...])Get and return a list of model_id dicts that contain the specified key value pair in their selection config.
get_selection_criteria_by_id
(model_id[, ...])Get and return the selection config dict for a specific model_id.
get_selection_criteria_by_ids
([model_ids, ...])Get and return a list of selection config dicts for each of the specified model_ids.
get_selection_keys
([model_id])Get and return all first level keys from the selection config dict for a specific model_id.
get_selection_values_for_key
(key[, model_id])Get and return the value of a specified key of the selection dict in the config for a specific model_id.
rank_models_by_performance
([model_ids, ...])Rank model based on a provided metric and return sorted list of model dicts.
recursive_search_for_values
(search_dict, ...)Do a recursive search to match values in the search_dict with values (and keys) in a model's config.
Methods Documentation
- find_matching_models_by_values(values: list, target_values_operator: str = 'AND', are_keys_also_matched: bool = False, is_case_sensitive: bool = False) list [source]
Search for values (and keys) in model configs and return a list of each matching ModelMatchCandidate.
Uses self.recursive_search_for_values to recursively populate each ModelMatchCandidate with MatchedEntry instances. After populating, each ModelMatchCandidate is evaluated based on the provided target_values_operator and values list using ModelMatchCandidate.check_if_is_match.
- Parameters
values (list) – list of values used to search and find models corresponding to these values
target_values_operator (str) – the operator indicating the relationship between values in the evaluation of model search results. Should be either “AND”, “OR”, or “XOR”.
are_keys_also_matched (bool) – flag indicating whether, apart from values, the keys in the model config should also be searchable
is_case_sensitive (bool) – flag indicating whether the search for values (and) keys in the model config should be case-sensitive.
- Returns
a list of ModelMatchCandidate class instances each of which was successfully matched against the search values.
- Return type
list
- find_models_and_rank(values: list, target_values_operator: str = 'AND', are_keys_also_matched: bool = False, is_case_sensitive: bool = False, metric: str = 'SSIM', order: str = 'asc') list [source]
Search for values (and keys) in model configs, rank results and return sorted list of model dicts.
- Parameters
values (list) – list of values used to search and find models corresponding to these values
target_values_operator (str) – the operator indicating the relationship between values in the evaluation of model search results. Should be either “AND”, “OR”, or “XOR”.
are_keys_also_matched (bool) – flag indicating whether, apart from values, the keys in the model config should also be searchable
is_case_sensitive (bool) – flag indicating whether the search for values (and) keys in the model config should be case-sensitive.
metric (str) – The key in the selection dict that corresponds to the metric of interest
order (str) – the sorting order of the ranked results. Should be either “asc” (ascending) or “desc” (descending)
- Returns
a list of the searched and matched model dictionaries containing metric and model_id, sorted by metric.
- Return type
list
- get_models_by_key_value_pair(key1: str, value1: str, is_case_sensitive: bool = False) list [source]
Get and return a list of model_id dicts that contain the specified key value pair in their selection config.
The key param can contain ‘.’ (dot) separations to allow for retrieval of nested config keys such as ‘execution.generator.name’. If key1 points to a list, any value in the list that matches value1` is accepted.
- Parameters
key1 (str`) – The key in the selection dict
value1 (str) – The value in the selection dict that corresponds to key1
is_case_sensitive (bool) – flag to evaluate keys and values with case sensitivity if set to True
- Returns
a list of the dictionaries each containing a model’s model_id and the found key-value pair in the models config
- Return type
list
- get_selection_criteria_by_id(model_id: str, is_model_id_removed: bool = True) dict [source]
Get and return the selection config dict for a specific model_id.
- Parameters
model_id (str) – The generative model’s unique id
is_model_id_removed (bool) – flag to to remove the model_id from first level of each dictionary.
- Returns
a dictionary corresponding to the selection config of a model
- Return type
dict
- get_selection_criteria_by_ids(model_ids: Optional[list] = None, are_model_ids_removed: bool = True) list [source]
Get and return a list of selection config dicts for each of the specified model_ids.
- Parameters
model_ids (list) – A list of generative models’ unique ids
are_model_ids_removed (bool) – flag to remove the model_ids from first level of dictionary.
- Returns
a list of dictionaries each corresponding to the selection config of a model
- Return type
list
- get_selection_keys(model_id: Optional[str] = None) list [source]
Get and return all first level keys from the selection config dict for a specific model_id.
- Parameters
model_id (str) – The generative model’s unique id
- Returns
a list containing the keys as strings of the selection config of the model_id.
- Return type
list
- get_selection_values_for_key(key: str, model_id: Optional[str] = None) list [source]
Get and return the value of a specified key of the selection dict in the config for a specific model_id.
The key param can contain ‘.’ (dot) separations to allow for retrieval of nested config keys such as ‘execution.generator.name’
- Parameters
key (str) – The key in the selection dict
model_id (str) – The generative model’s unique id
- Returns
a list of the values that correspond to the key in the selection config of the model_id.
- Return type
list
- rank_models_by_performance(model_ids: Optional[list] = None, metric: str = 'SSIM', order: str = 'asc') list [source]
Rank model based on a provided metric and return sorted list of model dicts.
The metric param can contain ‘.’ (dot) separations to allow for retrieval via nested metric config keys such as ‘downstream_task.CLF.accuracy’. If the value found for the metric key is of type list, then the largest value in the list is used for ranking if order is descending, while the smallest value is used if order is ascending.
- Parameters
model_ids (list) – only evaluate the model_ids in this list. If none, evaluate all available model_ids
metric (str) – The key in the selection dict that corresponds to the metric of interest
order (str) – the sorting order of the ranked results. Should be either “asc” (ascending) or “desc” (descending)
- Returns
a list of model dictionaries containing metric and model_id, sorted by metric.
- Return type
list
- recursive_search_for_values(search_dict: dict, model_match_candidate: medigan.select_model.model_match_candidate.ModelMatchCandidate)[source]
Do a recursive search to match values in the search_dict with values (and keys) in a model’s config.
The provided ModelMatchCandidate instance is recursively populated with MatchedEntry instances. A MatchedEntry instance contains a key-value pair found in the model’s config that matches with one search term of interest.
The search terms of interest are stored in ModelMatchCandidate.target_values. The model’s selection config is provided in the ‘search_dict’.
To traverse the search_dict, the value for each key in the search_dict is retrieved.
If that value is of type dictionary, the function calls itself with that nested dictionary as new search_dict.
If that value is of type list, each value in the list is compared with each search term of interest in ModelMatchCandidate.target_values.
If the value of the search_dict is of another type (i.e. str), it is compared with each search term of interest in ModelMatchCandidate.target_values.
- Parameters
search_dict (dict) – contains keys and values from a model’s config that are matched against a set of search values.
model_match_candidate (ModelMatchCandidate) – a class instance representing a model to be prepared for evaluation (populated with MatchedEntry objects), as to whether it is a match given its search values (self.target_values).
- Returns
a ModelMatchCandidate class instance that has been populated with MatchedEntry class instances.
- Return type
list