mowl.evaluation

class mowl.evaluation.base.Evaluator(device='cpu')[source]

Bases: object

Abstract class for evaluation of models.

Parameters

  • class_embeddings (dict or gensim.models.keyedvectors.KeyedVectors) – Embeddings dictionary for ontology classes

  • testing_set (list of mowl.projection.Edge) – List of triples in the testing set.

  • eval_method (callable) – Function that computes the score of the predictions

  • relation_embeddings (dict or gensim.models.keyedvectors.KeyedVectors, optional) – Embeddings dictionary for ontology classes

  • training_set (list of mowl.projection.Edge, optional) – List of triples in the training set. If not set, filtered metrics will not be computed

  • head_entities (list of str) – Entities, that are the head of each triple, to be considered in the evaluation

  • filter_fn_head (callable, optional) – Criterion to filter the head entities

  • filter_fn_tail (callable, optional) – Criterion to filter the tail entities

class mowl.evaluation.base.EvaluationMethod(embeddings, embeddings_relation=None, device='cpu')[source]

Bases: Module

forward()[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mowl.evaluation.base.AxiomsRankBasedEvaluator(eval_method, axioms_to_filter=None, device='cpu')[source]

Bases: object

Abstract method for evaluating axioms in a rank-based manner. To inherit from this class, 3 methods must be defined (dee the corresponding docstrings for each of them).

Parameters

  • eval_method (function) – The evaluation method for the axioms.

  • axioms_to_filter (any, optional) – Axioms to be put at the bottom of the rankings. If the axioms are empty, filtered metrics will not be computed. The input type of this parameter will depend on the signature of the _init_axioms_to_filter method. Defaults to None.

  • device (str, optional) – Device to run the evaluation. Defaults to “cpu”.

property metrics

Metrics as a dictionary with string metric names as keys and metrics as values.

Return type

dict

property fmetrics

Filtered metrics as a dictionary with string metric names as keys and metrics as values.

Return type

dict

compute_axiom_rank(axiom)[source]

This function should compute the rank of a single axiom. This method will be used iteratively by the __call__ method. This method returns a 3-tuple: rank of the axiom, frank of the axiom and the possible achievable worst rank.

Parameters

axiom – Axiom of the type congruent to the eval_method signature.

Return type

(int, int, int)

class mowl.evaluation.base.CosineSimilarity(embeddings, embeddings_relation=None, method=None, device='cpu')[source]

Bases: EvaluationMethod

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mowl.evaluation.base.TranslationalScore(embeddings, embeddings_relation, method, device='cpu')[source]

Bases: EvaluationMethod

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mowl.evaluation.rank_based.RankBasedEvaluator(class_index_emb, relation_index_emb, testing_set, eval_method, training_set, head_entities, tail_entities, device)[source]

Bases: Evaluator

This class corresponds to evaluation based on ranking. That is, for each testing triple \((h,r,t)\), scores are computed for triples \((h,r,t')\) for all possible \(t'\). After that, the ranking of the testing triple \((h,r,t)\) score is obtained.

Parameters

  • class_index_emb (dict(str, np.array)) – dictionary of classes and their embeddings

  • relation_index_emb (dict(str, np.array)) – dictionary of relations and their embeddings

  • testing_set (list(mowl.projection.edge.Edge)) – Set of triples that are true positives.

  • eval_method (function) – evaluation method score the triples

  • training_set (list(mowl.projection.edge.Edge)) – Set of triples that are true positives but exist in the training set. This is used to compute filtered metrics.

  • head_entities (list(str)) – List of entities that are used as head entities in the testing set.

  • tail_entities (list(str)) – List of entities that are used as tail entities in the testing set.

  • device (str) – Use cpu or cuda

class mowl.evaluation.rank_based.ModelRankBasedEvaluator(model, device='cpu', eval_method=None)[source]

Bases: RankBasedEvaluator

This class corresponds to evaluation based on ranking, where the embedding information of an entity is enclosed in some model.

Parameters

  • model (mowl.base_models.EmbeddingModel) – The model to be evaluated.

  • device (str, optional) – The device to be used for evaluation. Defaults to ‘cpu’.

  • eval_method (callable, optional) – The method used for the evaluation. If None, the method will be set to self.eval_method. Defaults to None.

class mowl.evaluation.rank_based.EmbeddingsRankBasedEvaluator(class_embeddings, testing_set, eval_method_class, score_func=None, training_set=None, relation_embeddings=None, head_entities=None, tail_entities=None, device='cpu')[source]

Bases: RankBasedEvaluator

This class corresponds to evaluation based on raking where the embedding information are just vectors and are not part of a model.

Parameters

  • class_embeddings (dict(str, numpy.ndarray)) – The embeddings of the classes.

  • testing_set (list(mowl.projection.edge.Edge)) – Set of triples that are true positives.

  • eval_method_class (mowl.evaluation.base.EvaluationMethod) – Class that contains placeholders for class and relation embeddings.

  • score_func – The function used to compute the score. Defaults to None. This function will be inserted into the eval_method_class.

  • training_set (list(mowl.projection.edge.Edge)) – Set of triples that are true positives but exist in the training set. This is used to compute filtered metrics.

  • relation_embeddings (dict(str, numpy.ndarray), optional) – The embeddings of the relations. Defaults to None.

  • head_entities (list(str)) – List of entities that are used as head entities in the testing set.

  • tail_entities (list(str)) – List of entities that are used as tail entities in the testing set.

  • device (str) – Use cpu or cuda