Summary of metrics used in the evaluation of biosignature detection algorithms. The table includes the metric name, a nickname describing its utility, its specific use in the context of evaluating planets for biosignatures/bioindicators, and extreme cases illustrating the possible values and interpretations of each metric.
| Metric . | Nickname . | Utility . | Extreme cases . |
|---|---|---|---|
| Recall | Wasting metric | When small we are excluding promising planets from future observational campaigns, and wasting research opportunities. | |$\bullet$| Recall = 1: all planets with biosignatures/bioindicators are labelled as interesting. |$\bullet$| Recall = 0: no planets with biosignatures/bioindicators are labelled as interesting. |
| Precision | Time-saving metric | Helps to minimize the time spent studying planets without biosignatures/bioindicators. | |$\bullet$| Precision = 1: All planets labelled as interesting truly have biosignatures/bioindicators. |$\bullet$| Precision = 0: Any planet is labelled as interesting even without biosignatures/bioindicators.* |
| F1 Score | Discovery metric | Quantifies the discovery opportunity by balancing Precision and Recall. MAXIMIZE the opportunity to achieve a successful discovery. | |$\bullet$| F1 Score = 1: when Recall = Precision = 1, meaning the algorithm has perfect precision and Recall, successfully identifying all interesting planets without any false positives. |$\bullet$| F1 Score = 0: algorithm cannot classify any interesting planets correctly. ** |
| TNR | Confusion metric | Helps to ensure that planets without biosignatures/bioindicators are not mislabelled as interesting, thereby avoiding wasted resources on unpromising candidates. | |$\bullet$| TNR = 1: all non-interesting planets are correctly identified as not interesting. |$\bullet$| TNR = 0: all non-interesting planets are incorrectly labelled as interesting. |
| Hamming Loss | Error rate metric | It measures the fraction of incorrectly predicted molecules, indicating the algorithm’s deficiency in misclassifying or omitting the presence of molecules in general. | |$\bullet$| Hamming Loss = 0: All predicted labels (biosignatures/bioindicators) are correct. |$\bullet$| Hamming Loss = 1: All predicted labels (biosignatures/bioindicators) are incorrect. |
| Exact Match Ratio | Perfect match metric | Measures how well the algorithm classifies all biosignatures/bioindicators for planets correctly at the same time. This metric is very demanding as it requires perfect classification for all labels. Higher values indicate better performance. | |$\bullet$| Exact Match Ratio = 1: The algorithm correctly assigns all biosignatures/bioindicators to all planets. |$\bullet$| Exact Match Ratio = 0: The algorithm fails to assign the correct set of biosignatures/bioindicators to any planet. |
| Metric . | Nickname . | Utility . | Extreme cases . |
|---|---|---|---|
| Recall | Wasting metric | When small we are excluding promising planets from future observational campaigns, and wasting research opportunities. | |$\bullet$| Recall = 1: all planets with biosignatures/bioindicators are labelled as interesting. |$\bullet$| Recall = 0: no planets with biosignatures/bioindicators are labelled as interesting. |
| Precision | Time-saving metric | Helps to minimize the time spent studying planets without biosignatures/bioindicators. | |$\bullet$| Precision = 1: All planets labelled as interesting truly have biosignatures/bioindicators. |$\bullet$| Precision = 0: Any planet is labelled as interesting even without biosignatures/bioindicators.* |
| F1 Score | Discovery metric | Quantifies the discovery opportunity by balancing Precision and Recall. MAXIMIZE the opportunity to achieve a successful discovery. | |$\bullet$| F1 Score = 1: when Recall = Precision = 1, meaning the algorithm has perfect precision and Recall, successfully identifying all interesting planets without any false positives. |$\bullet$| F1 Score = 0: algorithm cannot classify any interesting planets correctly. ** |
| TNR | Confusion metric | Helps to ensure that planets without biosignatures/bioindicators are not mislabelled as interesting, thereby avoiding wasted resources on unpromising candidates. | |$\bullet$| TNR = 1: all non-interesting planets are correctly identified as not interesting. |$\bullet$| TNR = 0: all non-interesting planets are incorrectly labelled as interesting. |
| Hamming Loss | Error rate metric | It measures the fraction of incorrectly predicted molecules, indicating the algorithm’s deficiency in misclassifying or omitting the presence of molecules in general. | |$\bullet$| Hamming Loss = 0: All predicted labels (biosignatures/bioindicators) are correct. |$\bullet$| Hamming Loss = 1: All predicted labels (biosignatures/bioindicators) are incorrect. |
| Exact Match Ratio | Perfect match metric | Measures how well the algorithm classifies all biosignatures/bioindicators for planets correctly at the same time. This metric is very demanding as it requires perfect classification for all labels. Higher values indicate better performance. | |$\bullet$| Exact Match Ratio = 1: The algorithm correctly assigns all biosignatures/bioindicators to all planets. |$\bullet$| Exact Match Ratio = 0: The algorithm fails to assign the correct set of biosignatures/bioindicators to any planet. |
Even if the false positives are at their maximum, meaning all non-interesting planets are incorrectly labelled as interesting, the precision is not zero as long as there are true positives. This occurs because precision is calculated as the ratio of true positives to the sum of true positives and false positives. Thus, the presence of any correctly identified interesting planets ensures that precision remains above zero.
When TNR = 0, F1 Score is not necessarily very low if TP is high.
Summary of metrics used in the evaluation of biosignature detection algorithms. The table includes the metric name, a nickname describing its utility, its specific use in the context of evaluating planets for biosignatures/bioindicators, and extreme cases illustrating the possible values and interpretations of each metric.
| Metric . | Nickname . | Utility . | Extreme cases . |
|---|---|---|---|
| Recall | Wasting metric | When small we are excluding promising planets from future observational campaigns, and wasting research opportunities. | |$\bullet$| Recall = 1: all planets with biosignatures/bioindicators are labelled as interesting. |$\bullet$| Recall = 0: no planets with biosignatures/bioindicators are labelled as interesting. |
| Precision | Time-saving metric | Helps to minimize the time spent studying planets without biosignatures/bioindicators. | |$\bullet$| Precision = 1: All planets labelled as interesting truly have biosignatures/bioindicators. |$\bullet$| Precision = 0: Any planet is labelled as interesting even without biosignatures/bioindicators.* |
| F1 Score | Discovery metric | Quantifies the discovery opportunity by balancing Precision and Recall. MAXIMIZE the opportunity to achieve a successful discovery. | |$\bullet$| F1 Score = 1: when Recall = Precision = 1, meaning the algorithm has perfect precision and Recall, successfully identifying all interesting planets without any false positives. |$\bullet$| F1 Score = 0: algorithm cannot classify any interesting planets correctly. ** |
| TNR | Confusion metric | Helps to ensure that planets without biosignatures/bioindicators are not mislabelled as interesting, thereby avoiding wasted resources on unpromising candidates. | |$\bullet$| TNR = 1: all non-interesting planets are correctly identified as not interesting. |$\bullet$| TNR = 0: all non-interesting planets are incorrectly labelled as interesting. |
| Hamming Loss | Error rate metric | It measures the fraction of incorrectly predicted molecules, indicating the algorithm’s deficiency in misclassifying or omitting the presence of molecules in general. | |$\bullet$| Hamming Loss = 0: All predicted labels (biosignatures/bioindicators) are correct. |$\bullet$| Hamming Loss = 1: All predicted labels (biosignatures/bioindicators) are incorrect. |
| Exact Match Ratio | Perfect match metric | Measures how well the algorithm classifies all biosignatures/bioindicators for planets correctly at the same time. This metric is very demanding as it requires perfect classification for all labels. Higher values indicate better performance. | |$\bullet$| Exact Match Ratio = 1: The algorithm correctly assigns all biosignatures/bioindicators to all planets. |$\bullet$| Exact Match Ratio = 0: The algorithm fails to assign the correct set of biosignatures/bioindicators to any planet. |
| Metric . | Nickname . | Utility . | Extreme cases . |
|---|---|---|---|
| Recall | Wasting metric | When small we are excluding promising planets from future observational campaigns, and wasting research opportunities. | |$\bullet$| Recall = 1: all planets with biosignatures/bioindicators are labelled as interesting. |$\bullet$| Recall = 0: no planets with biosignatures/bioindicators are labelled as interesting. |
| Precision | Time-saving metric | Helps to minimize the time spent studying planets without biosignatures/bioindicators. | |$\bullet$| Precision = 1: All planets labelled as interesting truly have biosignatures/bioindicators. |$\bullet$| Precision = 0: Any planet is labelled as interesting even without biosignatures/bioindicators.* |
| F1 Score | Discovery metric | Quantifies the discovery opportunity by balancing Precision and Recall. MAXIMIZE the opportunity to achieve a successful discovery. | |$\bullet$| F1 Score = 1: when Recall = Precision = 1, meaning the algorithm has perfect precision and Recall, successfully identifying all interesting planets without any false positives. |$\bullet$| F1 Score = 0: algorithm cannot classify any interesting planets correctly. ** |
| TNR | Confusion metric | Helps to ensure that planets without biosignatures/bioindicators are not mislabelled as interesting, thereby avoiding wasted resources on unpromising candidates. | |$\bullet$| TNR = 1: all non-interesting planets are correctly identified as not interesting. |$\bullet$| TNR = 0: all non-interesting planets are incorrectly labelled as interesting. |
| Hamming Loss | Error rate metric | It measures the fraction of incorrectly predicted molecules, indicating the algorithm’s deficiency in misclassifying or omitting the presence of molecules in general. | |$\bullet$| Hamming Loss = 0: All predicted labels (biosignatures/bioindicators) are correct. |$\bullet$| Hamming Loss = 1: All predicted labels (biosignatures/bioindicators) are incorrect. |
| Exact Match Ratio | Perfect match metric | Measures how well the algorithm classifies all biosignatures/bioindicators for planets correctly at the same time. This metric is very demanding as it requires perfect classification for all labels. Higher values indicate better performance. | |$\bullet$| Exact Match Ratio = 1: The algorithm correctly assigns all biosignatures/bioindicators to all planets. |$\bullet$| Exact Match Ratio = 0: The algorithm fails to assign the correct set of biosignatures/bioindicators to any planet. |
Even if the false positives are at their maximum, meaning all non-interesting planets are incorrectly labelled as interesting, the precision is not zero as long as there are true positives. This occurs because precision is calculated as the ratio of true positives to the sum of true positives and false positives. Thus, the presence of any correctly identified interesting planets ensures that precision remains above zero.
When TNR = 0, F1 Score is not necessarily very low if TP is high.
This PDF is available to Subscribers Only
View Article Abstract & Purchase OptionsFor full access to this pdf, sign in to an existing account, or purchase an annual subscription.
