Mastering Model Evaluation: Key Metrics for Machine Learning Success

The majority effort of a typical Data Analyst or a Data Scientist revolves around Model Evaluation and refinement. During my experimentation with model building, I used multiple metrics to understand the accuracy of my model.

Models specific to classification problems utilize a confusion matrix. The confusion matrix helps one to compare the predicted values against labels like True Negative, True Positive, False Negative, and False Positive. The Type one error is denoted by False Positives (FP) and the Type two error is denoted by False Negatives.

Source: https://www.analyticsvidhya.com/blog/2020/04/confusion-matrix-machine-learning/

The above cost matrix denotes the cost for the wrong prediction of disease. The highest cost occurs when a sick person is classified as healthy. As the impact due to wrong classification is very high in terms of lack of awareness leading to a lack of timely medical treatments, model accuracy needs to be carefully monitored and refined. The need for lack of TN in terms of critical applications like healthcare, and defence is highly essential. The cost matrix is multiplied by the confusion matrix to understand the actual impact.

The below image denotes the metrics used in a classification problem:

PRECISION

Precision is the ratio of TP /(TP+FP). This metric represents the correctly identified Positives as compared to the overall sum of positively classified data points.The application of Precision parameter is high in scenarios where the cost of False Positives are extremely high.

Application	Impact	Domain
Medical Diagnostics: A false positive result for a cancer test	It shall  lead to unnecessary harmful treatment and emotional distress to the falsely diagnosed individual.	Healthcare
Fraud Analytics: False classification of customer transactions	Low precision denotes that a large number of right transactions are classified as fraud,which denotes inconvenience and lack of trust from customers	Finance
Wrong Product/Movie Recommendation	Low precision denotes wasted efforts in retaining a customer for a subscription renewal at a platform like Netflix	Entertainment

Recall

Recall is the ratio of is the ratio of TP /(TP+FN). This parameter focuses on Fn too.The lower the recall,the higher the FN. Multiple methods like hyperparameter tuning could be performed to reduce the percentage of FN.

Application	Impact	Domain
Product Failure Predictions	Low recall leads to higher possibility of harm or accidents to customers using products	Product Safety
Targeted Advertising: Marketing Analytics	Low precision denotes that a huge effort and money is spent on targeting wrong customers for email campaigns	Digital Marketing
Viewer Churn Prediction	Low precision denotes wasted efforts in retaining a customer for a subscription renewal at a platform like Netflix	Entertainment

F1 Score

F1 score is the weighted average of Recall and Precision.Hence,it gives a broader view into the model performance.

AUC Score

An AUC score denotes the probability of right classification into one of the classes.Higher AUC denotes that model will be able to rightly classify a randomly chosen data point. AUC of less than or equal to 0.5 denotes a badly performing model. AUC >0.5 is somewhat acceptable. But a perfect score of 1 may suggest overfitting.Below diagram denotes the AUC for each iteration of cross validation.AUC is highly used when there is imbalance in data set.

RMSE

RMSE represents the square of the difference between predicted and actual. The final value is then passed through the square root function. RMSE is generally used in terms of regressor trees or time series. The below image denotes a high RMSE and as the best param found is 1,it may denote the underfitting of the model.

Examining the core of model evaluation metrics provides us with a road map to help us navigate the challenging field of machine learning. Every metric provides a way to improve and optimize our models, from the accuracy and lucidity of confusion matrices to the subtle insights of AUC scores and the fair-minded viewpoint of the F1 score. These instruments are more than just mathematical constructions; they serve as compass points for us when we apply models that are accurate, socially conscious, and useful in a variety of fields. We are reminded of the unique fusion of creativity and analysis that defines this ever-evolving area as we use these measures to evaluate and improve our machine learning models, motivating us to keep pushing the envelope of what is possible with data.