Conclusion

In this analysis, we tried to understand how predictions for email spam probability are made and especially how scammers can improve the contents of their emails, so they pass spam filters. By looking at Feature Importance, we found out which words or characters are most relevant for the different classification algorithms random forest and SVM. Both models agree that charExclamation, i.e. the frequency of the exclamation mark (!), remove and free are essential for prediction. In Relationship between Target & Features, we analyzed the direction in which the features impact the prediction. Most features push the prediction to spam, while personalized words like the recipients name george or the company he works at hp decrease the spam probability. Scammers should avoid creating feelings of urgency by overusing exclamation marks and words in capital letters. In Interactions, we discovered that interactions between features have no crucial impact on the prediction and the choosing the right amount of word/character occurrences can be done independently of other features. By approximating the models with surrogate trees in Global Surrogate Models, we confirmed that personalization, as well as decreasing the number of finance related words, decreases the spam probability. Those findings could also be seen when analyzing specific emails with the help of Shapley values in Local Methods. Finally, feature importance and the relationship between the target and the features based on Shapley values was found to be consistent with prior findings in Comparison of Interpretable Methods. All those findings might help scammers to write emails, which are not obviously spam.

Even though this analysis was based on the scammers perspective, understanding spam filters is essential for email providers as well, to not discriminate unfairly. How spam filters work might have a bigger impact on society as one might think. As AlgorithmWatch, a non-profit research organization that evaluates algorithmic decision-making processes, showed an internship application does not pass outlook's spam filter when the word Nigeria is included (see this article){target=_blank}. Interpretable Machine Learning is a substantial method for understanding hidden bias and for bringing transparency into black box models and more research needs to be done.