Bayesian spam filter

The Bayesian spam filter is a statistical filter used to classify unwanted e-mail messages ( spam ) using the naive Bayesian classifier .

functionality

E-mails are examined with the help of the Bayesian filter as follows: Characteristic words in an e-mail (event) indicate the property of being completely undesirable ( spam ). This statistical filtering , first introduced in 1998 by Sahami et al. proposed and popularized from 2002 by an influential article by Paul Graham , is intended to predict whether a message is spam.

The system is used by many programs for spam detection and is implemented, for example, in the e-mail programs Pegasus Mail , Opera Mail and Mozilla Thunderbird .

Statistical countermeasures are based on probability methods derived from Bayes' theorem . Bayesian filters are often organized in a "learning" (also "self-learning") manner and rely on word frequencies in classified emails already received by the user.

A Bayesian filter is trained by its user by dividing his e-mails into wanted and unwanted messages. The Bayesian filter now compiles a list of words that appear in unsolicited emails. For example, if the user has marked e-mails with the terms “sex” and “Viagra” as spam, all e-mails with these terms have a high probability of spam. Terms from desired e-mails such as “appointment” or “report” then lead in return to a downgrade of the negative rating. However, individual keywords are not sufficient; the sum of the ratings of the individual words is relevant.

The filter achieves high hit rates after just a short training session with just a few emails - even if training with several hundred emails from both categories is recommended for productive use. False positives are a risk for the user - e-mails that are incorrectly recognized as spam. This risk can be reduced by marking desired emails, but is particularly problematic for companies.

The senders of spam take countermeasures against Bayesian filters. Advertising messages are shown in images that the filter cannot examine. Suspicious terms are also deliberately spelled incorrectly (for example “V | agra” or “Va1ium”) or with interspersed spaces. However, the filter also rates HTML tags such as “img” and “src” negatively, so that images in e-mails are also rated with a higher probability of spam. Increasingly, random quotations from world literature (also in white letters or illegible as a meta tag) are inserted in order to mislead the statistical measures. However, this is not a successful strategy because randomly selected “harmless” terms or sentences have neither a particularly high nor a particularly low probability of spam, so they ultimately do not play a role.

A peculiarity in non-English-speaking countries arises from the fact that spam is mainly written in English. The probability of a Bayesian filter being hit is therefore likely to be higher in these countries, but there is also the risk that a desired English-language mail is incorrectly marked as spam.

Filtering based on statistics is a text classification. A number of researchers in applied linguistics , dealing with machine learning deal, have already addressed this problem. A further development in the field of e-mail processing is the Markov spam filter , in which not just individual words, but entire word chains and possible combinations are evaluated.

Mathematical basis

The Bayes' Theorem is

{\ displaystyle P (A | B) \; = \; {\ frac {P (B | A) \ cdot P (A)} {P (B)}},}

where the conditional probability of event A is given, provided that event B has occurred (before) . ${\ displaystyle P (A | B)}$

{\ displaystyle P ({\ text {search word occurs}} | {\ text {Spam}}) \,}

would be the probability that the relevant search term occurs in an email if it is a spam email, and vice versa

{\ displaystyle P ({\ text {Spam}} | {\ text {search word occurs}}) \,}

The probability of interest in this context that an e-mail is spam if it contains the relevant search term. According to the Bayesian formula above, this probability can now be calculated as follows:

{\ displaystyle P ({\ text {Spam}} | {\ text {Search word occurs}}) \; = \; {\ frac {P ({\ text {Search word occurs}} | {\ text {Spam} }) \ cdot P ({\ text {Spam}})} {P ({\ text {search word occurs}})}}.}

Web links

The mathematics in the Bayesian spam filter , the necessary tools and a calculation example

Individual evidence

↑ M. Sahami, S. Dumais, D. Heckerman, E. Horvitz: A Bayesian approach to filtering junk e-mail , AAAI'98 Workshop on Learning for Text Categorization., 1998
^ P. Graham: A Plan for Spam , August 2002.

[Sahami-1] M. Sahami, S. Dumais, D. Heckerman, E. Horvitz: A Bayesian approach to filtering junk e-mail , AAAI'98 Workshop on Learning for Text Categorization., 1998

[Graham-2] P. Graham: A Plan for Spam , August 2002.