spamassassin-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Bakhtiar <shashan...@hotmail.com>
Subject Re: I have some bad news
Date Wed, 17 Aug 2016 16:16:33 GMT

On Aug 17, 2016, at 3:43 AM, Matus UHLAR - fantomas <uhlar@fantomas.sk<mailto:uhlar@fantomas.sk>>
wrote:

On 16.08.16 20:06, Marc Perkel wrote:
What I'm doing is looking for fingerprints in email that intersect HAM and not in SPAM - which
would be a HAM result.
If it matches SPAM and does NOT match HAM - then it's SPAM.

The magic is in the NOT matching on the other side.

so, if mail matches both hammy and spammy tokens (or token sets), you don't
classify at all?


I guess what is confusing me (and I imagine others, as alluded to by Matus) is the fact that
you are describing a special condition of Bayes' probability theorem. You are testing two
variables (match SPAM and match HAM) (not matching is simply the negation of matching) thus
giving you four conditions:

1) SPAM  && HAM
2) SPAM   && ~HAM
3) ~SPAM && HAM
4) ~SPAM && ~HAM

Here is a great diagram to show the four probable conditions:
https://en.wikipedia.org/wiki/Bayes%27_theorem#/media/File:Bayes%27_Theorem_2D.svg<https://en.wikipedia.org/wiki/Bayes'_theorem#/media/File:Bayes'_Theorem_2D.svg>

So (if I am correct) Matus is asking what if condition 1 is true? How are you classifying
an email than? Which is often the state of most emails, and thus why the use of Naive Bayes
spam filtering, which generates a probability based on Bayes' probability theorem and is the
conventional methodology to date. A Rose by any other name....

Condition 4 is obvious it's nothing you have ever seen so classifying it anything other than
HAM would be a huge mistake (IMHO), and fully covered by the aforementioned theorem as the
probability of SPAM would (should) be 0. Same with Condition 3, obviously it never hits SPAM
so wether it matches HAM or not you're going to treat it as HAM anyway same as condition 4.

That leaves condition 2. Which (if I'm not mistaken) is "... it matches SPAM and does NOT
match HAM - then it's SPAM.". Which brings us back to Matus question, what if the email contains
a single HAM token? Two HAM tokens? This is exactly what Bayes' probability theorem is designed
for. All you are doing is defining a special condition in which the HAM probability is ZERO.

I think that's were I need to understand a bit more about what HAM means in this solution,
does getting a hit on HAM somehow negate it being SPAM completely? In other words if the email
contains some set of tokens that are SPAM, yet only one HAM token, that single HAM token makes
it not SPAM? If so, you have a long way to go in convincing me that this is a good solution.

So if I say to you, "Let's get some lunch" that's ham because spammers never say that, but
normal people do. So the way to test what "spammers never say" is to store what they do say
and see if it's NOT in the list. (Thus the infinite set)


Actually I get SPAM with that very set of tokes in it. If somehow the HAM rating of it overrides
the SPAM, I don't believe it would have a desirable effect.

I get plenty of:

"
Hay Shawn,

Hope you have time to do some lunch, click on this link and check out my new pictures!

Wannabe Phisher
"

Based on your example there's plenty of HAM and SPAM tokens in there, "Click on this link"
high probability of SPAM-e-ness, would it get HAMed based on "hope you have time to do lunch".
Or am I missing something?


Similarly, there's only so many ways to misspell viagra, and good email wouldn't have it spelled
wrong.

Does that make sense?


Again, what you are saying makes sense in that it is special condition of the probability
theory, What does not make sense is why would you not simply use the probability theory, that
already encompasses that condition?

--
Matus UHLAR - fantomas, uhlar@fantomas.sk<mailto:uhlar@fantomas.sk> ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Linux - It's now safe to turn on your computer.
Linux - Teraz mozete pocitac bez obav zapnut.


Mime
View raw message