spamassassin-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Katz <>
Subject Re: yahoo X-YMail-OSG
Date Fri, 21 May 2010 21:43:33 GMT
Michael Scheidell mused:
>> would adding 1 point for each 1K of header length help?

J.D. Falk responded:
> Interesting idea!  I don't know the precise semantics of the
> contents of that header, but this certainly sounds possible.


I don't think this is efficient at all (I'm leaning on "no way in
hell"), but here's a REALLY UGLY implementation:

#    header BIG_HEADER       ALL =~ /(?=.{1024})/m
#    tflags BIG_HEADER       multiple  # DO NOT USE THIS
#    score  BIG_HEADER       0.001
#    meta   1KB_HEADER       MS_BIG_HEADER
#    score  1KB_HEADER       0.999

Side note: "(?=pattern)" is a zero-width positive look-ahead assertion
that theoretically limits the memory dedicated to this operation.  An
even less efficient but otherwise equivalent version could more simply
just say /.{1024}/m

While it "works" (it adds 1.000 for the first 1K and 0.001 for each
subsequent character, i.e. 1.024 for each subsequent 1K of header
length) it *should not be used*; CPU-efficiency aside, you'll see a
BIG_HEADER for EVERY character beyond 1023 in each header, which can
make for a pretty darn long summary in your headers and in the body's
content analysis details.  Also, it doesn't add additional 1KB_HEADER
hits for additional big headers.

Instead, I've added the following to my sandbox:

# Idea from Michael Scheidell of SECNAP on SA Users List 2010-05-08
header   SINGLE_HEADER_1K  ALL:raw =~
header   SINGLE_HEADER_2K  ALL:raw =~ /^(?=.{2048,3071}$)/m
header   SINGLE_HEADER_3K  ALL:raw =~ /^(?=.{3072,4095}$)/m
header   SINGLE_HEADER_4K  ALL:raw =~ /^(?=.{4096,5119}$)/m
header   SINGLE_HEADER_5K  ALL:raw =~ /^(?=.{5120})/m
describe SINGLE_HEADER_1K  A single header contains 1K-2K characters
describe SINGLE_HEADER_2K  A single header contains 2K-3K characters
describe SINGLE_HEADER_3K  A single header contains 3K-4K characters
describe SINGLE_HEADER_4K  A single header contains 4K-5K characters
describe SINGLE_HEADER_5K  A single header contains 5K+ characters

header   BIG_HEADERS_2K    ALL:raw =~ /^(?=.{2048,3071}$)/s
header   BIG_HEADERS_3K    ALL:raw =~ /^(?=.{3072,4095}$)/s
header   BIG_HEADERS_4K    ALL:raw =~ /^(?=.{4096,5119}$)/s
header   BIG_HEADERS_5K    ALL:raw =~ /^(?=.{5120})/s
tflags   BIG_HEADERS_2K    userconf  # unset trusted_networks can mess
this up
describe BIG_HEADERS_2K    Headers contain 2K-3K characters total
describe BIG_HEADERS_3K    Headers contain 3K-4K characters total
describe BIG_HEADERS_4K    Headers contain 4K-5K characters total
describe BIG_HEADERS_5K    Headers contain 5K+ characters total

I could have made each SINGLE_HEADER_xK rule 'tflags multiple' so that
it penalizes for multiple egregiously large headers, but I think the
BIG_HEADERS_xK rules better facilitate that.  I also suspect that the
SINGLE_HEADER_xK rules won't be as useful (which isn't to say either
will be useful ... we just have to wait for a few reasonably sized
masscheck corpora to see, which might take a while these days!).

View raw message