Mailing-List: contact users-help@spamassassin.apache.org; run by ezmlm
Precedence: bulk
To: SA Mailing list <users@spamassassin.apache.org>
From: Rob McEwen <rob@invaluement.com>
Subject: potential new SA feature: Direct DNS Querying Per DNSBL Zone
Message-ID: <10893381-b523-ac10-8d2b-8ec53895f9a5@invaluement.com>
Date: Wed, 15 Nov 2017 12:03:58 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.4.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------4A9E70356A0F4CD811946222"
Content-Language: en-US
archived-at: Wed, 15 Nov 2017 17:04:15 -0000

This is a multi-part message in MIME format.
--------------4A9E70356A0F4CD811946222
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

RE: potential new SA feature: Direct DNS Querying Per DNSBL Zone

A couple of months ago, I commissioned the development of a potential 
new feature in SA. The code is mostly already written, but has at least 
one significant bug. Therefore, it is currently in the SA Bugzilla – Bug 
7477

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477

SUMMARY:

(1) (optionally!) on a per-DNSBL basis, DNS queries to a particular 
DNSBL can be set to DIRECTLY query a particular designated DNS server 
(individually set for that particular DNSBL)
--OR--
(2) (optionally!) on a per-DNSBL basis, DNS queries to a particular 
DNSBL can be set to DIRECTLY query that particular DNSBL's authoritative 
DNS server

(in both cases, bypassing the regular DNS "middle man")

ALSO: There is a cache component so that these DNSBL queries can be 
cached. DNSBLs *almost* don't need caching since most of them have such 
extremely short TTLs - HOWEVER - the problem with not caching is that 
there can often be bursts of *many* DNS lookups due to MANY similar 
spams being sent in close succession - THEREFORE - this caching 
component is critically important.

Why is this "Direct DNS Querying Per DNSBL Zone" feature needed/important?

(1) There have already been a few recent discussion threads on the SA 
lists where users have complained about their DNSBL lookups not working 
anymore - and then it was discovered that unknown or unwanted DNS 
settings (or setting changes) - caused major problems. This feature 
would enable more SA admins to have more "set it and forget it" success.

(2) Some DESIRE than their main DNS queries (for their server that is 
running SA) default to particular 3rd party DNS servers, such as 
Google's DNS servers - yet they need particular DNSBL queries to go 
directly to the DNSBL's authoritative DNS servers (and they may not even 
be aware of that need!). Yes, this can be done with a local DNS server 
combined with forwarders and conditional forwarders - HOWEVER...

(3) Some SA users do not have total control over their servers, and 
sometimes their DNS infrastructure and/or their DNS settings on their 
servers - CHANGE - without their knowledge or will. This was a primary 
motivation for my funding of this new feature (that is almost 
completed). While running the invaluement anti-spam blacklists - I've 
discovered that a LARGE percent of subscribers are continually having 
their DNS settings switched to either using Google's or OpenDNS's or 
their ISP's dns servers.  Then their queries to invaluement start to 
fail as we're not able to grant them access permissions from those IPs - 
nor are even able to distinguish such queries from others' queries. And 
even many of the (so-called) "free" DNSBLs have this same problem with 
their medium-sized customers who have moved into their paid access 
system AND are also doing directly queries (not RSYNC'ing the data). In 
fact, this problem is epidemic. Does it impact everyone? No. Does it 
impact you? Probably not. If you're reading this, you are probably one 
of the SA admins who DOES have more granularity of control and you 
probably already set up your own locally-hosted DNS server - and this 
all might sound like a solution looking for a problem... but then again, 
there have been those recent pesky SA list discussions that might not 
have existed if this feature were already implemented! And I see these 
issues FREQUENTLY with invaluement subscribers, especially with the 
numerous SA installations where the systems administrators didn't have 
as much control or knowledge as your average SA list participant. In 
many of these situations, the SA admin is a general IT guy who wears 100 
different hats besides being the email and spam filtering admin. That 
admin is typically pulled a million different directions and would 
GREATLY benefit from a feature that allows MORE "set it and forget it". 
Forgive that person for not having as much SA and DNS knowledge stored 
in his brain as you probably have!

Also, again, if this is accepted into official SA code, it will ALWAYS 
be an optional feature - that only impacts those DNSBLs for which it was 
specifically set up - other queries and other DNSBL lookups will 
continue to work as they always have. If you don't need or like this 
feature - just don't go out of your way to turn it on! Furthermore, this 
is NOT an intended replacement for running your own DNS caching server 
and pointing your SA queries to that. That is still the ideal/desired 
setup. Instead, this is for those who are unable to do that (or unable 
to do that reliably or without unexpected changes happening 
occasionally) AND it can also act as a sort of "insurance policy" to 
make sure that certain DNSBL queries continue to operate as intended, 
even if the DNS infrastructure changes unexpectedly or malfunctions.

If anyone is interested in this feature and is willing/able to help get 
this completed - that would be greatly appreciated!

NOTE: Once this feature is completed and bug free, it will then require 
the approval of the Apache Software Foundation (or something like that - 
I don't really understand their inner workings very well) - before it 
can be officially adopted. However, already, it has significant support 
from multiple important people! Therefore, I have high hopes that it 
will eventually earn this adoption, but that is not a guarantee and I 
have no control or say in that decision. But it makes the most sense to 
FIRST make this potential SA addition bug-free and outstanding - then 
cross that other bridge when we get there.

Thanks to anyone who is willing/able to help with this. Please let me 
know if you have any questions. I'm also considering putting some kind 
of modest "bounty" on this - to fund anyone's efforts that either fix 
the remaining bugs, or at least make significant and measurable progress 
to that end - send me a private message off-list if that interests you! 
(I would do this myself, but Perl "looks like Greek" to me!)

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477

-- 
Rob McEwen
https://www.invaluement.com


--------------4A9E70356A0F4CD811946222
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>RE: potential new SA feature: Direct DNS Querying Per DNSBL Zone</p>
    <p>A couple of months ago, I commissioned the development of a
      potential new feature in SA. The code is mostly already written,
      but has at least one significant bug. Therefore, it is currently
      in the SA Bugzilla – Bug 7477<br>
    </p>
    <p><a class="moz-txt-link-freetext" href="https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477">https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477</a></p>
    <p>SUMMARY:</p>
    <p>(1) (optionally!) on a per-DNSBL basis, DNS queries to a
      particular DNSBL can be set to DIRECTLY query a particular
      designated DNS server (individually set for that particular DNSBL)<br>
      --OR--<br>
      (2) (optionally!) on a per-DNSBL basis, DNS queries to a
      particular DNSBL can be set to DIRECTLY query that particular
      DNSBL's authoritative DNS server</p>
    <p>(in both cases, bypassing the regular DNS "middle man")<br>
    </p>
    <p>ALSO: There is a cache component so that these DNSBL queries can
      be cached. DNSBLs *almost* don't need caching since most of them
      have such extremely short TTLs - HOWEVER - the problem with not
      caching is that there can often be bursts of *many* DNS lookups
      due to MANY similar spams being sent in close succession -
      THEREFORE - this caching component is critically important.</p>
    <p>Why is this "Direct DNS Querying Per DNSBL Zone" feature
      needed/important?</p>
    <p>(1) There have already been a few recent discussion threads on
      the SA lists where users have complained about their DNSBL lookups
      not working anymore - and then it was discovered that unknown or
      unwanted DNS settings (or setting changes) - caused major
      problems. This feature would enable more SA admins to have more
      "set it and forget it" success.</p>
    <p>(2) Some DESIRE than their main DNS queries (for their server
      that is running SA) default to particular 3rd party DNS servers,
      such as Google's DNS servers - yet they need particular DNSBL
      queries to go directly to the DNSBL's authoritative DNS servers
      (and they may not even be aware of that need!). Yes, this can be
      done with a local DNS server combined with forwarders and
      conditional forwarders - HOWEVER...<br>
    </p>
    <p>(3) Some SA users do not have total control over their servers,
      and sometimes their DNS infrastructure and/or their DNS settings
      on their servers - CHANGE - without their knowledge or will. This
      was a primary motivation for my funding of this new feature (that
      is almost completed). While running the invaluement anti-spam
      blacklists - I've discovered that a LARGE percent of subscribers
      are continually having their DNS settings switched to either using
      Google's or OpenDNS's or their ISP's dns servers.  Then their
      queries to invaluement start to fail as we're not able to grant
      them access permissions from those IPs - nor are even able to
      distinguish such queries from others' queries. And even many of
      the (so-called) "free" DNSBLs have this same problem with their
      medium-sized customers who have moved into their paid access
      system AND are also doing directly queries (not RSYNC'ing the
      data). In fact, this problem is epidemic. Does it impact everyone?
      No. Does it impact you? Probably not. If you're reading this, you
      are probably one of the SA admins who DOES have more granularity
      of control and you probably already set up your own locally-hosted
      DNS server - and this all might sound like a solution looking for
      a problem... but then again, there have been those recent pesky SA
      list discussions that might not have existed if this feature were
      already implemented! And I see these issues FREQUENTLY with
      invaluement subscribers, especially with the numerous SA
      installations where the systems administrators didn't have as much
      control or knowledge as your average SA list participant. In many
      of these situations, the SA admin is a general IT guy who wears
      100 different hats besides being the email and spam filtering
      admin. That admin is typically pulled a million different
      directions and would GREATLY benefit from a feature that allows
      MORE "set it and forget it". Forgive that person for not having as
      much SA and DNS knowledge stored in his brain as you probably
      have!<br>
    </p>
    <p>Also, again, if this is accepted into official SA code, it will
      ALWAYS be an optional feature - that only impacts those DNSBLs for
      which it was specifically set up - other queries and other DNSBL
      lookups will continue to work as they always have. If you don't
      need or like this feature - just don't go out of your way to turn
      it on! Furthermore, this is NOT an intended replacement for
      running your own DNS caching server and pointing your SA queries
      to that. That is still the ideal/desired setup. Instead, this is
      for those who are unable to do that (or unable to do that reliably
      or without unexpected changes happening occasionally) AND it can
      also act as a sort of "insurance policy" to make sure that certain
      DNSBL queries continue to operate as intended, even if the DNS
      infrastructure changes unexpectedly or malfunctions.<br>
    </p>
    If anyone is interested in this feature and is willing/able to help
    get this completed - that would be greatly appreciated!<br>
    <br>
    NOTE: Once this feature is completed and bug free, it will then
    require the approval of the Apache Software Foundation (or something
    like that - I don't really understand their inner workings very
    well) - before it can be officially adopted. However, already, it
    has significant support from multiple important people! Therefore, I
    have high hopes that it will eventually earn this adoption, but that
    is not a guarantee and I have no control or say in that decision.
    But it makes the most sense to FIRST make this potential SA addition
    bug-free and outstanding - then cross that other bridge when we get
    there.<br>
    <br>
    Thanks to anyone who is willing/able to help with this. Please let
    me know if you have any questions. I'm also considering putting some
    kind of modest "bounty" on this - to fund anyone's efforts that
    either fix the remaining bugs, or at least make significant and
    measurable progress to that end - send me a private message off-list
    if that interests you! (I would do this myself, but Perl "looks like
    Greek" to me!)<br>
    <br>
    <a class="moz-txt-link-freetext" href="https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477">https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477</a>
    <pre class="moz-signature" cols="72">-- 
Rob McEwen
<a class="moz-txt-link-freetext" href="https://www.invaluement.com">https://www.invaluement.com</a>


</pre>
  </body>
</html>

--------------4A9E70356A0F4CD811946222--