spamassassin-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Mincy <j...@delphioutpost.com>
Subject Re: About upgrading
Date Sat, 09 Jan 2010 23:13:35 GMT
   From: Cecil Westerhof <Cecil@decebal.nl>
   Date: Sat, 09 Jan 2010 16:24:56 +0100
   
   Jeff Mincy <jeff@delphioutpost.com> writes:
   
   >    I upgraded from 3.0.4 to 3.2.5. I have the feeling that sa-learn takes
   >    more time with 3.2.5 as it took with 3.0.4. Can this be true?
   >    
   >    It is not a problem, because it is done by cron-tab, but I am just
   >    curious.
   >
   > You can use spamc -L spam/ham to learn messages.  Spamc -L is faster
   > than sa-learn.  The spamd daemon needs to be started with
   > --allow-tell.
   
   That is not really an answer on my question. ;-)

I doubt that bayes learning has slowed down significantly.
I would expect that choice of bayes_store_module, learning to
journal, whether auto expiration runs, and lock contention
matters more than the version.

   But it does not seem to be interesting in my situation.
   First my code has to grow from:
       sa-learn --${typeStr} ${HOME}/Maildir/.SpamDir.${dirStr}/cur/
   to:
       for i in ${HOME}/Maildir/.SpamDir.${dirStr}/cur/*; do
           spamc -L ${typeStr} <${i}
       done
   
   Which is not even enough, because I need to take care of the situation
   that the directory is empty and I need to implement code to show the
   messages delivered by sa-learn.

Oh.  You're learning all of the messages in a directory.  spamc -L is
faster than sa-learn for learning single messages because sa-learn is
a perl script that has to load Mail::SpamAssassin each time.  For a
large directory the slower startup of sa-learn is less of an issue.
sa-learn is fine for doing directories.

   Which a low level of spam it work, but if it becomes bigger, it does not
   work:
       date
       echo ${echoStr}
       sa-learn --${typeStr} ${HOME}/Maildir/.SpamDir.${dirStr}/cur/
       date
       for i in ${HOME}/Maildir/.SpamDir.${dirStr}/cur/*; do
           spamc -L ${typeStr} <${i}
       done
       echo learned in the new way
       date
   gives:
       za jan  9 16:09:25 CET 2010
       Increase
       Learned tokens from 0 message(s) (45 message(s) examined)
       za jan  9 16:09:40 CET 2010
       learned in the new way
       za jan  9 16:10:00 CET 2010
   
   So sa-learn takes 15 seconds and spamc -L 20 seconds. (And I need more
   code. Beside taking care of an empty directory, I also need to implement
   the feedback given by sa-learn.)
   
You learned tokens from 0 messages and looked at 45 messages.
You've already previously learned from those 45 messages, which is
just timing how fast it can do nothing.

   > You can try using bayes_learn_to_journal - and do a separate sa-learn
   > --sync job in cron.   Learning to the journal is faster.
   
   I'll look into that.
   
   
   > Also, What is the size of your database?   Maybe you are spending lots
   > of time doing expires or something.
   
   sa-learn --dump magic gives:
       0.000          0          3          0  non-token data: bayes db version
       0.000          0      57538          0  non-token data: nspam
       0.000          0      74876          0  non-token data: nham
       0.000          0     166338          0  non-token data: ntokens
       0.000          0 1257478501          0  non-token data: oldest atime
       0.000          0 1263049426          0  non-token data: newest atime
       0.000          0 1263049538          0  non-token data: last journal sync atime
       0.000          0 1263044805          0  non-token data: last expiry atime
       0.000          0    5529600          0  non-token data: last expire atime delta
       0.000          0       1868          0  non-token data: last expire reduction count
   
Your database has 166338 tokens which is larger than the default
bayes_expiry_max_db_size 150000.  The last expiration ran this morning
at 8:46.  You could try letting the bayes database get larger and turn
off bayes_auto_expire.  If you turn off bayes_auto_expire you'll have
to add something to cron to periodically expire tokens.
bayes_auto_expire is fine for lower volumes of email, but can get in
the way with higher volumes.
-jeff

Mime
View raw message