Mailing-List: contact sysadmins-help@spamassassin.apache.org; run by ezmlm
Precedence: bulk
Reply-To: sysadmins@spamassassin.apache.org
Subject: Fwd: Re: Problem with 72_scores.cf generation
References: <52cd20a1-c03c-8607-6766-d5b2fdb8b018@mcgrail.com>
To: sysadmins@spamassassin.apache.org
From: "Kevin A. McGrail" <kevin.mcgrail@mcgrail.com>
Message-ID: <0c36cf34-a139-bd39-284d-2e73e09c3c17@mcgrail.com>
Date: Tue, 4 Jul 2017 23:54:11 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <52cd20a1-c03c-8607-6766-d5b2fdb8b018@mcgrail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Wed, 05 Jul 2017 03:53:58 -0000

Resending without the screen shot...


Hi John,

Thanks.  We spent Friday as well trying to identify what's going on and 
I worked with Crashplan to restore the data I had for the old solaris 
box.  It was in an old format because the solaris client was deprecated 
so it was a lot of juggling.

In any case, I will A) look at this information again.  and B) I ran 
crashplan on the ASF servers because ASF Infra doesn't do backups.  I 
configure it to retain a pretty insane (unlimited) number of revisions 
as an anti-malware protection.  This means you can go in and restore a 
specific version of 72_scores.cf

(removed screenshot, too large and mailing list rejected)

I put the credentials in SVN for the ASF crashplan and happy to help 
walk you through things to see if I have a backup of the file you want 
to compare.

Regards,
KAM

On 7/4/2017 12:59 PM, Dave Jones wrote:
> Kevin,
> I have spent about 5 hours this morning trying to track down the 
> 72_scores.cf generation problem.  I haven't pinpointed the problem yet 
> but here's what I have found so far:
>
> NOTE: su - automc for proper paths below.
>
> 1. ~/svn/masses/rule-update-score-gen/generate-new-scores.sh is the 
> script in question
>
> https://svn.apache.org/viewvc/spamassassin/trunk/masses/rule-update-score-gen/generate-new-scores.sh?revision=1798589&view=markup 
>
>
> Line 271 runs "runGA"
>
> 2. runGA creates 
> ~/tmp/generate-new-scores/trunk-new-rules-set0/masses/gen-set0-5-5.0-6000-ga/scores
>
> This 'scores' file has 345 scores in it.  I wish we had a copy of this 
> file from mid March to see if it also had around the same number of 
> scores to confirm the runGA/garescorer is not the problem.  However, 
> this file is a temp file that used to be in /tmp so it's probably not 
> backed up anywhere and definitely not in SVN.
>
> 3. Back in the generate-new-scores.sh at line 289, the 
> "extract-new-scores" script creates scores-new from the scores file 
> but excludes/culls out anything manually scored in 50_scores.cf.
>
> The culled scores-new file has the same 42 lines and ends at 
> MILLION_USD just like our 72_scores.cf so this is the smoking gun but 
> I haven't found the what pulled the trigger yet.  There is something 
> different about this step than back on March 15th when we had our last 
> good 72_scores.cf.
>
>
> THINGS I HAVE CHECKED:
>
> At first I thought that 50_scores.cf changed a lot which caused more 
> exclusion/culling in the 72_scores.cf but that's not it. The revision 
> only shows a few minor changes in 50_scores.cf:
>
> https://svn.apache.org/viewvc/spamassassin/trunk/rules/50_scores.cf?view=log 
>
>
> Next I looked a the garescorer since it get's compiled from 
> garescorer.c every run.  The garescorer.c is identical in the backups 
> and in SVN so that's not it.  Again, I wish I had a 'scores' file to 
> compare to from mid March.
>