spamassassin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzilla-dae...@bugzilla.spamassassin.org
Subject [Bug 6793] PATCH reduce sa-awl memory usage
Date Mon, 23 Apr 2012 23:12:10 GMT
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6793

--- Comment #1 from Mark Martinec <Mark.Martinec@ijs.si> 2012-04-23 23:12:10 UTC ---
> current version of sa-awl loads full database key list to memory before
> showing any stats or performing maintenance. I believe it's obvious that
> this behavior is undesirable and makes large databases impossible to handle

I fully agree. Operations should not depend on being able to load a
complete database into memory if the size of a database is potentially
unlimited.

> The patch below improves sa-awl scaling and responsiveness by scanning
> database row-by-row basis instead of loading all keys to memory first.
> Tested cleaning db with over 8 million rows.
> For a cached db with 850K rows memory usage lowers from 1G to 6M,

Great, thanks!

> execution time is around 12% slower (26 vs. 29 sec), though.

Despite Kevin's concern on the ML, I don't think this is worth
introducing yet another option (and risking people using it
without fully understanding the implications).

One can make a program arbitrarily fast
if correct and reliable operation is not mandated  :)

> I'm not a perl expert, please review.

My only concern is deleting a key while the each() operator is active.

Please see my alternative patch.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

Mime
View raw message