On Tue, May 05, 2009 at 01:19:10PM -0700, Chris Anderson wrote:
> It looks like this reduce would eventually
> overwhelm the interpreter, as your set of hash keys looks like it may
> grow without bounds as it encounters more data.
As you can probably see, it's counting IP address prefixes, and it's
bounded. Even encountering all possible IPv4 prefixes (/0 to /32) and IPv6
(/0 to /128), there will be never be any more than 162 keys in the hash.
> Perhaps I'm wrong. 200 bytes is a bit small, but I'd be worried that
> with 4kb users wouldn't get a warning until they had moved a "bad"
> reduce to production data.
It's not so much a warning as a hard error :-)
> If your reduce is ok even on giant data sets, maybe you can experiment
> with the minimum value in share/server/views.js line 52 that will
> allow you to proceed.
In my case, I'm happy to turn off the checking entirely. I was just
following the request in default.ini:
; If you think you're hitting reduce_limit with a "good" reduce function,
; please let us know on the mailing list so we can fine tune the heuristic.
Regards,
Brian.
|