crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: performance impact of batching emit(...)
Date Thu, 09 Jan 2014 23:37:23 GMT
Hey Leen,

I don't have a better idea than trial and error at this point, since the
best choice of flushEvery would depend on a combination of how much memory
is available to the tasks, how large the cached objects are, and a rough
estimate of how many unique elements there are in the data set. It's the
sort of thing that our much-discussed-but-not-implemented-yet framework for
tracking stats on runtime metrics for optimizing pipelines should track.


On Thu, Jan 9, 2014 at 1:30 PM, Leen Toelen <> wrote:

> Hi,
> when looking at PreDistinct I notice that calls to emitter.emit(...) are
> stored in memory until more than 'flushEvery' records are found. How does
> this batching impact performance, since the calls to emit(...) are not
> batched in the cleanup method but called in a loop?
> Is there an easy way to find the best size for 'flushEvery' other than try
> and error?
> Best regards,
> Leen

Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

View raw message