crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: counters in multistage pipeline
Date Wed, 25 Mar 2015 20:26:25 GMT
My guess is yes, it's okay to just sum the results, but I'll double check
the code- I'm assuming the counts are different in the different
StageResults?

Nothing that does an automatic merge, but agree that it would be useful.
Post a patch? Maybe a new method on PipelineResult?
On Wed, Mar 25, 2015 at 10:01 AM Tycho Lamerigts <Tycho.Lamerigts@imc.nl>
wrote:

>  I have one DoFn that increments a counter (and a bunch of other DoFns
> that don’t use counters). When my DoFn with counter runs in a MRPipeline
> that returns multiple StageResults, I notice that my counter appears with a
> non-zero value in 2 StageResults. Is it correct to just sum up these
> values? Is there any utility class to this automatically for all counters
> in PipelineResult?
>
>  T
>
>
> ------------------------------
>
> The information in this e-mail is intended only for the person or entity
> to which it is addressed.
>
> It may contain confidential and /or privileged material. If someone other
> than the intended recipient should receive this e-mail, he / she shall not
> be entitled to read, disseminate, disclose or duplicate it.
>
> If you receive this e-mail unintentionally, please inform us immediately
> by "reply" and then delete it from your system. Although this information
> has been compiled with great care, neither IMC Financial Markets & Asset
> Management nor any of its related entities shall accept any responsibility
> for any errors, omissions or other inaccuracies in this information or for
> the consequences thereof, nor shall it be bound in any way by the contents
> of this e-mail or its attachments. In the event of incomplete or incorrect
> transmission, please return the e-mail to the sender and permanently delete
> this message and any attachments.
>
> Messages and attachments are scanned for all known viruses. Always scan
> attachments before opening them.
>

Mime
View raw message