Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of mkwhitacre@gmail.com designates
 209.85.219.45 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1362797600.68212.YahooMailNeo@web185001.mail.gq1.yahoo.com>
References: <1362797600.68212.YahooMailNeo@web185001.mail.gq1.yahoo.com>
Date: Fri, 8 Mar 2013 22:16:29 -0600
Message-ID: 
 <CADH-_NL9abo_QgMhB3Vjynq5DB-x7aDbQtjCAsadLJ9cL2c0Jg@mail.gmail.com>
Subject: Re: MultipleOutput in crunch
From: Micah Whitacre <mkwhitacre@gmail.com>
To: user@crunch.apache.org, Peter Knap <pknap@yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1

Instead of implementing a filter could you switch to using a DoFn and
emit a Pair?  Then the first part of the pair would be the identifier
for the category of data.  You can then group by key to process them
differently or just keep processing them by the same DoFn using the
key as a flag to how to process it.

That being said I'm not really sure this would be any more efficient
than filtering twice.


On Fri, Mar 8, 2013 at 8:53 PM, Peter Knap <pknap@yahoo.com> wrote:
> Hi,
>
> Is multiple output functionality supported by crunch? I have looked at the
> source code but could find a way to do it. I have the following scenario:
> input file would be processed by multiple sequential filters, the records
> passing the filter criteria need to be processed differently than the ones
> which are not. What's the best way to do it in crunch? I know I can proccess
> the input data twice by two different fillters but this is not efficient.
> Any suggestion from you guys?
>
> Thanks,
> Piotr
>