pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Turn off Speculative Execution in a UDF?
Date Fri, 02 Mar 2012 22:40:52 GMT
I thought it was too late in the workflow to do this, but it would be
really cool if you could.  I don't think to think about MapReduce much when
I Pig, except to group my scripts by jobs... so this was a surprise for me.
 Made sense once i thought of it.  But it was a surprise.

On Fri, Mar 2, 2012 at 1:49 PM, Bill Graham <billgraham@gmail.com> wrote:

> I was also curious about this and will try it, but my initial thought was
> that at that point it might be tool late in the workflow of the job. I'll
> give it a shot and report back.
>
>
> On Fri, Mar 2, 2012 at 1:45 PM, Dmitriy Ryaboy <dvryaboy@gmail.com> wrote:
>
> > In a StoreFunc, you could do that when you get passed the jobconf, right?
> >
> > On Thu, Mar 1, 2012 at 9:37 PM, Bill Graham <billgraham@gmail.com>
> wrote:
> > > I don't think so. We just do it in the pig script before using the
> store
> > > func:
> > >
> > > SET mapred.map.tasks.speculative.execution false
> > >
> > >
> > > On Thu, Mar 1, 2012 at 4:19 PM, Russell Jurney <
> russell.jurney@gmail.com
> > >wrote:
> > >
> > >> Can you turn off speculative execution in a StoreFunc?  I believe it
> is
> > >> leading to duplicates in MongoStorage();
> > >>
> > >> --
> > >> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
> > >> datasyndrome.com
> > >>
> > >
> > >
> > >
> > > --
> > > *Note that I'm no longer using my Yahoo! email address. Please email me
> > at
> > > billgraham@gmail.com going forward.*
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> billgraham@gmail.com going forward.*
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message