Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1BC10DFD6 for ; Wed, 14 Nov 2012 08:52:31 +0000 (UTC) Received: (qmail 42895 invoked by uid 500); 14 Nov 2012 08:52:25 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 42655 invoked by uid 500); 14 Nov 2012 08:52:24 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 42545 invoked by uid 99); 14 Nov 2012 08:52:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Nov 2012 08:52:23 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dechouxb@gmail.com designates 209.85.216.176 as permitted sender) Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com) (209.85.216.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Nov 2012 08:52:18 +0000 Received: by mail-qc0-f176.google.com with SMTP id n41so108459qco.35 for ; Wed, 14 Nov 2012 00:51:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VcIkcwZZX2YTDdV1YCstZIitVcA6iEkXUkC451VGD1w=; b=Ln3PCpegxV4Q0/aPUuNi5Mbijk2Uq7fTm5ctjx7YFO7J1tBQOo/rH6+2siROnxOMTF Hc2PqxZCJMDADFGvBhHlwdawEnUC2wn5TREgHbwjvssV8aMLCdp2GB+ZSDLyzS6HO/d2 iXuyxokU7aFMveVVIpgO55XDyRquqYmp2hjHJEB5SpmeK70lSKIqa+h3fZoG/s9iJltt QBjcoImHNQea3PKZ9Q2UbQxeIxeioIopnKsrjLwcJ24cxuvktY+QZLmfl+yVch7DpiRF o/Gs/qnyBHkzwwEnUEuBfFJOmiw+a7MJnfs5qMuK5+AeUyWLeicbEHSy7nWCqBvBl7RQ mZyQ== MIME-Version: 1.0 Received: by 10.224.72.202 with SMTP id n10mr2040165qaj.94.1352883118035; Wed, 14 Nov 2012 00:51:58 -0800 (PST) Received: by 10.49.74.193 with HTTP; Wed, 14 Nov 2012 00:51:57 -0800 (PST) In-Reply-To: References: Date: Wed, 14 Nov 2012 09:51:57 +0100 Message-ID: Subject: Re: Question about MapReduce From: Bertrand Dechoux To: user@hbase.apache.org Cc: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf3074d1feac54ee04ce70a53e X-Virus-Checked: Checked by ClamAV on apache.org --20cf3074d1feac54ee04ce70a53e Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable That would be "mapred.task.timeout". You should download hadoop source, that way you would have the conf files with all the default configuration. (You could also get them online but lots of links are broken.) And the name of the property is only half the story. The error says explicitly that no report status was sent for this period which is the reason for the killing. If you do send a report status, you don't need to change the mapred.task.timeout value which is already pretty high (IMHO/YMMV). The way you can send a status would depend on whether you are using mapred or mapreduce API. Regards Bertrand PS : I cced this discussion to the non hbase specific user mailing list. Replies could be done only on this done because it is a non HBase specific question. On Wed, Nov 14, 2012 at 3:41 AM, Jean-Marc Spaggiari < jean-marc@spaggiari.org> wrote: > One more question about MapReduce. > > One of my servers is slower than the others. I don't have any time > constraint for the job to finish. > > But I'm getting this message: > > "Task attempt_201211122318_0014_m_000021_0 failed to report status for > 601 seconds. Killing!" > > Where can I chance this timeout to something like 1800 seconds? Is it > on the mapred-site.xml file? If so, which property should I insert? > > Thanks, > > JM > > 2012/11/2, Jean-Marc Spaggiari : > > That was my initial plan too, but I was wondering if there was any > > other best practice about the delete. So I will go that way. > > > > Thanks, > > > > JM > > > > 2012/11/2, Shrijeet Paliwal : > >> Not sure what exactly is happening in your job. But in one of the dele= te > >> jobs I wrote I was creating an instance of HTable in setup method of m= y > >> mapper > >> > >> delTab =3D new HTable(conf, conf.get(TABLE_NAME)); > >> > >> And performing delete in map() call using delTab. So no, you do not ha= ve > >> access to table directly *usually*. > >> > >> > >> -Shrijeet > >> > >> > >> On Fri, Nov 2, 2012 at 12:47 PM, Jean-Marc Spaggiari < > >> jean-marc@spaggiari.org> wrote: > >> > >>> Sorry, one last question. > >>> > >>> On the map method, I have access to the row using the values > >>> parameter. Now, based on the value content, I might want to delete it= . > >>> Do I have access to the table directly from one of the parameters? Or > >>> should I call the delete using an HTableInterface from my pool? > >>> > >>> Thanks, > >>> > >>> JM > >>> > >>> 2012/11/2, Jean-Marc Spaggiari : > >>> > Yep, you perfectly got my question. > >>> > > >>> > I just tried and it's working perfectly! > >>> > > >>> > Thanks a lot! I now have a lot to play with. > >>> > > >>> > JM > >>> > > >>> > 2012/11/2, Shrijeet Paliwal : > >>> >> JM, > >>> >> > >>> >> I personally would chose to put it neither hadoop libs nor hbase > >>> >> libs. > >>> >> Have > >>> >> them go to your application's own install directory. > >>> >> > >>> >> Then you could sent the variable HADOOP_CLASSPATH to have your jar > >>> >> (also > >>> >> include hbase jars, hbase dependencies and dependencies your progr= am > >>> >> needs) > >>> >> And to execute fire 'hadoop jar' command. > >>> >> > >>> >> An example[1]: > >>> >> > >>> >> Set classpath: > >>> >> export HADOOP_CLASSPATH=3D`hbase > >>> classpath`:mycool.jar:mycooldependency.jar > >>> >> > >>> >> Fire following to launch your job: > >>> >> hadoop jar mycool.jar hbase.experiments.MyCoolProgram > >>> >> -Dmapred.running.map.limit=3D50 > >>> >> -Dmapred.map.tasks.speculative.execution=3Dfalse aCommandLineArg > >>> >> > >>> >> > >>> >> Did I get your question right? > >>> >> > >>> >> [1] In the example I gave `hbase classpath` gets you set with all > >>> >> hbase > >>> >> jars. > >>> >> > >>> >> > >>> >> > >>> >> On Fri, Nov 2, 2012 at 11:56 AM, Jean-Marc Spaggiari < > >>> >> jean-marc@spaggiari.org> wrote: > >>> >> > >>> >>> Hi Shrijeet, > >>> >>> > >>> >>> Helped a lot! Thanks! > >>> >>> > >>> >>> Now, the only think I need is to know where's the best place to p= ut > >>> >>> my > >>> >>> JAR on the server. Should I put it on the hadoop lib directory? O= r > >>> >>> somewhere on the HBase structure? > >>> >>> > >>> >>> Thanks, > >>> >>> > >>> >>> JM > >>> >>> > >>> >>> 2012/10/29, Shrijeet Paliwal : > >>> >>> > In line. > >>> >>> > > >>> >>> > On Mon, Oct 29, 2012 at 8:11 AM, Jean-Marc Spaggiari < > >>> >>> > jean-marc@spaggiari.org> wrote: > >>> >>> > > >>> >>> >> I'm replying to myself ;) > >>> >>> >> > >>> >>> >> I found "cleanup" and "setup" methods from the TableMapper > table. > >>> >>> >> So > >>> >>> >> I > >>> >>> >> think those are the methods I was looking for. I will init the > >>> >>> >> HTablePool there. Please let me know if I'm wrong. > >>> >>> >> > >>> >>> >> Now, I still have few other questions. > >>> >>> >> > >>> >>> >> 1) context.getCurrentValue() can throw a InterrruptedException= , > >>> >>> >> but > >>> >>> >> when can this occur? Is there a timeout on the Mapper side? Of > >>> >>> >> it's > >>> >>> >> if > >>> >>> >> the region is going down while the job is running? > >>> >>> >> > >>> >>> > > >>> >>> > You do not need to call context.getCurrentValue(). The 'value' > >>> >>> > argument > >>> >>> to > >>> >>> > map method[1] has the information you are looking for. > >>> >>> > > >>> >>> > > >>> >>> >> 2) How can I pass parameters to the Map method? Can I use > >>> >>> >> job.getConfiguration().put to add some properties there, can g= et > >>> them > >>> >>> >> back in context.getConfiguration.get? > >>> >>> >> > >>> >>> > > >>> >>> > Yes, thats how it is done. > >>> >>> > > >>> >>> > > >>> >>> >> 3) What's the best way to log results/exceptions/traces from t= he > >>> >>> >> map > >>> >>> >> method? > >>> >>> >> > >>> >>> > > >>> >>> > In most cases, you'll have mapper and reducer classes as nested > >>> static > >>> >>> > classes within some enclosing class. You can get handle to the > >>> >>> > Logger > >>> >>> from > >>> >>> > the enclosing class and do your usual LOG.info, LOG.warn yada > >>> >>> > yada. > >>> >>> > > >>> >>> > Hope it helps. > >>> >>> > > >>> >>> > [1] map(KEYIN key, *VALUEIN value*, Context context) > >>> >>> > > >>> >>> >> > >>> >>> >> I will search on my side, but some help will be welcome becaus= e > >>> >>> >> it > >>> >>> >> seems there is not much documentation when we start to dig a b= it > >>> >>> >> :( > >>> >>> >> > >>> >>> >> JM > >>> >>> >> > >>> >>> >> 2012/10/27, Jean-Marc Spaggiari : > >>> >>> >> > Hi, > >>> >>> >> > > >>> >>> >> > I'm thinking about my firs MapReduce class and I have some > >>> >>> >> > questions. > >>> >>> >> > > >>> >>> >> > The goal of it will be to move some rows from one table to > >>> >>> >> > another > >>> >>> >> > one > >>> >>> >> > based on the timestamp only. > >>> >>> >> > > >>> >>> >> > Since this is pretty new for me, I'm starting from the > >>> >>> >> > RowCounter > >>> >>> >> > class to have a baseline. > >>> >>> >> > > >>> >>> >> > There are few things I will have to update. First, the > >>> >>> >> > createSumittableJob method to get timestamp range instead of > >>> >>> >> > key > >>> >>> >> > range, and "play2 with the parameters. This part is fine. > >>> >>> >> > > >>> >>> >> > Next, I need to update the map method, and this is where I > have > >>> >>> >> > some > >>> >>> >> > questions. > >>> >>> >> > > >>> >>> >> > I'm able to find the timestamp of all the cf:c from the > >>> >>> >> > context.getCurrentValue() method, that's fine. Now, my conce= rn > >>> >>> >> > is > >>> >>> >> > on > >>> >>> >> > the way to get access to the table to store this field, and > the > >>> >>> >> > table > >>> >>> >> > to delete it. Should I instantiate an HTable for the source > >>> >>> >> > table, > >>> >>> >> > and > >>> >>> >> > execute and delete on it, then do an insert on another HTabl= e > >>> >>> >> > instance? Should I use an HTablePool? Also, since I=92m alr= eady > >>> >>> >> > on > >>> >>> >> > the > >>> >>> >> > row, can=92t I just mark it as deleted instead of calling a = new > >>> >>> >> > HTable? > >>> >>> >> > > >>> >>> >> > Also, instead of calling the delete and put one by one, I > would > >>> >>> >> > like > >>> >>> >> > to put them on a list and execute it only when it=92s over 1= 0 > >>> >>> >> > members. > >>> >>> >> > How can I make sure that at the end of the job, this is > >>> >>> >> > flushed? > >>> >>> >> > Else, > >>> >>> >> > I will lose some operations. Is there a kind of =93dispose= =94 > >>> >>> >> > method > >>> >>> >> > called on the region when the job is done? > >>> >>> >> > > >>> >>> >> > Thanks, > >>> >>> >> > > >>> >>> >> > JM > >>> >>> >> > > >>> >>> >> > >>> >>> > > >>> >>> > >>> >> > >>> > > >>> > >> > > > --=20 Bertrand Dechoux --20cf3074d1feac54ee04ce70a53e Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable That would be "mapred.task.timeout".
You should download hadoo= p source, that way you would have the conf files with all the default confi= guration.
(You could also get them online but lots of links are broken.)=

And the name of the property is only half the story.
The error says = explicitly that no report status was sent for this period which is the reas= on for the killing.
If you do send a report status, you don't need t= o change the mapred.task.timeout value which is already pretty high (IMHO/Y= MMV).

The way you can send a status would depend on whether you are using map= red or mapreduce API.

Regards

Bertrand

PS : I cced thi= s discussion to the non hbase specific user mailing list. Replies could be = done only on this done because it is a non HBase specific question.

On Wed, Nov 14, 2012 at 3:41 AM, Jean-Marc S= paggiari <jean-marc@spaggiari.org> wrote:
One more question about MapReduce.

One of my servers is slower than the others. I don't have any time
constraint for the job to finish.

But I'm getting this message:

"Task attempt_201211122318_0014_m_000021_0 failed to report status for=
601 seconds. Killing!"

Where can I chance this timeout to something like 1800 seconds? Is it
on the mapred-site.xml file? If so, which property should I insert?

Thanks,

JM

2012/11/2, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
> That was my initial plan too, but I was wondering if there was any
> other best practice about the delete. =A0So I will go that way.
>
> Thanks,
>
> JM
>
> 2012/11/2, Shrijeet Paliwal <shrijeet@rocketfuel.com>:
>> Not sure what exactly is happening in your job. But in one of the = delete
>> jobs I wrote I was creating an instance of HTable in setup method = of my
>> mapper
>>
>> delTab =3D new HTable(conf, conf.get(TABLE_NAME));
>>
>> And performing delete in map() call using delTab. So no, you do no= t have
>> access to table directly *usually*.
>>
>>
>> -Shrijeet
>>
>>
>> On Fri, Nov 2, 2012 at 12:47 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org= > wrote:
>>
>>> Sorry, one last question.
>>>
>>> On the map method, I have access to the row using the values >>> parameter. Now, based on the value content, I might want to de= lete it.
>>> Do I have access to the table directly from one of the paramet= ers? Or
>>> should I call the delete using an HTableInterface from my pool= ?
>>>
>>> Thanks,
>>>
>>> JM
>>>
>>> 2012/11/2, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>> > Yep, you perfectly got my question.
>>> >
>>> > I just tried and it's working perfectly!
>>> >
>>> > Thanks a lot! I now have a lot to play with.
>>> >
>>> > JM
>>> >
>>> > 2012/11/2, Shrijeet Paliwal <shrijeet@rocketfuel.com>:
>>> >> JM,
>>> >>
>>> >> I personally would chose to put it neither hadoop lib= s nor hbase
>>> >> libs.
>>> >> Have
>>> >> them go to your application's own install directo= ry.
>>> >>
>>> >> Then you could sent the variable HADOOP_CLASSPATH to = have your jar
>>> >> (also
>>> >> include hbase jars, hbase dependencies and dependenci= es your program
>>> >> needs)
>>> >> And to execute fire 'hadoop jar' command.
>>> >>
>>> >> An example[1]:
>>> >>
>>> >> Set classpath:
>>> >> export HADOOP_CLASSPATH=3D`hbase
>>> classpath`:mycool.jar:mycooldependency.jar
>>> >>
>>> >> Fire following to launch your job:
>>> >> hadoop jar mycool.jar hbase.experiments.MyCoolProgram=
>>> >> -Dmapred.running.map.limit=3D50
>>> >> -Dmapred.map.tasks.speculative.execution=3Dfalse aCom= mandLineArg
>>> >>
>>> >>
>>> >> Did I get your question right?
>>> >>
>>> >> [1] In the example I gave `hbase classpath` gets you = set with all
>>> >> hbase
>>> >> jars.
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Nov 2, 2012 at 11:56 AM, Jean-Marc Spaggiari = <
>>> >> jean-marc@= spaggiari.org> wrote:
>>> >>
>>> >>> Hi Shrijeet,
>>> >>>
>>> >>> Helped a lot! Thanks!
>>> >>>
>>> >>> Now, the only think I need is to know where's= the best place to put
>>> >>> my
>>> >>> JAR on the server. Should I put it on the hadoop = lib directory? Or
>>> >>> somewhere on the HBase structure?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> JM
>>> >>>
>>> >>> 2012/10/29, Shrijeet Paliwal <shrijeet@rocketfuel.com>:
>>> >>> > In line.
>>> >>> >
>>> >>> > On Mon, Oct 29, 2012 at 8:11 AM, Jean-Marc S= paggiari <
>>> >>> > j= ean-marc@spaggiari.org> wrote:
>>> >>> >
>>> >>> >> I'm replying to myself ;)
>>> >>> >>
>>> >>> >> I found "cleanup" and "se= tup" methods from the TableMapper table.
>>> >>> >> So
>>> >>> >> I
>>> >>> >> think those are the methods I was lookin= g for. I will init the
>>> >>> >> HTablePool there. Please let me know if = I'm wrong.
>>> >>> >>
>>> >>> >> Now, I still have few other questions. >>> >>> >>
>>> >>> >> 1) context.getCurrentValue() can throw a= InterrruptedException,
>>> >>> >> but
>>> >>> >> when can this occur? Is there a timeout = on the Mapper side? Of
>>> >>> >> it's
>>> >>> >> if
>>> >>> >> the region is going down while the job i= s running?
>>> >>> >>
>>> >>> >
>>> >>> > You do not need to call =A0context.getCurren= tValue(). The 'value'
>>> >>> > argument
>>> >>> to
>>> >>> > map method[1] has the information you are lo= oking for.
>>> >>> >
>>> >>> >
>>> >>> >> 2) How can I pass parameters to the Map = method? Can I use
>>> >>> >> job.getConfiguration().put to add some p= roperties there, can get
>>> them
>>> >>> >> back in context.getConfiguration.get? >>> >>> >>
>>> >>> >
>>> >>> > Yes, thats how it is done.
>>> >>> >
>>> >>> >
>>> >>> >> 3) What's the best way to log result= s/exceptions/traces from the
>>> >>> >> map
>>> >>> >> method?
>>> >>> >>
>>> >>> >
>>> >>> > In most cases, you'll have mapper and re= ducer classes as nested
>>> static
>>> >>> > classes within some enclosing class. You can= get handle to the
>>> >>> > Logger
>>> >>> from
>>> >>> > the enclosing class and do your usual LOG.in= fo, LOG.warn yada
>>> >>> > yada.
>>> >>> >
>>> >>> > Hope it helps.
>>> >>> >
>>> >>> > [1] map(KEYIN key, *VALUEIN value*, Context = context)
>>> >>> >
>>> >>> >>
>>> >>> >> I will search on my side, but some help = will be welcome because
>>> >>> >> it
>>> >>> >> seems there is not much documentation wh= en we start to dig a bit
>>> >>> >> :(
>>> >>> >>
>>> >>> >> JM
>>> >>> >>
>>> >>> >> 2012/10/27, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>>> >>> >> > Hi,
>>> >>> >> >
>>> >>> >> > I'm thinking about my firs MapR= educe class and I have some
>>> >>> >> > questions.
>>> >>> >> >
>>> >>> >> > The goal of it will be to move some= rows from one table to
>>> >>> >> > another
>>> >>> >> > one
>>> >>> >> > based on the timestamp only.
>>> >>> >> >
>>> >>> >> > Since this is pretty new for me, I&= #39;m starting from the
>>> >>> >> > RowCounter
>>> >>> >> > class to have a baseline.
>>> >>> >> >
>>> >>> >> > There are few things I will have to= update. First, the
>>> >>> >> > createSumittableJob method to get t= imestamp range instead of
>>> >>> >> > key
>>> >>> >> > range, and "play2 with the par= ameters. This part is fine.
>>> >>> >> >
>>> >>> >> > Next, I need to update the map meth= od, and this is where I have
>>> >>> >> > some
>>> >>> >> > questions.
>>> >>> >> >
>>> >>> >> > I'm able to find the timestamp = of all the cf:c from the
>>> >>> >> > context.getCurrentValue() method, t= hat's fine. Now, my concern
>>> >>> >> > is
>>> >>> >> > on
>>> >>> >> > the way to get access to the table = to store this field, and the
>>> >>> >> > table
>>> >>> >> > to delete it. Should I instantiate = an HTable for the source
>>> >>> >> > table,
>>> >>> >> > and
>>> >>> >> > execute and delete on it, then do a= n insert on another HTable
>>> >>> >> > instance? =A0Should I use an HTable= Pool? Also, since I=92m already
>>> >>> >> > on
>>> >>> >> > the
>>> >>> >> > row, can=92t I just mark it as dele= ted instead of calling a new
>>> >>> >> > HTable?
>>> >>> >> >
>>> >>> >> > Also, instead of calling the delete= and put one by one, I would
>>> >>> >> > like
>>> >>> >> > to put them on a list and execute i= t only when it=92s over 10
>>> >>> >> > members.
>>> >>> >> > How can I make sure that at the end= of the job, this is
>>> >>> >> > flushed?
>>> >>> >> > Else,
>>> >>> >> > I will lose some operations. Is the= re a kind of =93dispose=94
>>> >>> >> > method
>>> >>> >> > called on the region when the job i= s done?
>>> >>> >> >
>>> >>> >> > Thanks,
>>> >>> >> >
>>> >>> >> > JM
>>> >>> >> >
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>> >
>>>
>>
>



--
Bertrand Dechoux
--20cf3074d1feac54ee04ce70a53e--