hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J Mohamed Zahoor <jmo...@gmail.com>
Subject Re: Scan exception when running MR
Date Tue, 23 Oct 2012 06:51:14 GMT
No Worries.
It was a stray hbase jar that caused the problem.

./zahoor

On 22-Oct-2012, at 10:43 PM, Bryan Beaudreault <bbeaudreault@hubspot.com> wrote:

> Oh, sorry.
> 
> This sounds like a version mismatch.  Do you have the same version
> installed on your servers and being pulled in your hadoop job?
> 
> On Mon, Oct 22, 2012 at 12:32 PM, J Mohamed Zahoor <jmozah@gmail.com> wrote:
> 
>> Cool… But my map reduce doesn't even start…
>> It fails while creating a record reader...
>> The record reader fails in
>> 
>> TableMapReduceUtil.convertStringToScan(conf.get(SCAN));
>> 
>> and throws a
>> 
>> java.io.IOException: version not supported
>>        at org.apache.hadoop.hbase.client.Scan.readFields(Scan.java:558)
>>        at
>> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertStringToScan(TableMapReduceUtil.java:255)
>>        at
>> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:105)
>>        at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>>        at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.security.auth.Subject.doAs(Subject.java:415)
>>        at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>        at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> 
>> 
>> Basically the deserializing of Scan object from the conf is failing for
>> some reason.
>> 
>> ./zahoor
>> 
>> 
>> 
>> 
>> 
>> On 22-Oct-2012, at 9:52 PM, Bryan Beaudreault <bbeaudreault@hubspot.com>
>> wrote:
>> 
>>> I'm not on 0.94.1, but I've found a lot of situations that can cause
>>> scanner timeouts and other scanner exceptions from M/R.  The primary ones
>>> probably still apply in later versions:
>>> 
>>> 
>>>  - Caching or batching set too high.  If caching is set to, e.g. 1000,
>>>  and hbase.rpc.timeout is set to 30 seconds, it means you need to be
>> able to
>>>  be able to process 1000 records in your mapper in under 30 seconds
>> (minus
>>>  the overhead of actually returning that many records). Otherwise the
>>>  mapper's next call to next() will throw a timeout.
>>>  - Similar to the above, this can happen if just the logic in your
>> mapper
>>>  is too heavy and taking too long.  Just keep in mind that the
>>>  hbase.rpc.timeout can be triggered by the time between calls to next()
>>>  - hbase.rpc.timeout > hbase.regionserver.lease.period.  If this is the
>>>  case, the RS will timeout first and kill the scan.  Then your mapper
>> will
>>>  call next() and since the scan no longer exists will throw a scan/lease
>>>  exception.
>>>  - The filters on the mapper scan are causing too many rows to be
>>>  skipped, such that not enough rows can be collected to return within
>> the
>>>  timeout.
>>> 
>>> Hope this helps and/or is still accurate for your version.
>>> 
>>> On Mon, Oct 22, 2012 at 11:51 AM, J Mohamed Zahoor <jmozah@gmail.com>
>> wrote:
>>> 
>>>> I am using 0.94.1
>>>> 
>>>> ./zahoor
>>>> 
>>>> 
>>>> On 22-Oct-2012, at 9:17 PM, J Mohamed Zahoor <jmozah@gmail.com> wrote:
>>>> 
>>>>> Hi
>>>>> 
>>>>> I am facing a scanner exception like this when i run a mr job.
>>>>> Both the input and output are hbase tables (different tables)…
>>>>> This comes sporadically on some mapper and all other mappers runs
>> fine..
>>>>> Even the failed mapper gets passed in the next attempt.
>>>>> Any clue on what might be wrong?
>>>>> 
>>>>> java.lang.NullPointerException
>>>>>     at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
>>>>>     at
>>>> 
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
>>>>>     at
>>>> 
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
>>>>>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
>>>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>>     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>>     at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>     at
>>>> 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>>>     at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>> 
>>>>> 
>>>>> ./zahoor
>>>> 
>>>> 
>> 
>> 


Mime
View raw message