hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Trailer 'header' is wrong; does the trailer size match content?
Date Mon, 21 Dec 2009 16:36:54 GMT
I opened https://issues.apache.org/jira/browse/HBASE-2063 to cover this
issue.
Thanks,
St.Ack

On Mon, Dec 21, 2009 at 8:35 AM, stack <stack@duboce.net> wrote:

> I was going to write that perhaps you needed to
> turn mapred.reduce.tasks.speculative.execution off, but if enabling it and
> things work, that would seem to indicate that a our reducer first takes
> longer than the task timeout maximum and secondly, on failure, we should
> clean up the hfile.
>
> On the first issue, you are using KeyValueSortReducer?  Are your values
> large?  We set reducer status every 100 values.  Maybe this is not enough?
>  We should set status more frequently?  If you call context setstatus more
> frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile
> and then close it.  On kill, this code is not being called.   Let me see if
> can do something about that (e.g. register a shutdown hook to clean away
> incomplete files -- ).
>
> Thanks,
> St.Ack
>
>
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <chingshenchen@gmail.com>wrote:
>
>> I think I found a way.
>> I set the "mapred.reduce.tasks.speculative.execution" to true and output
>> hfiles again, then successfully load hfiles into hbase.
>> Is it best solution? or HFileOutputFormat bug?
>>
>> Shen
>>
>> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <chingshenchen@gmail.com>
>> wrote:
>>
>> > Thanks, stack.
>> >
>> > I checked this file that isn't empty. But I found that as long as the
>> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb
>> script
>> > to load hfiles then failed.
>> > How to avoid this problem?
>> >
>> > Thanks.
>> >
>> > Shen
>> >
>> >
>> > On Sat, Dec 19, 2009 at 3:49 AM, stack <stack@duboce.net> wrote:
>> >
>> >> Check the
>> >> file
>> >>
>> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
>> >>  Is it empty?  Was there an error during running of your MR job?
>>  Perhaps
>> >> a
>> >> task failed?
>> >>
>> >> St.Ack
>> >>
>> >>
>> >>
>> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <chingshenchen@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi,
>> >> >  I use the script loadtable.rb to load my hfiles into hbase, but I
>> got
>> >> an
>> >> > exception as below.
>> >> >  Does anyone have any suggestions?
>> >> >
>> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
>> >> from
>> >> >
>> >> >
>> >>
>> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
>> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
>> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
>> >> match
>> >> > content? (NativeException)
>> >> >    from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
>> `readTrailer'
>> >> >    from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
>> >> `loadFileInfo'
>> >> >    from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
>> >> >    from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>> >> >    from java/lang/reflect/Method.java:597:in `invoke'
>> >> >    from org/jruby/javasupport/JavaMethod.java:298:in
>> >> > `invokeWithExceptionHandling'
>> >> >    from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>> >> >    from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in
>> `call'
>> >> >     ... 18 levels...
>> >> >    from org/jruby/Main.java:94:in `main'
>> >> >    from loadtable.rb:83:in `each'
>> >> >    from loadtable.rb:83
>> >> > Complete Java stackTrace
>> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
>> >> match
>> >> > content?
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
>> >> >    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> >> >    at
>> >> >
>> >> >
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> >    at java.lang.reflect.Method.invoke(Method.java:597)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
>> >> >    at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
>> >> >    at
>> >> >
>> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
>> >> >    at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
>> >> >    at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
>> >> >    at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
>> >> Source)
>> >> >    at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
>> >> >    at org.jruby.runtime.Block.yield(Block.java:100)
>> >> >    at
>> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
>> >> >    at loadtable.__file__(loadtable.rb:83)
>> >> >    at loadtable.load(loadtable.rb)
>> >> >    at org.jruby.Ruby.runScript(Ruby.java:577)
>> >> >    at org.jruby.Ruby.runNormally(Ruby.java:480)
>> >> >    at org.jruby.Ruby.runFromMain(Ruby.java:354)
>> >> >    at org.jruby.Main.run(Main.java:229)
>> >> >    at org.jruby.Main.run(Main.java:110)
>> >> >    at org.jruby.Main.main(Main.java:94)
>> >> >
>> >>
>> >
>> >
>> >
>> >
>>
>>
>> --
>> *****************************************************
>> Ching-Shen Chen
>> Advanced Technology Center,
>> Information & Communications Research Lab.
>> E-mail: chenchingshen@itri.org.tw
>> Tel:+886-3-5915542
>> *****************************************************
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message