hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: SequenceFileLogReader uses a reflection hack resulting in runtime failures
Date Fri, 02 Dec 2011 06:32:49 GMT
This reflection should occur only once, not at every write to the HLog, so
the performance impact should be minimal, is it not?

why are you seeing the exception now? are u using a new unit test or a new
hdfs jar?

On Thu, Dec 1, 2011 at 9:59 PM, Mikhail Bautin <
bautin.mailing.lists@gmail.com> wrote:

> Hello,
>
> The following reflection hack is from SequenceFileLogReader.java.
>
>         try {
>           Field fIn = FilterInputStream.class.getDeclaredField("in");
>           fIn.setAccessible(true);
>           Object realIn = fIn.get(this.in);
>           Method getFileLength = realIn.getClass().
>             getMethod("getFileLength", new Class<?> []{});
>           getFileLength.setAccessible(true);
>           long realLength = ((Long)getFileLength.
>             invoke(realIn, new Object []{})).longValue();
>           assert(realLength >= this.length);
>           adjust = realLength - this.length;
>         } catch(Exception e) {
>           SequenceFileLogReader.LOG.warn(
>             "Error while trying to get accurate file length.  " +
>             "Truncation / data loss may occur if RegionServers die.", e);
>         }
>
> In my testing this resulted in the following error:
>
> 11/12/01 21:40:07 WARN wal.SequenceFileLogReader: Error while trying to get
> accurate file length.  Truncation / data loss may occur if RegionServers
> die.
> java.lang.NoSuchMethodException:
>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.getFileLength()
>        at java.lang.Class.getMethod(Class.java:1605)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:108)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1434)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
>        at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:681)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:821)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:734)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:382)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:351)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:113)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:266)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:197)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165)
>        at java.lang.Thread.run(Thread.java:680)
>
> Besides, even when it works, the reflection-based solution is probably
> _much_ slower than straightforward method access. Should we create a Hadoop
> patch to expose the appropriate API call and get rid of the reflection
> hack?
>
> Thanks,
> --Mikhail
>



-- 
Subscribe to my posts at http://www.facebook.com/dhruba

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message