accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josef Roehrl <jroe...@phemi.com>
Subject Re: Unable to import RFile produced by AccumuloFileOutputFormat
Date Fri, 08 Jul 2016 21:36:33 GMT
Russ is a co-worker.

There are at least 3 variants, depending, if that helps:

Unable to find tablets that overlap file 
hdfs://mas1.soaktest.phemi.com:8020/apps/accumulo/data/tables/3/b-00019av/I0001b1u.rf
java.io.IOException: Corrupted Meta region Index Unable to find tablets 
that overlap file 
hdfs://mas1.soaktest.phemi.com:8020/apps/accumulo/data/tables/3/b-00016t1/I00017xv.rf 
java.lang.RuntimeException: Incompatible BCFile fileBCFileVersion. 
Unable to find tablets that overlap file 
hdfs://mas1.soaktest.phemi.com:8020/apps/accumulo/data/tables/3/b-00016t1/I000173f.rf 
java.lang.IndexOutOfBoundsException: Invalid offset/length: 6230/-25
On 16-07-08 11:49 AM, Russ Weeks wrote:
> Hi, folks,
> Has anybody ever encountered a problem where the RFiles that are 
> generated by AccumuloFileOutputFormat can't be imported using 
> TableOperations.importDirectory?
> I'm seeing this problem very frequently for small RFiles and 
> occasionally for larger RFiles. The errors shown in the monitor's log 
> UI suggest a corrupt file, to me. For instance, the stack trace below 
> shows a case where the BCFileVersion was incorrect, but sometimes it 
> will complain about an invalid length, negative offset, or invalid codec.
> I'm using HDP Accumulo 1.7.0 (1.7.0.2.3.4.12-1) on an encrypted HDFS 
> volume, with Kerberos turned on. The RFiles are generated by 
> AccumuloFileOutputFormat from a Spark job.
> A very small RFile that exhibits this problem is available here: 
> http://firebar.newbrightidea.com/downloads/bad_rfiles/I0000waz.rf
> I'm pretty confident that the keys are being written to the RFile in 
> order. Are there any tools I could use to inspect the internal 
> structure of the RFile?
> Thanks,
> -Russ
> Unable to find tablets that overlap file 
> hdfs://[redacted]/accumulo/data/tables/f/b-0000ze9/I0000zeb.rf
> java.lang.RuntimeException: Incompatible BCFile fileBCFileVersion.
> at 
> org.apache.accumulo.core.file.rfile.bcfile.BCFile$Reader.<init>(BCFile.java:828)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.init(CachableBlockFile.java:246)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBCFile(CachableBlockFile.java:257)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.access$100(CachableBlockFile.java:137)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$MetaBlockLoader.get(CachableBlockFile.java:209)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBlock(CachableBlockFile.java:313)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:368)
> at 
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:137)
> at org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:843)
> at 
> org.apache.accumulo.core.file.rfile.RFileOperations.openReader(RFileOperations.java:79)
> at 
> org.apache.accumulo.core.file.DispatchingFileFactory.openReader(DispatchingFileFactory.java:69)
> at 
> org.apache.accumulo.server.client.BulkImporter.findOverlappingTablets(BulkImporter.java:644)
> at 
> org.apache.accumulo.server.client.BulkImporter.findOverlappingTablets(BulkImporter.java:615)
> at 
> org.apache.accumulo.server.client.BulkImporter$1.run(BulkImporter.java:146)
> at 
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
-- 
> Josef Roehrl
> Senior Software Developer
> *PHEMI Systems*
> 180-887 Great Northern Way
> Vancouver, BC V5T 4T5
> 604-336-1119
> Website <http://www.phemi.com/> Twitter 
> <https://twitter.com/PHEMISystems> Linkedin 
> <http://www.linkedin.com/company/3561810?trk=tyah&amp;trkInfo=tarId%3A1403279580554%2Ctas%3Aphemi%20hea%2Cidx%3A1-1-1>



Mime
View raw message