accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Problem during compacting a table
Date Wed, 05 Aug 2015 15:21:23 GMT
Replication of 5 for the root and metadata tables are done by default 
with Accumulo. As long as you have enough datanodes, HDFS should 
eventually replicate the blocks out, but this isn't an error unless you 
have no replicas (which would imply something wrong with HDFS).

mohit.kaushik wrote:
> yes, One of my datanode was down because disk was detached for some time
> and tserver was lost for that node but Its Up and running again.
>
> fsck show that the file system is healthy. but with so many msgs
> reporting under replicated blocks while my replication factor is 3 it
> shows required is 5.
>
> //user/root/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A0000d29.rf+F0000d28.rf:
> Under replicated
> BP-2102462487-192.168.10.124-1436956492274:blk_1073796198_55442. Target
> Replicas is 5 but found 3 replica(s).///
>
> Thanks & Regards
> Mohit Kaushik
>
> On 08/04/2015 09:18 PM, John Vines wrote:
>> It looks like an hdfs issue. Did a datanode go down? Did you turn
>> replication down to 1? The combination of those two errors would
>> definitely cause the problems your seeing as the latter disables any
>> sort of robustness of the underlying filesystem.
>>
>> On Tue, Aug 4, 2015 at 8:10 AM mohit.kaushik <mohit.kaushik@orkash.com
>> <mailto:mohit.kaushik@orkash.com>> wrote:
>>
>>     On 08/04/2015 05:35 PM, mohit.kaushik wrote:
>>>     Hello All,
>>>
>>>     I am using Apache Accumulo-1.6.3 with Apache Hadoop-2.7.0 on a 3
>>>     node cluster. when I give compact command from the shell it gives
>>>     the folloing warn.
>>>
>>>     root@orkash testScan> compact -w
>>>     2015-08-04 17:10:52,702 [Shell.audit] INFO : root@orkash
>>>     testScan> compact -w
>>>     2015-08-04 17:10:52,706 [shell.Shell] INFO : Compacting table ...
>>>     2015-08-04 17:12:53,986 [impl.ThriftTransportPool] *WARN : Thread
>>>     "shell" stuck on IO  to orkash4:9999 (0) for at least 120034 ms*
>>>
>>>
>>>     Tablet Servers show problem regarding a data block. which is
>>>     something like HDFS-8659
>>>     <https://issues.apache.org/jira/browse/HDFS-8659>
>>>
>>>     /2015-08-04 15:00:27,825 [hdfs.DFSClient] WARN : Failed to
>>>     connect to /192.168.10.121:50010 <http://192.168.10.121:50010>
>>>     for block, add to deadNodes and continue. java.io.IOException:
>>>     Got error, status message opReadBlock
>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911
>>>     received exception
>>>     org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException:
>>>     Replica not found for
>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911,
>>>     for OP_READ_BLOCK, self=/192.168.10.121:38752
>>>     <http://192.168.10.121:38752>, remote=/192.168.10.121:50010
>>>     <http://192.168.10.121:50010>, for file
>>>     /accumulo/tables/h/t-000016s/F000016t.rf, for pool
>>>     BP-2102462487-192.168.10.124-1436956492274 block 1073780678_39911//
>>>     //java.io.IOException: Got error, status message opReadBlock
>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911
>>>     received exception
>>>     org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException:
>>>     Replica not found for
>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911,
>>>     for OP_READ_BLOCK, self=/192.168.10.121:38752
>>>     <http://192.168.10.121:38752>, remote=/192.168.10.121:50010
>>>     <http://192.168.10.121:50010>, for file
>>>     /accumulo/tables/h/t-000016s/F000016t.rf, for pool
>>>     BP-2102462487-192.168.10.124-1436956492274 block 1073780678_39911//
>>>     //        at
>>>     org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:814)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:693)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:352)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)//
>>>     //        at
>>>     org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)//
>>>     //        at java.io.DataInputStream.read(DataInputStream.java:149)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream$1.run(BoundedRangeFileInputStream.java:104)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream$1.run(BoundedRangeFileInputStream.java:100)//
>>>     //        at java.security.AccessController.doPrivileged(Native
>>>     Method)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:100)//
>>>     //        at
>>>     org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)//
>>>     //        at
>>>     org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)//
>>>     //        at
>>>     org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)//
>>>     //        at
>>>     java.io.BufferedInputStream.fill(BufferedInputStream.java:235)//
>>>     //        at
>>>     java.io.BufferedInputStream.read(BufferedInputStream.java:254)//
>>>     //        at
>>>     java.io.FilterInputStream.read(FilterInputStream.java:83)//
>>>     //        at
>>>     java.io.DataInputStream.readInt(DataInputStream.java:387)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:269)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:724)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:497)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.getNext(MultiLevelIndex.java:587)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.getNextNode(MultiLevelIndex.java:593)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$IndexIterator.getNextNode(MultiLevelIndex.java:616)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$IndexIterator.next(MultiLevelIndex.java:659)//
>>>     //        at
>>>     org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:559)/
>>>
>>>     Regards
>>>     Mohit Kaushik
>>>
>>>     **
>>>
>>     And Compaction never completes
>>
>>
>>     --
>>
>>     *Mohit Kaushik*
>>     Software Engineer
>>     A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
>>     *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>>
>>     <http://politicomapper.orkash.com>interactive social intelligence
>>     at work...
>>
>>     <https://www.facebook.com/Orkash2012>
>>     <http://www.linkedin.com/company/orkash-services-private-limited>
>>     <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
>>     <http://www.orkash.com>
>>     <http://www.orkash.com> ... ensuring Assurance in complexity and
>>     uncertainty
>>
>>     /This message including the attachments, if any, is a confidential
>>     business communication. If you are not the intended recipient it
>>     may be unlawful for you to read, copy, distribute, disclose or
>>     otherwise use the information in this e-mail. If you have received
>>     it in error or are not the intended recipient, please destroy it
>>     and notify the sender immediately. Thank you /
>>
>
>
> --
>
> *Mohit Kaushik*
> Software Engineer
> A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
> *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>
> <http://politicomapper.orkash.com>interactive social intelligence at work...
>
> <https://www.facebook.com/Orkash2012>
> <http://www.linkedin.com/company/orkash-services-private-limited>
> <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
> <http://www.orkash.com>
> <http://www.orkash.com> ... ensuring Assurance in complexity and uncertainty
>
> /This message including the attachments, if any, is a confidential
> business communication. If you are not the intended recipient it may be
> unlawful for you to read, copy, distribute, disclose or otherwise use
> the information in this e-mail. If you have received it in error or are
> not the intended recipient, please destroy it and notify the sender
> immediately. Thank you /
>

Mime
View raw message