Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 73CA6181B4 for ; Thu, 6 Aug 2015 06:58:24 +0000 (UTC) Received: (qmail 50128 invoked by uid 500); 6 Aug 2015 06:58:18 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 50020 invoked by uid 500); 6 Aug 2015 06:58:18 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 50003 invoked by uid 99); 6 Aug 2015 06:58:17 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2015 06:58:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5E4F11A9900 for ; Thu, 6 Aug 2015 06:58:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.002 X-Spam-Level: *** X-Spam-Status: No, score=3.002 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 6JhTvuK57leQ for ; Thu, 6 Aug 2015 06:58:04 +0000 (UTC) Received: from smtp73.ord1c.emailsrvr.com (smtp73.ord1c.emailsrvr.com [108.166.43.73]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id A1A0B2092B for ; Thu, 6 Aug 2015 06:58:04 +0000 (UTC) Received: from smtp18.relay.ord1c.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp18.relay.ord1c.emailsrvr.com (SMTP Server) with ESMTP id 2C9F6280100; Thu, 6 Aug 2015 02:57:58 -0400 (EDT) Received: by smtp18.relay.ord1c.emailsrvr.com (Authenticated sender: mohit.kaushik-AT-orkash.com) with ESMTPSA id 7C031280170 for ; Thu, 6 Aug 2015 02:57:56 -0400 (EDT) X-Sender-Id: mohit.kaushik@orkash.com Received: from [192.168.0.121] ([UNAVAILABLE]. [14.141.49.198]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA) by 0.0.0.0:465 (trex/5.4.2); Thu, 06 Aug 2015 06:57:58 GMT Message-ID: <55C3059F.4030201@orkash.com> Date: Thu, 06 Aug 2015 12:28:39 +0530 From: "mohit.kaushik" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: user@hadoop.apache.org Subject: Hadoop logs show "Not scanning suspicious block" References: <55C22AAC.9050000@gmail.com> In-Reply-To: <55C22AAC.9050000@gmail.com> X-Forwarded-Message-Id: <55C22AAC.9050000@gmail.com> Content-Type: multipart/alternative; boundary="------------030008030209060202060608" This is a multi-part message in MIME format. --------------030008030209060202060608 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit -------- Forwarded Message -------- Subject: Re: Problem during compacting a table Date: Wed, 05 Aug 2015 11:24:28 -0400 From: Josh Elser Reply-To: user@accumulo.apache.org To: user@accumulo.apache.org I'm not really sure what that error message means without doing more digging. Copying your email to user@hadoop.apache.org might shed some light on what the error means if you want to try that. mohit.kaushik wrote: > There errors are shown in logs of Hadoop namenode and slaves... > > *Namenode**log* > /2015-08-05 12:05:14,518 INFO > org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment > at 391508 > 2015-08-05 12:05:14,664 INFO BlockStateChange: BLOCK* ask > 192.168.10.121:50010 to replicate blk_1073780327_39560 to datanode(s) > 192.168.10.122:50010 > 2015-08-05 12:05:14,664 INFO BlockStateChange: BLOCK* ask > 192.168.10.121:50010 to replicate blk_1073780379_39612 to datanode(s) > 192.168.10.122:50010 > 2015-08-05 12:05:24,621 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.122:50010 is added to blk_1073782847_42080 > size 134217728 > 2015-08-05 12:05:26,665 INFO BlockStateChange: BLOCK* ask > 192.168.10.121:50010 to replicate blk_1073780611_39844 to datanode(s) > 192.168.10.122:50010 > 2015-08-05 12:05:27,232 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.122:50010 is added to blk_1073793941_53178 > size 134217728 > 2015-08-05 12:05:27,950 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.122:50010 is added to blk_1073783859_43092 > size 134217728 > 2015-08-05 12:05:28,798 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.122:50010 is added to blk_1073793387_52620 > size 22496 > 2015-08-05 12:05:29,666 INFO BlockStateChange: BLOCK* ask > 192.168.10.123:50010 to replicate blk_1073780678_39911 to datanode(s) > 192.168.10.121:50010 > 2015-08-05 12:05:29,666 INFO BlockStateChange: BLOCK* ask > 192.168.10.121:50010 to replicate blk_1073780682_39915 to datanode(s) > 192.168.10.122:50010 > 2015-08-05 12:05:32,002 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.122:50010 is added to > blk_1073796582_55826{UCState=UNDER_CONSTRUCTION, truncateBlock=null, > primaryNodeIndex=-1, > replicas=[ReplicaUC[[DISK]DS-896dada5-52c0-4a69-beed-dfbc5d437fc6:NORMAL:192.168.10.123:50010|RBW], > ReplicaUC[[DISK]DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11:NORMAL:192.168.10.121:50010|RBW], > ReplicaUC[[DISK]DS-188489f9-89d3-40bd-9d20-9db358d644c9:NORMAL:192.168.10.122:50010|RBW]]} > size 0 > 2015-08-05 12:05:32,072 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.121:50010 is added to > blk_1073796582_55826{UCState=UNDER_CONSTRUCTION, truncateBlock=null, > primaryNodeIndex=-1, > replicas=[ReplicaUC[[DISK]DS-896dada5-52c0-4a69-beed-dfbc5d437fc6:NORMAL:192.168.10.123:50010|RBW], > ReplicaUC[[DISK]DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11:NORMAL:192.168.10.121:50010|RBW], > ReplicaUC[[DISK]DS-188489f9-89d3-40bd-9d20-9db358d644c9:NORMAL:192.168.10.122:50010|RBW]]} > size 0 > 2015-08-05 12:05:32,129 INFO BlockStateChange: BLOCK* addStoredBlock: > blockMap updated: 192.168.10.123:50010 is added to > blk_1073796582_55826{UCState=UNDER_CONSTRUCTION, truncateBlock=null, > primaryNodeIndex=-1, > replicas=[ReplicaUC[[DISK]DS-896dada5-52c0-4a69-beed-dfbc5d437fc6:NORMAL:192.168.10.123:50010|RBW], > ReplicaUC[[DISK]DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11:NORMAL:192.168.10.121:50010|RBW], > ReplicaUC[[DISK]DS-188489f9-89d3-40bd-9d20-9db358d644c9:NORMAL:192.168.10.122:50010|RBW]]} > size 0/................and more > > *Slave log **(too many)* > /k_1073794728_53972 on DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because > the block scanner is disabled. > 2015-08-05 11:50:30,438 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794738_53982 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled. > 2015-08-05 11:50:31,024 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794728_53972 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled. > 2015-08-05 11:50:31,027 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794738_53982 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled. > 2015-08-05 11:50:31,095 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled. > 2015-08-05 11:50:31,105 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled. > 2015-08-05 11:50:31,136 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled. > 2015-08-05 11:50:31,136 INFO > org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning > suspicious block > BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on > DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is > disabled./ > > > I am using locality groups so its a *NEED* to compact tables.... plz > explain how can I get rid of suspicious blocks. > > Thanks > > On 08/05/2015 10:53 AM, mohit.kaushik wrote: >> yes, One of my datanode was down because disk was detached for some >> time and tserver was lost for that node but Its Up and running again. >> >> fsck show that the file system is healthy. but with so many msgs >> reporting under replicated blocks while my replication factor is 3 it >> shows required is 5. >> >> //user/root/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A0000d29.rf+F0000d28.rf: >> Under replicated >> BP-2102462487-192.168.10.124-1436956492274:blk_1073796198_55442. >> Target Replicas is 5 but found 3 replica(s)./// >> >> Thanks & Regards >> Mohit Kaushik >> >> On 08/04/2015 09:18 PM, John Vines wrote: >>> It looks like an hdfs issue. Did a datanode go down? Did you turn >>> replication down to 1? The combination of those two errors would >>> definitely cause the problems your seeing as the latter disables any >>> sort of robustness of the underlying filesystem. >>> >>> On Tue, Aug 4, 2015 at 8:10 AM mohit.kaushik >>> > wrote: >>> >>> On 08/04/2015 05:35 PM, mohit.kaushik wrote: >>>> Hello All, >>>> >>>> I am using Apache Accumulo-1.6.3 with Apache Hadoop-2.7.0 on a 3 >>>> node cluster. when I give compact command from the shell it >>>> gives the folloing warn. >>>> >>>> root@orkash testScan> compact -w >>>> 2015-08-04 17:10:52,702 [Shell.audit] INFO : root@orkash >>>> testScan> compact -w >>>> 2015-08-04 17:10:52,706 [shell.Shell] INFO : Compacting table ... >>>> 2015-08-04 17:12:53,986 [impl.ThriftTransportPool] *WARN : >>>> Thread "shell" stuck on IO to orkash4:9999 (0) for at least >>>> 120034 ms* >>>> >>>> >>>> Tablet Servers show problem regarding a data block. which is >>>> something like HDFS-8659 >>>> >>>> >>>> /2015-08-04 15:00:27,825 [hdfs.DFSClient] WARN : Failed to >>>> connect to /192.168.10.121:50010 >>>> for block, add to deadNodes and continue. java.io.IOException: >>>> Got error, status message opReadBlock >>>> BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911 >>>> received exception >>>> org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: >>>> Replica not found for >>>> BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911, >>>> for OP_READ_BLOCK, self=/192.168.10.121:38752 >>>> , remote=/192.168.10.121:50010 >>>> , for file >>>> /accumulo/tables/h/t-000016s/F000016t.rf, for pool >>>> BP-2102462487-192.168.10.124-1436956492274 block 1073780678_39911// >>>> //java.io.IOException: Got error, status message opReadBlock >>>> BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911 >>>> received exception >>>> org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: >>>> Replica not found for >>>> BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911, >>>> for OP_READ_BLOCK, self=/192.168.10.121:38752 >>>> , remote=/192.168.10.121:50010 >>>> , for file >>>> /accumulo/tables/h/t-000016s/F000016t.rf, for pool >>>> BP-2102462487-192.168.10.124-1436956492274 block 1073780678_39911// >>>> // at >>>> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)// >>>> // at >>>> org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456)// >>>> // at >>>> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424)// >>>> // at >>>> org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:814)// >>>> // at >>>> org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:693)// >>>> // at >>>> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:352)// >>>> // at >>>> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)// >>>> // at >>>> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)// >>>> // at >>>> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)// >>>> // at >>>> java.io.DataInputStream.read(DataInputStream.java:149)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream$1.run(BoundedRangeFileInputStream.java:104)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream$1.run(BoundedRangeFileInputStream.java:100)// >>>> // at java.security.AccessController.doPrivileged(Native >>>> Method)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:100)// >>>> // at >>>> org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)// >>>> // at >>>> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)// >>>> // at >>>> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)// >>>> // at >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)// >>>> // at >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)// >>>> // at >>>> java.io.FilterInputStream.read(FilterInputStream.java:83)// >>>> // at >>>> java.io.DataInputStream.readInt(DataInputStream.java:387)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:269)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:724)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:497)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.getNext(MultiLevelIndex.java:587)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.getNextNode(MultiLevelIndex.java:593)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$IndexIterator.getNextNode(MultiLevelIndex.java:616)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$IndexIterator.next(MultiLevelIndex.java:659)// >>>> // at >>>> org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:559)/ >>>> >>>> Regards >>>> Mohit Kaushik >>>> >>>> ** >>>> >>> And Compaction never completes >>> --------------030008030209060202060608 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit


-------- Forwarded Message --------
Subject: Re: Problem during compacting a table
Date: Wed, 05 Aug 2015 11:24:28 -0400
From: Josh Elser <josh.elser@gmail.com>
Reply-To: user@accumulo.apache.org
To: user@accumulo.apache.org 


I'm not really sure what that error message means without doing more 
digging. Copying your email to user@hadoop.apache.org might shed some 
light on what the error means if you want to try that.

mohit.kaushik wrote:
> There errors are shown in logs of Hadoop namenode and slaves...
>
> *Namenode**log*
> /2015-08-05 12:05:14,518 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment
> at 391508
> 2015-08-05 12:05:14,664 INFO BlockStateChange: BLOCK* ask
> 192.168.10.121:50010 to replicate blk_1073780327_39560 to datanode(s)
> 192.168.10.122:50010
> 2015-08-05 12:05:14,664 INFO BlockStateChange: BLOCK* ask
> 192.168.10.121:50010 to replicate blk_1073780379_39612 to datanode(s)
> 192.168.10.122:50010
> 2015-08-05 12:05:24,621 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.122:50010 is added to blk_1073782847_42080
> size 134217728
> 2015-08-05 12:05:26,665 INFO BlockStateChange: BLOCK* ask
> 192.168.10.121:50010 to replicate blk_1073780611_39844 to datanode(s)
> 192.168.10.122:50010
> 2015-08-05 12:05:27,232 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.122:50010 is added to blk_1073793941_53178
> size 134217728
> 2015-08-05 12:05:27,950 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.122:50010 is added to blk_1073783859_43092
> size 134217728
> 2015-08-05 12:05:28,798 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.122:50010 is added to blk_1073793387_52620
> size 22496
> 2015-08-05 12:05:29,666 INFO BlockStateChange: BLOCK* ask
> 192.168.10.123:50010 to replicate blk_1073780678_39911 to datanode(s)
> 192.168.10.121:50010
> 2015-08-05 12:05:29,666 INFO BlockStateChange: BLOCK* ask
> 192.168.10.121:50010 to replicate blk_1073780682_39915 to datanode(s)
> 192.168.10.122:50010
> 2015-08-05 12:05:32,002 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.122:50010 is added to
> blk_1073796582_55826{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-896dada5-52c0-4a69-beed-dfbc5d437fc6:NORMAL:192.168.10.123:50010|RBW],
> ReplicaUC[[DISK]DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11:NORMAL:192.168.10.121:50010|RBW],
> ReplicaUC[[DISK]DS-188489f9-89d3-40bd-9d20-9db358d644c9:NORMAL:192.168.10.122:50010|RBW]]}
> size 0
> 2015-08-05 12:05:32,072 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.121:50010 is added to
> blk_1073796582_55826{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-896dada5-52c0-4a69-beed-dfbc5d437fc6:NORMAL:192.168.10.123:50010|RBW],
> ReplicaUC[[DISK]DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11:NORMAL:192.168.10.121:50010|RBW],
> ReplicaUC[[DISK]DS-188489f9-89d3-40bd-9d20-9db358d644c9:NORMAL:192.168.10.122:50010|RBW]]}
> size 0
> 2015-08-05 12:05:32,129 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 192.168.10.123:50010 is added to
> blk_1073796582_55826{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-896dada5-52c0-4a69-beed-dfbc5d437fc6:NORMAL:192.168.10.123:50010|RBW],
> ReplicaUC[[DISK]DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11:NORMAL:192.168.10.121:50010|RBW],
> ReplicaUC[[DISK]DS-188489f9-89d3-40bd-9d20-9db358d644c9:NORMAL:192.168.10.122:50010|RBW]]}
> size 0/................and more
>
> *Slave log **(too many)*
> /k_1073794728_53972 on DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because
> the block scanner is disabled.
> 2015-08-05 11:50:30,438 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794738_53982 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled.
> 2015-08-05 11:50:31,024 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794728_53972 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled.
> 2015-08-05 11:50:31,027 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794738_53982 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled.
> 2015-08-05 11:50:31,095 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled.
> 2015-08-05 11:50:31,105 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled.
> 2015-08-05 11:50:31,136 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled.
> 2015-08-05 11:50:31,136 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Not scanning
> suspicious block
> BP-2102462487-192.168.10.124-1436956492274:blk_1073794740_53984 on
> DS-896dada5-52c0-4a69-beed-dfbc5d437fc6, because the block scanner is
> disabled./
>
>
> I am using locality groups so its a *NEED* to compact tables.... plz
> explain how can I get rid of suspicious blocks.
>
> Thanks
>
> On 08/05/2015 10:53 AM, mohit.kaushik wrote:
>> yes, One of my datanode was down because disk was detached for some
>> time and tserver was lost for that node but Its Up and running again.
>>
>> fsck show that the file system is healthy. but with so many msgs
>> reporting under replicated blocks while my replication factor is 3 it
>> shows required is 5.
>>
>> //user/root/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A0000d29.rf+F0000d28.rf:
>> Under replicated
>> BP-2102462487-192.168.10.124-1436956492274:blk_1073796198_55442.
>> Target Replicas is 5 but found 3 replica(s).///
>>
>> Thanks & Regards
>> Mohit Kaushik
>>
>> On 08/04/2015 09:18 PM, John Vines wrote:
>>> It looks like an hdfs issue. Did a datanode go down? Did you turn
>>> replication down to 1? The combination of those two errors would
>>> definitely cause the problems your seeing as the latter disables any
>>> sort of robustness of the underlying filesystem.
>>>
>>> On Tue, Aug 4, 2015 at 8:10 AM mohit.kaushik
>>> <mohit.kaushik@orkash.com <mailto:mohit.kaushik@orkash.com>> wrote:
>>>
>>>     On 08/04/2015 05:35 PM, mohit.kaushik wrote:
>>>>     Hello All,
>>>>
>>>>     I am using Apache Accumulo-1.6.3 with Apache Hadoop-2.7.0 on a 3
>>>>     node cluster. when I give compact command from the shell it
>>>>     gives the folloing warn.
>>>>
>>>>     root@orkash testScan> compact -w
>>>>     2015-08-04 17:10:52,702 [Shell.audit] INFO : root@orkash
>>>>     testScan> compact -w
>>>>     2015-08-04 17:10:52,706 [shell.Shell] INFO : Compacting table ...
>>>>     2015-08-04 17:12:53,986 [impl.ThriftTransportPool] *WARN :
>>>>     Thread "shell" stuck on IO  to orkash4:9999 (0) for at least
>>>>     120034 ms*
>>>>
>>>>
>>>>     Tablet Servers show problem regarding a data block. which is
>>>>     something like HDFS-8659
>>>>     <https://issues.apache.org/jira/browse/HDFS-8659>
>>>>
>>>>     /2015-08-04 15:00:27,825 [hdfs.DFSClient] WARN : Failed to
>>>>     connect to /192.168.10.121:50010 <http://192.168.10.121:50010>
>>>>     for block, add to deadNodes and continue. java.io.IOException:
>>>>     Got error, status message opReadBlock
>>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911
>>>>     received exception
>>>>     org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException:
>>>>     Replica not found for
>>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911,
>>>>     for OP_READ_BLOCK, self=/192.168.10.121:38752
>>>>     <http://192.168.10.121:38752>, remote=/192.168.10.121:50010
>>>>     <http://192.168.10.121:50010>, for file
>>>>     /accumulo/tables/h/t-000016s/F000016t.rf, for pool
>>>>     BP-2102462487-192.168.10.124-1436956492274 block 1073780678_39911//
>>>>     //java.io.IOException: Got error, status message opReadBlock
>>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911
>>>>     received exception
>>>>     org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException:
>>>>     Replica not found for
>>>>     BP-2102462487-192.168.10.124-1436956492274:blk_1073780678_39911,
>>>>     for OP_READ_BLOCK, self=/192.168.10.121:38752
>>>>     <http://192.168.10.121:38752>, remote=/192.168.10.121:50010
>>>>     <http://192.168.10.121:50010>, for file
>>>>     /accumulo/tables/h/t-000016s/F000016t.rf, for pool
>>>>     BP-2102462487-192.168.10.124-1436956492274 block 1073780678_39911//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:814)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:693)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:352)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)//
>>>>     //        at
>>>>     org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)//
>>>>     //        at
>>>>     java.io.DataInputStream.read(DataInputStream.java:149)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream$1.run(BoundedRangeFileInputStream.java:104)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream$1.run(BoundedRangeFileInputStream.java:100)//
>>>>     //        at java.security.AccessController.doPrivileged(Native
>>>>     Method)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.bcfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:100)//
>>>>     //        at
>>>>     org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)//
>>>>     //        at
>>>>     org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)//
>>>>     //        at
>>>>     org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)//
>>>>     //        at
>>>>     java.io.BufferedInputStream.fill(BufferedInputStream.java:235)//
>>>>     //        at
>>>>     java.io.BufferedInputStream.read(BufferedInputStream.java:254)//
>>>>     //        at
>>>>     java.io.FilterInputStream.read(FilterInputStream.java:83)//
>>>>     //        at
>>>>     java.io.DataInputStream.readInt(DataInputStream.java:387)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:269)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:724)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:497)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.getNext(MultiLevelIndex.java:587)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.getNextNode(MultiLevelIndex.java:593)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$IndexIterator.getNextNode(MultiLevelIndex.java:616)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$IndexIterator.next(MultiLevelIndex.java:659)//
>>>>     //        at
>>>>     org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:559)/
>>>>
>>>>     Regards
>>>>     Mohit Kaushik
>>>>
>>>>     **
>>>>
>>>     And Compaction never completes
>>>


--------------030008030209060202060608--