hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: hdfs fsck -locations
Date Sat, 25 Jan 2014 01:49:10 GMT
The right syntax is to use "-files -blocks -locations", so it drills
down all the way. You are not "missing" a feature - this has existed
for as long as I've known HDFS.

In Rodrigo's output, he's seeing a BlockPool ID, which is not
equivalent to a location, but just carries an IP in it for
identification purposes.

On Sat, Jan 25, 2014 at 3:53 AM, Mark Kerzner <mark.kerzner@shmsoft.com> wrote:
> Yes, Rodrigo,
>
> that's what I was looking for. So in my install I somehow don't have it at
> all. Was asked by my students, so I got the answer.
>
> Mark
>
>
> On Fri, Jan 24, 2014 at 4:00 PM, Nascimento, Rodrigo
> <Rodrigo.Nascimento@netapp.com> wrote:
>>
>> Mark,
>>
>> there we go ;-)
>>
>> Rodrigo Nascimento
>> Systems Engineer @ Brazil
>> Mobile +55 11 991.873.810
>>
>> Sent from my iPhone
>>
>> Begin forwarded message:
>>
>> From: "Nascimento, Rodrigo" <Rodrigo.Nascimento@netapp.com>
>> Date: 24 de janeiro de 2014 15:59:33 BRST
>> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Subject: RE: hdfs fsck -locations
>>
>> Hi Mark,
>>
>> It is a sample from my sandbox. Your question is about the part that is in
>> RED at the output below, right?
>>
>> [root@sandbox ~]# hdfs fsck /user/ambari-qa/passwd  -locations
>> Connecting to namenode via http://sandbox.hortonworks.com:50070
>> FSCK started by root (auth:SIMPLE) from /172.16.13.30 for path
>> /user/ambari-qa/passwd at Fri Jan 24 09:53:43 PST 2014
>> .
>> /user/ambari-qa/passwd:  Under replicated
>> BP-1578958328-10.0.2.15-1382306880516:blk_1073742464_1640. Target Replicas
>> is 3 but found 1 replica(s).
>> Status: HEALTHY
>>  Total size:    1708 B
>>  Total dirs:    0
>>  Total files:    1
>>  Total symlinks:        0
>>  Total blocks (validated):    1 (avg. block size 1708 B)
>>  Minimally replicated blocks:    1 (100.0 %)
>>  Over-replicated blocks:    0 (0.0 %)
>>  Under-replicated blocks:    1 (100.0 %)
>>  Mis-replicated blocks:        0 (0.0 %)
>>  Default replication factor:    3
>>  Average block replication:    1.0
>>  Corrupt blocks:        0
>>  Missing replicas:        2 (66.666664 %)
>>  Number of data-nodes:        1
>>  Number of racks:        1
>> FSCK ended at Fri Jan 24 09:53:43 PST 2014 in 1 milliseconds
>>
>>
>> The filesystem under path '/user/ambari-qa/passwd' is HEALTHY
>> [root@sandbox ~]#
>>
>> Rod Nascimento
>>
>> ________________________________
>> From: Nascimento, Rodrigo [Rodrigo.Nascimento@netapp.com]
>> Sent: Friday, January 24, 2014 3:34 PM
>> To: user@hadoop.apache.org
>> Subject: RE: hdfs fsck -locations
>>
>> I’m not seeing locations flag yet.
>>
>>
>>
>> Rod Nascimento
>>
>> Systems Engineer @ Brazil
>>
>>
>>
>> People don’t buy WHAT you do. They buy WHY you do it.
>>
>>
>>
>> From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
>> Sent: Friday, January 24, 2014 3:16 PM
>> To: Hadoop User
>> Subject: Re: hdfs fsck -locations
>>
>>
>>
>> Sorry, did not copy the full command
>>
>>
>>
>> hdfs fsck /user/mark/data/word_count.csv -locations
>>
>> Connecting to namenode via http://mark-7:50070
>>
>> FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path
>> /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014
>>
>> .Status: HEALTHY
>>
>>  Total size:       7217 B
>>
>>  Total dirs:       0
>>
>>  Total files:      1
>>
>>  Total blocks (validated):        1 (avg. block size 7217 B)
>>
>>  Minimally replicated blocks:  1 (100.0 %)
>>
>>  Over-replicated blocks:          0 (0.0 %)
>>
>>  Under-replicated blocks:        0 (0.0 %)
>>
>>  Mis-replicated blocks:                        0 (0.0 %)
>>
>>  Default replication factor:      1
>>
>>  Average block replication:     1.0
>>
>>  Corrupt blocks:                      0
>>
>>  Missing replicas:                     0 (0.0 %)
>>
>>  Number of data-nodes:                      1
>>
>>  Number of racks:                   1
>>
>> FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds
>>
>>
>>
>>
>>
>> The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY
>>
>>
>>
>>
>>
>> On Fri, Jan 24, 2014 at 11:08 AM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Sorry, but what was the question? I also do not see a locations option
>> flag.
>>
>> On Jan 24, 2014 7:17 PM, "Mark Kerzner" <mark.kerzner@shmsoft.com> wrote:
>>
>> Here is an example
>>
>>
>>
>>  hdfs fsck /user/mark/data/word_count.csv
>>
>> Connecting to namenode via http://mark-7:50070
>>
>> FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path
>> /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014
>>
>> .Status: HEALTHY
>>
>>  Total size: 7217 B
>>
>>  Total dirs: 0
>>
>>  Total files: 1
>>
>>  Total blocks (validated): 1 (avg. block size 7217 B)
>>
>>  Minimally replicated blocks: 1 (100.0 %)
>>
>>  Over-replicated blocks: 0 (0.0 %)
>>
>>  Under-replicated blocks: 0 (0.0 %)
>>
>>  Mis-replicated blocks: 0 (0.0 %)
>>
>>  Default replication factor: 1
>>
>>  Average block replication: 1.0
>>
>>  Corrupt blocks: 0
>>
>>  Missing replicas: 0 (0.0 %)
>>
>>  Number of data-nodes: 1
>>
>>  Number of racks: 1
>>
>> FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds
>>
>>
>>
>>
>>
>> On Fri, Jan 24, 2014 at 4:34 AM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hi Mark,
>>
>> Yes, the locations are shown as IP.
>>
>>
>> On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner <mark.kerzner@shmsoft.com>
>> wrote:
>> > Hi,
>> >
>> > hdfs fsck -locations
>> >
>> > is supposed to show every block with its location? Is location the ip of
>> > the
>> > datanode?
>> >
>> > Thank you,
>> > Mark
>>
>>
>> --
>> Harsh J
>>
>>
>>
>>
>
>



-- 
Harsh J

Mime
View raw message