hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2486) Review issues with UnderReplicatedBlocks
Date Wed, 14 Dec 2011 15:05:30 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169420#comment-13169420
] 

Uma Maheswara Rao G commented on HDFS-2486:
-------------------------------------------

Hi Steve,

I just verified your points here.

{quote}
remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there
is a risk of race conditions there.
{quote}
there is an other remove method which is synchronized.
The non synchronized remove method will be invoked from 
 1) BlockManager#computeReplicationWorkForBlocks -- it has synchronized block on neededReplications
(UnderReplicatedBlocks) and called remove from this block. So, should not be a problem.
 2)synchronized UnderReplicatedBlocks#update
   This is already synchronized on UnderReplicatedBlocks, should nt be a problem.
 3)synchronized UnderReplicatedBlocks#remove
   This is also synchronized on UnderReplicatedBlocks, should not be a problem.


{quote}
some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt
to queue the blocks. As this return value is not currently possible, those checks can be removed.

{quote}
 I agree, there are unnecessary checks here, can be removed.

{quote}
The queue gives priority to blocks whose replication count is less than a third of its expected
count over those that are "normally under replicated". While this is good for ensuring that
files scheduled for large replication are replicated fast, it may not be the best strategy
for maintaining data integrity. For that it may be better to give whichever blocks have only
two replicas priority over blocks that may, for example, already have 3 out of 10 copies in
the filesystem.
{quote}
I am not sure, whether i understud your argument correctly here.
We are consideing 'very under replicated' , if block are replicated at 1:3 ratio.
considering a case 10 replication factor and 3 blocks present, should be consider as just
'under replicated'? here my understanding about your argument is yes. 
But i argue here is that, still we should consider that blocks as 'very under replicated'
because, user set the replication factor as 10 means, that files might be very important/valuable
info. Please corect me if i understud wrongly about this point.

Thanks
Uma

                
> Review issues with UnderReplicatedBlocks
> ----------------------------------------
>
>                 Key: HDFS-2486
>                 URL: https://issues.apache.org/jira/browse/HDFS-2486
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: name-node
>    Affects Versions: 0.23.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> Here are some things I've noted in the UnderReplicatedBlocks class that someone else
should review and consider if the code is correct. If not, they are easy to fix.
> remove(Block block, int priLevel) is not synchronized, and as the inner classes are not,
there is a risk of race conditions there.
> some of the code assumes that getPriority can return the value LEVEL, and if so does
not attempt to queue the blocks. As this return value is not currently possible, those checks
can be removed. 
> The queue gives priority to blocks whose replication count is less than a third of its
expected count over those that are "normally under replicated". While this is good for ensuring
that files scheduled for large replication are replicated fast, it may not be the best strategy
for maintaining data integrity. For that it may be better to give whichever blocks have only
two replicas priority over blocks that may, for example, already have 3 out of 10 copies in
the filesystem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message