hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomasz Nykiel (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2362) More Improvements on NameNode Scalability
Date Sun, 23 Oct 2011 23:46:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133781#comment-13133781
] 

Tomasz Nykiel commented on HDFS-2362:
-------------------------------------

I have a general question regarding the JUnit test. I observed a bizarre behaviour.
When running some tests, they fail due to:

Cannot lock storage /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1.
The directory is already locked - so the MiniHDFSCluster cannot initialize properly.

I observed on my local machine that sometimes, probably after running some previous tests
which fail, the datanode data directory is left in some strange state:

For instance:
hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1 listing shows that the files
inside (e.g., "current", "in_use.lock" - as far as I remember the name) are listed with "?"
for all permissions, also the file owner and group are shown to be "?". I am not sure why
this thing is happening, I don't think that this is an issue with any of my patches, as for
example org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool.testDfsAdminDeleteBlockPool
was failing due to this reason previously.

                
> More Improvements on NameNode Scalability
> -----------------------------------------
>
>                 Key: HDFS-2362
>                 URL: https://issues.apache.org/jira/browse/HDFS-2362
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>
> This jira acts as an umbrella jira to track all the improvements we've done recently
to improve Namenode's performance, responsiveness, and hence scalability. Those improvements
include:
> 1. Incremental block reports (HDFS-395)
> 2. BlockManager.reportDiff optimization for processing block reports (HDFS-2477)
> 3. Upgradable lock to allow simutaleous read operation while reportDiff is in progress
in processing block reports (HDFS-2490)
> 4. More CPU efficient data structure for under-replicated/over-replicated/invalidate
blocks (HDFS-2476)
> 5. Increase granularity of write operations in ReplicationMonitor thus reducing contention
for write lock
> 6. Support variable block sizes
> 7. Release RPC handlers while waiting for edit log is synced to disk
> 8. Reduce network traffic pressure to the master rack where NN is located by lowering
read priority of the replicas on the rack
> 9. A standalone KeepAlive heartbeat thread
> 10. Reduce Multiple traversals of path directory to one for most namespace manipulations
> 11. Move logging out of write lock section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message