hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lohit vijayarenu (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3348) TestUrlStreamHandler hangs on LINUX
Date Wed, 07 May 2008 03:27:56 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594758#action_12594758
] 

lohit edited comment on HADOOP-3348 at 5/6/08 8:26 PM:
------------------------------------------------------------------

After looking a bit with hints from Raghu, looks like this is what is causing the problem.
once setUrlStreamHandle is set in JVM, opening a device file /dev/random is  opened as ChecksumFileSystem.

SecureRandom wraps up the stream returned by opening /dev/random into BufferedInputStream
and calls read to get 20 byte DIGEST to generate the seed. This read when passed through FSInputChecker.read
with a buffer size of 8K, which used to loop until we read 8K bytes. We invoke readFully()
which loops calling multiple reads until 8K buffer is filled up. /dev/random was unable to
produce random bytes so fast and hence registration of Datanode used to take forever.

      was (Author: lohit):
    After looking a bit with hints from Raghu, looks like this is what is causing the problem.
once setUrlStreamHandle is set in JVM, opening a device file /dev/random is readFully opened
as ChecksumFileSystem. 
SecureRandom wraps up the stream returned by opening /dev/random into BufferedInputStream
and invokes read to get 20 byte DIGEST to generate the seed. This read when passed through
FSInputChecker.read with a buffer size of 8K used to loop until we read 8K bytes. We invoke
readFully() which loops calling multiple reads until 8K buffer is filled up. /dev/random was
unable to produce random bytes so fast and hence registration of Datanode used to take forever.
  
> TestUrlStreamHandler hangs on LINUX
> -----------------------------------
>
>                 Key: HADOOP-3348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3348
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0
>         Environment: LINUX 2.6.9
>            Reporter: lohit vijayarenu
>         Attachments: Datanode_jstack.txt
>
>
> TestUrlStreamHandler sets setURLStreamHandlerFactory as
> {noformat}
> FsUrlStreamHandlerFactory factory =
>         new org.apache.hadoop.fs.FsUrlStreamHandlerFactory();
>     java.net.URL.setURLStreamHandlerFactory(factory);
> {noformat}
> After this, MiniDFSCluster seems to hang while Datanodes tries to register in setNewStorageID,
specifically at
> {noformat}
> rand = SecureRandom.getInstance("SHA1PRNG").nextInt(Integer.MAX_VALUE);
> {noformat}
> jstack output shows that the main thread is stuck in RawLocalFileSystem$LocalFSFileInputStream.read
> (Attaching the jstack)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message