Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <8739014.40851276726946778.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 18:22:26 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Updated: (HDFS-1194) Secondary namenode fails to fetch the
 image from the primary
In-Reply-To: <32255624.36211276039273034.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HDFS-1194:
------------------------------

    Affects Version/s:     (was: 0.20-append)

Removing append tag, since it's unrelated.

> Secondary namenode fails to fetch the image from the primary
> ------------------------------------------------------------
>
>                 Key: HDFS-1194
>                 URL: https://issues.apache.org/jira/browse/HDFS-1194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.2, 0.21.0, 0.22.0
>         Environment: Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
> Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
> CentOS 5
>            Reporter: Dmytro Molkov
>            Assignee: Dmytro Molkov
>
> We just hit the problem described in HDFS-1024 again.
> After more investigation of the underlying problems with CancelledKeyException there are some findings:
> One of the symptoms: the transfer becomes really slow (it does 700 kb/s) when I am doing the fetch using wget. At the same time disk and network are OK since I can copy at 50 mb/s using scp.
> I was taking jstacks of the namenode while the transfer is in process and we found that every stack trace has one thread of jetty sitting in this place:
> {code}
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:452)
> 	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
> 	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> 	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
> 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> {code}
> Here is a jetty code that corresponds to this:
> {code}
> // Look for JVM bug 
>                     if (selected==0 && wait>0 && (now-before)<wait/2 && _selector.selectedKeys().size()==0)
>                     {
>                         if (_jvmBug++>5)  // TODO tune or configure this
>                         {
>                             // Probably JVM BUG!
>                             
>                             Iterator iter = _selector.keys().iterator();
>                             while(iter.hasNext())
>                             {
>                                 key = (SelectionKey) iter.next();
>                                 if (key.isValid()&&key.interestOps()==0)
>                                 {
>                                     key.cancel();
>                                 }
>                             }
>                             try
>                             {
>                                 Thread.sleep(20);  // tune or configure this
>                             }
>                             catch (InterruptedException e)
>                             {
>                                 Log.ignore(e);
>                             }
>                         } 
>                     }
> {code}
> Based on this it is obvious we are hitting a jetty workaround for a JVM bug that doesn't handle select() properly.
> There is a jetty JIRA for this http://jira.codehaus.org/browse/JETTY-937 (it actually introduces the workaround for the JVM bug that we are hitting)
> They say that the problem was fixed in 6.1.22, there is a person on that JIRA also saying that switching to using SocketConnector instead of SelectChannelConnector helped in their case.
> Since we are hitting the same bug in our world we should either adopt the newer Jetty version where there is a better workaround, but it might not help if we are still hitting that bug constantly, the workaround might be better though.
> Another approach is to switch to using SocketConnector which will eliminate the problem completely, although I am not sure what problems that will bring.
> The java version we are running is in Environment
> Any thoughts

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.