hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Homan <jgho...@gmail.com>
Subject Re: Secondary Namenode doCheckpoint, FileNotFoundException
Date Thu, 06 Jan 2011 22:16:18 GMT
Is the 2NN reachable at http://10.1.1.5:50090? This is the addr the NN
is being told to grab the merged image from.  There can be problems
with VIPs, etc. if this address is not reachable.

On Thu, Jan 6, 2011 at 12:57 PM, Tyler Coffin <tcoffin@rim.com> wrote:
>
>
> 2011-01-06 15:52:00,814 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.10 cmd=open
> src=/logs/remote.log.01.20110105.2045.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:00,881 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.12 cmd=open
> src=/logs/remote.log.01.20110104.1815.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:00,891 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.9  cmd=open
> src=/logs/remote.log.01.20110105.0015.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:01,253 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.7  cmd=open
> src=/logs/remote.log.01.20110105.2035.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:01,612 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.11 cmd=open
> src=/logs/remote.log.01.20110105.2145.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:01,715 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.8  cmd=open
> src=/logs/remote.log.01.20110104.1915.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:02,030 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.13 cmd=open
> src=/logs/remote.log.01.20110105.1235.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask
> 10.11.138.52:50010 to delete  blk_5975970115065306335_564427
> blk_-4283643305091895835_564439
>
> 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask
> 10.90.37.8:50010 to delete  blk_7719330829554745618_564439
>
> 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask
> 10.11.138.73:50010 to delete  blk_5975970115065306335_564427
>
> 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask
> 10.11.138.156:50010 to delete  blk_-4283643305091895835_564439
> blk_7719330829554745618_564439
>
> 2011-01-06 15:52:04,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.90.37.13
>
> 2011-01-06 15:52:04,494 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.12 cmd=open
> src=/logs/remote.log.01.20110105.1120.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:04,529 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.7  cmd=open
> src=/logs/remote.log.01.20110104.1950.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:04,563 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.9  cmd=open
> src=/logs/remote.log.01.20110105.0215.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:05,255 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.11 cmd=open
> src=/logs/remote.log.01.20110105.2245.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:05,261 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.10 cmd=open
> src=/logs/remote.log.01.20110105.0835.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:06,960 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.8  cmd=open
> src=/logs/remote.log.01.20110104.0855.lzo       dst=null        perm=null
>
> 2011-01-06 15:52:07,147 WARN org.mortbay.log: /getimage:
> java.io.IOException: GetImage failed. java.io.IOException: Server returned
> HTTP response code: 503 for URL:
> http://10.90.37.13:50090/getimage?getimage=1
>
>         at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1313)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:58)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>
>         at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>
>         at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>
>         at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>
>         at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>
>         at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>
>        at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>
>         at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
>         at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>
>         at org.mortbay.jetty.Server.handle(Server.java:324)
>
>         at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>
>         at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>
>         at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>
>         at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
>
>
> 2011-01-06 15:52:07,522 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=admin,users,admin   ip=/10.90.37.10 cmd=open
> src=/logs/remote.log.01.20110104.2125.lzo       dst=null        perm=null
>
>
>
> The errors from the 2nn repeat frequently, I grabbed a snippet of the nn log
> from the most recent occurange (i.e. it’s not the same occurance as my
> original post)
>
>
>
>
>
>
>
> From: suresh srinivas [mailto:srini30005@gmail.com]
> Sent: January 6, 2011 15:27
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Secondary Namenode doCheckpoint, FileNotFoundException
>
>
>
> Can you add namenode log around this time?
>
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential
> information, privileged material (including material protected by the
> solicitor-client or other applicable privileges), or constitute non-public
> information. Any use of this information by anyone other than the intended
> recipient is prohibited. If you have received this transmission in error,
> please immediately reply to the sender and delete this information from your
> system. Use, dissemination, distribution, or reproduction of this
> transmission by unintended recipients is not authorized and may be unlawful.

Mime
View raw message