Return-Path: 
 <user-return-4344-apmail-accumulo-user-archive=accumulo.apache.org@accumulo.apache.org>
X-Original-To: apmail-accumulo-user-archive@www.apache.org
Delivered-To: apmail-accumulo-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0007C101F3
	for <apmail-accumulo-user-archive@www.apache.org>;
 Tue, 27 May 2014 22:20:23 +0000 (UTC)
Received: (qmail 37765 invoked by uid 500); 27 May 2014 22:20:23 -0000
Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org
Received: (qmail 37711 invoked by uid 500); 27 May 2014 22:20:23 -0000
Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@accumulo.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@accumulo.apache.org>
List-Post: <mailto:user@accumulo.apache.org>
List-Id: <user.accumulo.apache.org>
Reply-To: user@accumulo.apache.org
Delivered-To: mailing list user@accumulo.apache.org
Received: (qmail 37703 invoked by uid 99); 27 May 2014 22:20:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 May 2014 22:20:23 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of roshanp@gmail.com designates
 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-wg0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 May 2014 22:20:18 +0000
Received: by mail-wg0-f48.google.com with SMTP id k14so5885472wgh.31
        for <user@accumulo.apache.org>; Tue, 27 May 2014 15:19:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=R/28XB/oUvISAWV0diys8UxLFPomaTKA4SRF+GobbUY=;
        b=ogCq6x3z1DKBUGzMQLNf1s6cH5lYycR8hCb4z1rtgGiqlO/fYIs3uf5LXqbQaV3ULc
         XLd5Sqv8KLdWb+ybMbVSVrgn2SzcY10s7jB5w7g4Vzt62B7tAYeHZVVZywEH1uh8wXcB
         9ttCyhi5zc7oIDoD6P2WjjGG8sICu0SzAOxSmCKnGimb1/5qGVOxvg/QDwV3gtSCncGo
         F8tgKc1PbP2/UbhILEcBUesmBKNl3O+7ZST/f6FFdS5kuPrBBt5pLhMH68FzSVEUUadt
         +vyaBIn8sn1OgbOunXQtTbdQaRU7K1d/KK33eqBHIwYTsxYInTOHbGwtrX/OPFy7Med+
         rRHA==
MIME-Version: 1.0
X-Received: by 10.194.7.232 with SMTP id m8mr46051465wja.47.1401229195343;
 Tue, 27 May 2014 15:19:55 -0700 (PDT)
Received: by 10.194.34.8 with HTTP; Tue, 27 May 2014 15:19:55 -0700 (PDT)
Date: Tue, 27 May 2014 18:19:55 -0400
Message-ID: 
 <CA+zmODg3Xb+WuQUeuQbWNX-VSy8CbMJj-Zw4R7r8v5vbLuyHNA@mail.gmail.com>
Subject: HDFS Closed Channel
From: Roshan Punnoose <roshanp@gmail.com>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=047d7b5d327c7000ec04fa691883
X-Virus-Checked: Checked by ClamAV on apache.org

--047d7b5d327c7000ec04fa691883
Content-Type: text/plain; charset=UTF-8

Has anyone seen a case were a large parallel scan in mapreduce causes HDFS
to close on the tablets? My accumulo instance (version 1.5.1) seems to be
dying during an intensive mapreduce job. These are the only errors I see in
the tablet server logs:

2014-05-27 10:59:34,822 [log.DfsLogger] ERROR:
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1320)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
	at org.apache.accumulo.server.logger.LogFileKey.write(LogFileKey.java:108)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.write(DfsLogger.java:435)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logFileData(DfsLogger.java:448)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logManyTablets(DfsLogger.java:479)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.log(DfsLogger.java:440)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger$5.write(TabletServerLogger.java:345)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:273)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(TabletServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:744)
2014-05-27 10:59:34,826 [log.DfsLogger] WARN : Exception syncing
java.lang.reflect.InvocationTargetException
2014-05-27 10:59:34,827 [log.TabletServerLogger] ERROR: Unexpected
error writing to log, retrying attempt 1
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LoggerOperation.await(DfsLogger.java:178)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:279)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(TabletServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LogSyncingTask.run(DfsLogger.java:118)
	... 1 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1635)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1618)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1601)
	at org.apache.hadoop.fs.FSDataOutputStream.hsync(FSDataOutputStream.java:137)
	... 5 more
2014-05-27 10:59:34,881 [log.DfsLogger] ERROR:
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1320)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
	at org.apache.accumulo.server.logger.LogFileKey.write(LogFileKey.java:108)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.write(DfsLogger.java:435)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logFileData(DfsLogger.java:448)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logManyTablets(DfsLogger.java:479)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.log(DfsLogger.java:440)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger$5.write(TabletServerLogger.java:345)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:273)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(TabletServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:744)
2014-05-27 10:59:34,883 [log.DfsLogger] WARN : Exception syncing
java.lang.reflect.InvocationTargetException
2014-05-27 10:59:34,883 [log.TabletServerLogger] ERROR: Unexpected
error writing to log, retrying attempt 1
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LoggerOperation.await(DfsLogger.java:178)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:279)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(TabletServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LogSyncingTask.run(DfsLogger.java:118)
	... 1 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1635)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1618)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1601)
	at org.apache.hadoop.fs.FSDataOutputStream.hsync(FSDataOutputStream.java:137)
	... 5 more
2014-05-27 10:59:34,885 [log.DfsLogger] ERROR:
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1320)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
	at org.apache.accumulo.server.logger.LogFileKey.write(LogFileKey.java:108)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.write(DfsLogger.java:435)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logFileData(DfsLogger.java:448)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logManyTablets(DfsLogger.java:479)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.log(DfsLogger.java:440)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger$5.write(TabletServerLogger.java:345)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:273)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(TabletServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:744)
2014-05-27 10:59:34,888 [log.DfsLogger] WARN : Exception syncing
java.lang.reflect.InvocationTargetException
2014-05-27 10:59:34,889 [log.TabletServerLogger] ERROR: Unexpected
error writing to log, retrying attempt 1
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LoggerOperation.await(DfsLogger.java:178)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:279)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(TabletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(TabletServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LogSyncingTask.run(DfsLogger.java:118)
	... 1 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1635)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1618)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1601)
	at org.apache.hadoop.fs.FSDataOutputStream.hsync(FSDataOutputStream.java:137)
	... 5 more

--047d7b5d327c7000ec04fa691883
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Has anyone seen a case were a large parallel scan in mapre=
duce causes HDFS to close on the tablets? My accumulo instance (version 1.5=
.1) seems to be dying during an intensive mapreduce job. These are the only=
 errors I see in the tablet server logs:<div>
<br></div><div><pre style=3D"margin-top:0px;margin-bottom:0px;border:0px">2=
014-05-27 10:59:34,822 [log.DfsLogger] ERROR: java.nio.channels.ClosedChann=
elException
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java=
:1320)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutpu=
tStream.java:59)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
	at org.apache.accumulo.server.logger.LogFileKey.write(LogFileKey.java:108)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.write(DfsLogger.j=
ava:435)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logFileData(DfsLo=
gger.java:448)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logManyTablets(Df=
sLogger.java:479)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.log(DfsLogger.jav=
a:440)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger$5.write(=
TabletServerLogger.java:345)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:273)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(Tabl=
etServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandle=
r.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWra=
p.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TSe=
rverUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(A=
bstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run=
(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.ja=
va:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:=
34)
	at java.lang.Thread.run(Thread.java:744)
2014-05-27 10:59:34,826 [log.DfsLogger] WARN : Exception syncing java.lang.=
reflect.InvocationTargetException
2014-05-27 10:59:34,827 [log.TabletServerLogger] ERROR: Unexpected error wr=
iting to log, retrying attempt 1
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LoggerOperation.a=
wait(DfsLogger.java:178)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:279)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(Tabl=
etServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandle=
r.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWra=
p.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TSe=
rverUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(A=
bstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run=
(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.ja=
va:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:=
34)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LogSyncingTask.ru=
n(DfsLogger.java:118)
	... 1 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java=
:1635)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1618)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1601)
	at org.apache.hadoop.fs.FSDataOutputStream.hsync(FSDataOutputStream.java:1=
37)
	... 5 more
2014-05-27 10:59:34,881 [log.DfsLogger] ERROR: java.nio.channels.ClosedChan=
nelException
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java=
:1320)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutpu=
tStream.java:59)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
	at org.apache.accumulo.server.logger.LogFileKey.write(LogFileKey.java:108)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.write(DfsLogger.j=
ava:435)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logFileData(DfsLo=
gger.java:448)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logManyTablets(Df=
sLogger.java:479)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.log(DfsLogger.jav=
a:440)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger$5.write(=
TabletServerLogger.java:345)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:273)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(Tabl=
etServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandle=
r.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWra=
p.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TSe=
rverUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(A=
bstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run=
(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.ja=
va:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:=
34)
	at java.lang.Thread.run(Thread.java:744)
2014-05-27 10:59:34,883 [log.DfsLogger] WARN : Exception syncing java.lang.=
reflect.InvocationTargetException
2014-05-27 10:59:34,883 [log.TabletServerLogger] ERROR: Unexpected error wr=
iting to log, retrying attempt 1
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LoggerOperation.a=
wait(DfsLogger.java:178)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:279)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(Tabl=
etServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandle=
r.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWra=
p.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TSe=
rverUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(A=
bstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run=
(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.ja=
va:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:=
34)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LogSyncingTask.ru=
n(DfsLogger.java:118)
	... 1 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java=
:1635)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1618)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1601)
	at org.apache.hadoop.fs.FSDataOutputStream.hsync(FSDataOutputStream.java:1=
37)
	... 5 more
2014-05-27 10:59:34,885 [log.DfsLogger] ERROR: java.nio.channels.ClosedChan=
nelException
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java=
:1320)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutpu=
tStream.java:59)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
	at org.apache.accumulo.server.logger.LogFileKey.write(LogFileKey.java:108)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.write(DfsLogger.j=
ava:435)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logFileData(DfsLo=
gger.java:448)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.logManyTablets(Df=
sLogger.java:479)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger.log(DfsLogger.jav=
a:440)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger$5.write(=
TabletServerLogger.java:345)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:273)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(Tabl=
etServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandle=
r.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWra=
p.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TSe=
rverUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(A=
bstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run=
(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.ja=
va:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:=
34)
	at java.lang.Thread.run(Thread.java:744)
2014-05-27 10:59:34,888 [log.DfsLogger] WARN : Exception syncing java.lang.=
reflect.InvocationTargetException
2014-05-27 10:59:34,889 [log.TabletServerLogger] ERROR: Unexpected error wr=
iting to log, retrying attempt 1
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LoggerOperation.a=
wait(DfsLogger.java:178)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:279)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.write(Ta=
bletServerLogger.java:229)
	at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.log(Tabl=
etServerLogger.java:342)
	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandle=
r.update(TabletServer.java:1680)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWra=
p.java:63)
	at com.sun.proxy.$Proxy11.update(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2125)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Proces=
sor$update.getResult(TabletClientService.java:2109)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TSe=
rverUtils.java:155)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(A=
bstractNonblockingServer.java:478)
	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run=
(TServerUtils.java:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.ja=
va:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:=
34)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.server.tabletserver.log.DfsLogger$LogSyncingTask.ru=
n(DfsLogger.java:118)
	... 1 more
Caused by: java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java=
:1635)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1618)
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1601)
	at org.apache.hadoop.fs.FSDataOutputStream.hsync(FSDataOutputStream.java:1=
37)
	... 5 more</pre></div></div>

--047d7b5d327c7000ec04fa691883--