hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anurag Phadke" <Anurag.Pha...@fox.com>
Subject nfs to hadoop dfs copy timeout
Date Thu, 22 May 2008 22:00:01 GMT
Hello,
We are running a nightly cron job using hadoop-0.14.4-0.1 that copies
files from "nfs" to hadoop dfs. There are around 40 files with an
average size of 3.5GB/file, the peak being 9.1GB. 
 
The job crapped out today morning with the following stack trace:
 
08/05/22 01:55:39 WARN fs.DFSClient: Error while writing.
java.net.SocketException: Connection reset
        at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:
1656)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.jav
a:1610)
        at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.ja
va:140)
        at
org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
        at
org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
        at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutput
Stream.java:39)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:258)
        at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:248)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:133)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:126)
        at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
        at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
        at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:115)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:1220)
        at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1333)
copyFromLocal: Connection reset
dfs_copy_logs_failed
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current
                                 Dload  Upload   Total   Spent    Left
Speed
100   173    0   173    0     0    347      0 --:--:-- --:--:-- --:--:--
625

Any tips on how this can be fixed?
 
-ANurag
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message