Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 27 Jul 2017 09:06:00 +0000 (UTC)
From: "Eric Lei (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13090459.1501146325000.25755.1501146360127@Atlassian.JIRA>
In-Reply-To: <JIRA.13090459.1501146325000@Atlassian.JIRA>
References: <JIRA.13090459.1501146325000@Atlassian.JIRA> <JIRA.13090459.1501146325998@jira-lw-us.apache.org>
Subject: [jira] [Created] (HADOOP-14691) Shell command "hadoop fs -put"
 multiple close problem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 27 Jul 2017 09:06:11 -0000

Eric Lei created HADOOP-14691:
---------------------------------

             Summary: Shell command "hadoop fs -put" multiple close problem
                 Key: HADOOP-14691
                 URL: https://issues.apache.org/jira/browse/HADOOP-14691
             Project: Hadoop Common
          Issue Type: Bug
          Components: common
    Affects Versions: 2.7.3
         Environment: CentOS7.0
JDK1.8.0_121
hadoop2.7.3
            Reporter: Eric Lei


1.=09Bug description
Shell command =E2=80=9CHadoop fs -put=E2=80=9D is a write operation. In thi=
s process, FSDataOutputStream is new created and closed lastly. Finally, th=
e FSDataOutputStream.close() calls the close method in HDFS to end up the c=
ommunication of this write process between the server and client.
With the command =E2=80=9CHadoop fs -put=E2=80=9D, for each created FSDataO=
utputStream object, FSDataOutputStream.close() is called twice, which means=
 the close method, in the underlying distributed file system, is called twi=
ce. This is the error, that=E2=80=99s because the communication process, fo=
r example socket, might be repeated shut down. Unfortunately, if there is n=
o error protection for the socket, there might be error for the socket in t=
he second close.=20
Further, we think a correct upper file system design should keep the one ti=
me close principle. It means that each creation of underlying distributed f=
ile system object should correspond with close only once.=20
For the command =E2=80=9CHadoop fs -put=E2=80=9D, there are double close as=
 follows:
a.=09The first close process:
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutput=
Stream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:10=
6)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:61)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.write=
StreamToFile(CommandWithDestination.java:466)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(Com=
mandWithDestination.java:391)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(Comma=
ndWithDestination.java:328)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWit=
hDestination.java:263)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWit=
hDestination.java:248)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(Co=
mmandWithDestination.java:243)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(Comma=
ndWithDestination.java:220)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommand=
s.java:267)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

b.=09The second close process:
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutput=
Stream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:10=
6)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:261)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.write=
StreamToFile(CommandWithDestination.java:468)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(Com=
mandWithDestination.java:391)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(Comma=
ndWithDestination.java:328)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWit=
hDestination.java:263)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWit=
hDestination.java:248)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(Co=
mmandWithDestination.java:243)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(Comma=
ndWithDestination.java:220)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommand=
s.java:267)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

2.=09Code analysis
void writeStreamToFile(InputStream in, PathData target, boolean lazyPersist=
) throws IOException {
    FSDataOutputStream out =3D null;

    try {
        out =3D this.create(target, lazyPersist);
        IOUtils.copyBytes(in, out, this.getConf(), true);
    } finally {
        IOUtils.closeStream(out);
    }

}

public static void copyBytes(InputStream in, OutputStream out, int buffSize=
, boolean close) throws IOException {
    try {
        copyBytes(in, out, buffSize);
        if(close) {
            out.close();
            out =3D null;
            in.close();
            in =3D null;
        }
    } finally {
        if(close) {
            closeStream(out);
            closeStream(in);
        }
    }
}

The problem is caused by these two methods.=20
FSDataOutputStream out is defined in the method writeStreamToFile, and it i=
s a input parameter of copyBytes. When an object is as an input parameter o=
f a API, it equals to a new reference in this API. It points to the same ad=
dress with the external API. In the API copyBytes(), the code =E2=80=9Cout =
=3D null=E2=80=9D means that within this API, the reference object out poin=
ts to the address null, but it doesn=E2=80=99t change the external referenc=
e object=E2=80=99s address value. As a result, =E2=80=9Cout =3D null=E2=80=
=9D is only valid for =E2=80=9CcloseStream(out)=E2=80=9D in copyBytes(), so=
 for=E2=80=9D IOUtils.closeStream(out)=E2=80=9D  in writeStreamToFile (), o=
ut is not null in the normal process. That=E2=80=99s why there are double c=
lose calls.

3.=09Solution
a.=09Suggest solution:
For the input object in an API, it is not suitable to set it as null within=
 this API. It should to be set in its defined API. And for this problem, th=
e parameter =E2=80=9CBoolean close=E2=80=9D seems unsuitable. The close ope=
ration should be written in the definition API of =E2=80=9CFSDataOutputStre=
am out=E2=80=9D.=20
This problem is also occurred for =E2=80=9CFSDataInputStream in=E2=80=9D wh=
ich is defined in copyFileToTarget().
This modification solution could solve this problem thoroughly. And it make=
s the design to be more suitable for the file system.
However, with this modification solution, there are many related code for t=
his process. Only the =E2=80=9Cpublic static void copyBytes(InputStream in,=
 OutputStream out, int buffSize, boolean close)=E2=80=9D, there are 16 call=
s of this method. Considering the call path through =E2=80=9CcopyFileToTarg=
et()=E2=80=9D to =E2=80=9CcopyBytes()=E2=80=9D and related tests to submit =
the patch, it will be a huge work for us. We use the above workaround solut=
ion to test our idea.

b.=09Our workaround solution:
Our solution is to add a return value to record the close status of out and=
 in. If it has been closed, it won=E2=80=99t be re-closed in the external A=
PI.=20
This solution could make sure the out and in are closed only once easily.
The details could refer to the patch. It shows the least modification for t=
his =E2=80=9CHadoop fs -put=E2=80=9D command with this solution.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org