Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAOEek3O83Wjeo4XMLJ1UYAx--kSrLr8w1sV13FuaWJ4xu+y47g@mail.gmail.com>
References: <CAOEek3O83Wjeo4XMLJ1UYAx--kSrLr8w1sV13FuaWJ4xu+y47g@mail.gmail.com>
From: Manjunath A <manjunath.tech.a@gmail.com>
Date: Tue, 3 Jan 2017 23:17:52 +0530
Message-ID: <CA+K5wF_gQEyQnk0_Ew8MHVKU+LTrYooyh+dQN68k5maSQffx-Q@mail.gmail.com>
Subject: Re: Question regarding streamer thread implementation at DFSOutputStream.java
To: wang shu <uchicagoshuwang@gmail.com>
Cc: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=94eb2c05be087c991005453444ca
archived-at: Tue, 03 Jan 2017 17:48:02 -0000

--94eb2c05be087c991005453444ca
Content-Type: text/plain; charset=UTF-8

Hi Shu,

Apologies for dealy in response.

Close packet is something which is the last packet of the current block
after writing which , the current block is closed and a new pipeline is
setup for the next block.

Your understanding of the one.isLastPacketInBlock() is correct.

Regards,
Manjunath

On Wed, Dec 14, 2016 at 9:37 PM, wang shu <uchicagoshuwang@gmail.com> wrote:

> Hi all,
>
> I have a question regarding the streamer thread (Line 382-575)
>  implementation at DFSOutputStream.java(http://gr
> epcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/had
> oop-hdfs/2.7.1/org/apache/hadoop/hdfs/DFSOutputStream.java#
> DFSOutputStream.DataStreamer.run%28%29).
>
>
> What's the logic of line 468-485 and line 535-548, why it check the packet
> is the last one twice and the comments are different, one is "
> // wait for all data packets have been successfully acked" and another is
> "// wait for the close packet has been acked".  What's the close packet?
>
> My understanding of this is:
> If a packet is not the last in block, then the packet is sent directly,
> and no need to wait for ACK.
> If a packet is the last one, then it will first wait for all previous
> packet to be ACKed, and then it will send the last packet in this block(
> called close packet, I guess?), and then will need wait the last packet to
> be ACKed.
>
> Thanks a lot for your help.
>
>
> Best,
>
> Shu
>

--94eb2c05be087c991005453444ca
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Shu,<div><br></div><div>Apologies for dealy in response=
.</div><div><br></div><div>Close packet is something which is the last pack=
et of the current block after writing which , the current block is closed a=
nd a new pipeline is setup for the next block.=C2=A0</div><div><br></div><d=
iv>Your understanding of the=C2=A0<span style=3D"color:rgb(0,0,0);font-fami=
ly:&quot;dejavu sans mono&quot;;font-size:9pt">one.isLastPacketInBlock()</s=
pan>=C2=A0is correct.</div><div><br></div><div>Regards,</div><div>Manjunath=
</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On We=
d, Dec 14, 2016 at 9:37 PM, wang shu <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:uchicagoshuwang@gmail.com" target=3D"_blank">uchicagoshuwang@gmail.com</a=
>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><spa=
n style=3D"font-size:14px">Hi all,</span><div style=3D"font-size:14px"><br>=
</div><div style=3D"font-size:14px">I have a question regarding the=C2=A0st=
reamer thread (Line 382-575) =C2=A0implementation at DFSOutputStream.java(<=
a href=3D"http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop=
/hadoop-hdfs/2.7.1/org/apache/hadoop/hdfs/DFSOutputStream.java#DFSOutputStr=
eam.DataStreamer.run%28%29" target=3D"_blank">http://gr<wbr>epcode.com/file=
/repo1.maven.or<wbr>g/maven2/org.apache.hadoop/had<wbr>oop-hdfs/2.7.1/org/a=
pache/hado<wbr>op/hdfs/DFSOutputStream.java#<wbr>DFSOutputStream.DataStream=
er.<wbr>run%28%29</a>).</div><div style=3D"font-size:14px"><br></div><div s=
tyle=3D"font-size:14px"><br></div><div style=3D"font-size:14px">What&#39;s =
the logic of line 468-485 and line 535-548, why it check the packet is the =
last one twice and the comments are different, one is &quot;<span style=3D"=
color:rgb(103,103,103);font-family:monospace;font-size:medium;white-space:p=
re-wrap">//=C2=A0wait=C2=A0for=C2=A0all=C2=A0data=C2=A0packets=C2=A0<wbr>ha=
ve=C2=A0been=C2=A0successfully=C2=A0acked</span>&quot; and another is &quot=
;<span style=3D"color:rgb(103,103,103);font-family:monospace;font-size:medi=
um;white-space:pre-wrap">//=C2=A0wait=C2=A0for=C2=A0the=C2=A0close=C2=A0pac=
ket=C2=A0<wbr>has=C2=A0been=C2=A0acked</span>&quot;.=C2=A0 What&#39;s the c=
lose packet?</div><div style=3D"font-size:14px"><br></div><div style=3D"fon=
t-size:14px">My understanding of this is:<br></div><div style=3D"font-size:=
14px">If a packet is not the last in block, then the packet is sent directl=
y, and no need to wait for ACK.</div><div style=3D"font-size:14px">If a pac=
ket is the last one, then it will first wait for all previous packet to be =
ACKed, and then it will send the last packet in this block( called close pa=
cket, I guess?), and then will need wait the last packet to be ACKed.</div>=
<div style=3D"font-size:14px"><br></div><div style=3D"font-size:14px">Thank=
s a lot for your help.</div><div style=3D"font-size:14px"><br></div><div st=
yle=3D"font-size:14px"><br></div><div style=3D"font-size:14px">Best,</div><=
div style=3D"font-size:14px"><br></div><div style=3D"font-size:14px">Shu</d=
iv></div>
</blockquote></div><br></div>

--94eb2c05be087c991005453444ca--