Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of yanbohappy@gmail.com designates
 209.85.215.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABbGW3yhbytA8KaQ2rRAPEYwkuxueBar0c=ERmyov1GcBndGrQ@mail.gmail.com>
References: 
 <CABbGW3yhbytA8KaQ2rRAPEYwkuxueBar0c=ERmyov1GcBndGrQ@mail.gmail.com>
Date: Thu, 28 Mar 2013 23:40:05 +0800
Message-ID: 
 <CALDQvddsg3UXOuWhd79XM+zAA0n4Hk7MWWcYwxCBDnSD8ST+Mw@mail.gmail.com>
Subject: Re: DFSOutputStream.sync() method latency time
From: Yanbo Liang <yanbohappy@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e0cb4efe29d2f8b16a04d8fdf7cc

--e0cb4efe29d2f8b16a04d8fdf7cc
Content-Type: text/plain; charset=ISO-8859-1

1st when client wants to write data to HDFS, it should be create
DFSOutputStream.
Then the client write data to this output stream and this stream will
transfer data to all DataNodes with the constructed pipeline by the means
of Packet whose size is 64KB.
These two operations is concurrent, so the write latency is not simple
superposition.

2nd the sync method only flush the last packet ( at most 64KB ) data to the
pipeline.

Because of the cocurrent processing of all these operations, so the latency
is smaller than the superposition of each operation.
It's parallel computing rather than serial computing in a sense.


2013/3/28 lei liu <liulei412@gmail.com>

> When client  write data, if there are three replicates,  the sync method
> latency time formula should be:
> sync method  latency time = first datanode receive data time + sencond
> datanode receive data  time +  third datanode receive data time.
>
> if the three datanode receive data time all are 2 millisecond, so the sync
> method  latency time should is 6 millisecond,  but according to our our
> monitor, the the sync method  latency time is 2 millisecond.
>
>
> How to calculate sync method  latency time?
>
>
> Thanks,
>
> LiuLei
>
>

--e0cb4efe29d2f8b16a04d8fdf7cc
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

1st when client wants to write data to HDFS, it should be create DFSOutputS=
tream.<div>Then the client write data to this output stream and this stream=
 will transfer data to all DataNodes with the constructed pipeline by the m=
eans of Packet whose size is 64KB.=A0</div>
<div>These two operations is concurrent, so the write latency is not simple=
 superposition.</div><div><br></div><div>2nd the sync method only flush the=
 last packet ( at most 64KB ) data to the pipeline.</div><div><br></div>
<div>Because of the cocurrent processing of all these operations, so the la=
tency is smaller than the superposition of each operation.</div><div>It&#39=
;s parallel computing rather than serial computing in a sense.</div><div>
<br></div><div><br><div class=3D"gmail_quote">2013/3/28 lei liu <span dir=
=3D"ltr">&lt;<a href=3D"mailto:liulei412@gmail.com" target=3D"_blank">liule=
i412@gmail.com</a>&gt;</span><br><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr"><div><div><div><div>When client=A0 write data, if there ar=
e three replicates,=A0 the sync method latency time formula should be:<br>s=
ync method=A0 latency time =3D first datanode receive data time + sencond d=
atanode receive data=A0 time +=A0 third datanode receive data time.<br>

<br></div>if the three datanode receive data time all are 2 millisecond, so=
 the sync method=A0 latency time should is 6 millisecond,=A0 but according =
to our our monitor, the the sync method=A0 latency time is 2 millisecond.<b=
r>

<br><br></div>How to calculate sync method=A0 latency time?<br><br><br></di=
v>Thanks,<br><br></div>LiuLei<br><div><div><div><div><div><br></div></div><=
/div></div></div></div>
</blockquote></div><br></div>

--e0cb4efe29d2f8b16a04d8fdf7cc--