Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <58DA995C.4000501@apache.org>
References: <CABK4GYO3_Seoc91UPPeoejZuBtqaz3kV09NFaz+v+jByP=i8hg@mail.gmail.com>
 <CABK4GYNiqaNdPb=Se1sH7Yc_=BuSOhBr036XCKvWgZBM-dbykg@mail.gmail.com>
 <CALte62zzgz_8eyArs2LmdNJRYBaBtQuEcX_-9W8+3c2fzNYMrw@mail.gmail.com>
 <CABK4GYPfx5D=Ae7+hm0baMg6RC0xZj4NkOurJ9ZHYcmTHV4_Ew@mail.gmail.com> <58DA995C.4000501@apache.org>
From: =?UTF-8?B?5p2o6IuP56uLIFlhbmcgU3UgTGk=?= <yangsuli@gmail.com>
Date: Thu, 30 Mar 2017 15:26:38 -0500
Message-ID: <CABK4GYNf1j-dPmWX2D7USKpSeLORGgBAZGhgPCQaFNc7+cJ9Mw@mail.gmail.com>
Subject: Re: How threads interact with each other in HBase
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11425bf8d35a90054bf8831e
archived-at: Thu, 30 Mar 2017 20:27:06 -0000

--001a11425bf8d35a90054bf8831e
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Also, please correct me if I am wrong, but I don't think a put is durable
when an RPC returns to the client. Just its corresponding WAL entry is
pushed to the memory of all three data nodes, so it has a low probability
of being lost. But nothing is persisted at this point.

And this is true no mater you use SYNC_WAL or FSYNC_WAL flag.

On Tue, Mar 28, 2017 at 12:11 PM, Josh Elser <elserj@apache.org> wrote:

> 1.1 -> 2: don't forget about the block cache which can invalidate the nee=
d
> for any HDFS read.
>
> I think you're over-simplifying the write-path quite a bit. I'm not sure
> what you mean by an 'asynchronous write', but that doesn't exist at the
> HBase RPC layer as that would invalidate the consistency guarantees (if a=
n
> RPC returns to the client that data was "put", then it is durable).
>
> Going off of memory (sorry in advance if I misstate something): the
> general way that data is written to the WAL is a "group commit". You have
> many threads all trying to append data to the WAL -- performance would be
> terrible if you serially applied all of these writes. Instead, many write=
s
> can be accepted and a the caller receives a Future. The caller must wait
> for the Future to complete. What's happening behind the scene is that the
> writes are being bundled together to reduce the number of syncs to the WA=
L
> ("grouping" the writes together). When one caller's future would complete=
,
> what really happened is that the write/sync which included the caller's
> update was committed (along with others). All of this is happening inside
> the RS's implementation of accepting an update.
>
> https://github.com/apache/hbase/blob/55d6dcaf877cc5223e67973
> 6eb613173229c18be/hbase-server/src/main/java/org/apache/hadoop/hbase/
> regionserver/wal/FSHLog.java#L74-L106
>
>
> =E6=9D=A8=E8=8B=8F=E7=AB=8B Yang Su Li wrote:
>
>> The attachment can be found in the following URL:
>> http://pages.cs.wisc.edu/~suli/hbase.pdf
>>
>> Sorry for the inconvenience...
>>
>>
>> On Mon, Mar 27, 2017 at 8:25 PM, Ted Yu<yuzhihong@gmail.com>  wrote:
>>
>> Again, attachment didn't come thru.
>>>
>>> Is it possible to formulate as google doc ?
>>>
>>> Thanks
>>>
>>> On Mon, Mar 27, 2017 at 6:19 PM, =E6=9D=A8=E8=8B=8F=E7=AB=8B Yang Su Li=
<yangsuli@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>>
>>>> I am a graduate student working on scheduling on storage systems, and =
we
>>>> are interested in how different threads in HBase interact with each
>>>> other
>>>> and how it might affect scheduling.
>>>>
>>>> I have written down my understanding on how HBase/HDFS works based on
>>>> its
>>>> current thread architecture (attached). I am wondering if the develope=
rs
>>>>
>>> of
>>>
>>>> HBase could take a look at it and let me know if anything is incorrect
>>>> or
>>>> inaccurate, or if I have missed anything.
>>>>
>>>> Thanks a lot for your help!
>>>>
>>>> On Wed, Mar 22, 2017 at 3:39 PM, =E6=9D=A8=E8=8B=8F=E7=AB=8B Yang Su L=
i<yangsuli@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> I am a graduate student working on scheduling on storage systems, and
>>>>> we
>>>>> are interested in how different threads in HBase interact with each
>>>>>
>>>> other
>>>
>>>> and how it might affect scheduling.
>>>>>
>>>>> I have written down my understanding on how HBase/HDFS works based on
>>>>>
>>>> its
>>>
>>>> current thread architecture (attached). I am wondering if the
>>>>>
>>>> developers of
>>>
>>>> HBase could take a look at it and let me know if anything is incorrect
>>>>>
>>>> or
>>>
>>>> inaccurate, or if I have missed anything.
>>>>>
>>>>> Thanks a lot for your help!
>>>>>
>>>>> --
>>>>> Suli Yang
>>>>>
>>>>> Department of Physics
>>>>> University of Wisconsin Madison
>>>>>
>>>>> 4257 Chamberlin Hall
>>>>> Madison WI 53703
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Suli Yang
>>>>
>>>> Department of Physics
>>>> University of Wisconsin Madison
>>>>
>>>> 4257 Chamberlin Hall
>>>> Madison WI 53703
>>>>
>>>>
>>>>
>>
>>
>>


--=20
Suli Yang

Department of Physics
University of Wisconsin Madison

4257 Chamberlin Hall
Madison WI 53703

--001a11425bf8d35a90054bf8831e--