Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAMiz3FM4yG30pSx+fZz=xPj1zfpGD-5=HuKXq+UL6FM0F4=hHw@mail.gmail.com>
References: 
 <CAPQV63XO81hjxKLLTFHsvcwq1c=LLZH7_eYtRh4N56TF_eppyQ@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEC2161@SZXEML553-MBX.china.huawei.com>
	<CAMiz3FNpZDWoAkE7YfFSeJkOocX-hRaCn71--wKPzcs6GhW+EQ@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEC247C@SZXEML553-MBX.china.huawei.com>
	<CAMiz3FOBgZryRe+SuKvu97TTVguCyLOY2JcmCRXwL5ecwcLVGQ@mail.gmail.com>
	<CAPQV63Xdnv8kNPM1afY1N4kE3OKHntO2AMUqgi==he3DfD6btw@mail.gmail.com>
	<CAMiz3FM4yG30pSx+fZz=xPj1zfpGD-5=HuKXq+UL6FM0F4=hHw@mail.gmail.com>
Date: Fri, 1 Feb 2013 15:09:44 -0500
Message-ID: 
 <CAPQV63Wxw1S9Gazv9-nM+UXRVNx75Or1jxJ6LW7x9i32KRtXDg@mail.gmail.com>
Subject: Re: HBase Checksum
From: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
To: user@hbase.apache.org
Content-Type: text/plain; charset=UTF-8

I have done the major compaction just to be sure. From what I
understand, Checksums are not there if this is not activated... So I
think files need to be re-write to have those checkums added.

I will still try to find a way to see that from the logs.

Worst case, I will add some logs directly into the code and re-deploy...

2013/2/1, Robert Dyer <rdyer@iastate.edu>:
> Yes that log is a debug level log, as I saw in the source.  But I too
> enabled DEBUG and still never saw that log message.
>
> But I, unlike you, see absolutely no change in performance.
>
> One test I did however that makes me think it is actually enabled: if I
> submit from another user I start getting security warnings about that user
> not having permission for shortcircuit.  So perhaps it is working, but I
> have no clue why that log fails to show anywhere.
>
> Regarding enabling checksums that is an interesting question.  Do I have to
> do a major compaction after enabling so HBase writes the checksum?  Or will
> it detect the setting change and do that automatically?  What if I disable,
> will it remove the checksums?
>
>
> On Fri, Feb 1, 2013 at 6:30 AM, Jean-Marc Spaggiari
> <jean-marc@spaggiari.org
>> wrote:
>
>> Hi Robert,
>>
>> That's perfectly fine, it was my next question ;)
>>
>>
>> Anoop, I saw a 5% performance increase by activating HBase Checksum.
>> Can I disable it again to retry the baseline and see the difference?
>> Or now that it's there, it's to late?
>>
>> Also, regarding BlockReaderLocal, I don't find that in my logs, but
>> after I have activated the shortcircuit, I saw a 41% performance
>> increase, so I'm almost sure it's working, but I don't know either how
>>  to check that.
>>
>> What's the best way to see that on the logs? It's not display when
>> HBase is starting. Even not displayed when I'n doing major
>> compactions.
>>
>> I turned org.apache.hadoop.hdfs.BlockReaderLocal loglevel to debug and
>> still can't see anything. Not in the region server, and not in the
>> datanode.
>>
>> Also, to check with HDFS level logs whether the checksum meta file is
>> getting read to the DFS client, I'm not really sure how to acheive
>> that.
>>
>> JM
>>
>> 2013/2/1, Robert Dyer <rdyer@iastate.edu>:
>> > Ok grepping the RS logs I see nothing with 'local' in any of them.
>>  Thanks
>> > for that hint.
>> >
>> > For the test I was using, I know it is data local.  Every map task
>> launched
>> > data local, and no regions were moving recently.
>> >
>> > I think I've hijacked this thread enough, I'll move my issues to
>> > another.
>> > ;-)
>> >
>> >
>> > On Thu, Jan 31, 2013 at 11:51 PM, Anoop Sam John <anoopsj@huawei.com>
>> > wrote:
>> >
>> >> Hi Robert
>> >>           When HDFS is doing the local short circuit read, it will use
>> >> BlockReaderLocal class for reading.  There should be some logs at the
>> DFS
>> >> client side (RS) which tells abt creating new BlockReaderLocal .  If
>> >> you
>> >> can see this then sure the local read is happening.
>> >>
>> >> Also check DN log.  If local read happening, then you will not see
>> >> read
>> >> request related logs for the HFile at the DN side.
>> >> You check your no# of HFiles and names for checking the logs
>> >>
>> >> Are you sure that when you tested, u have data locality? Region
>> movements
>> >> across RSs can break the full data locality.
>> >>
>> >> -Anoop-
>> >> ________________________________________
>> >> From: Robert Dyer [psybers@gmail.com]
>> >> Sent: Friday, February 01, 2013 11:10 AM
>> >> To: Hbase-User
>> >> Subject: Re: HBase Checksum
>> >>
>> >> Not trying to hijack your thread here...
>> >>
>> >> But can you verify via logs that the shortcircuit is working?  Because
>> >> I
>> >> enabled shortcircuit but I sure didn't see any performance increase.
>> >>
>> >> I haven't tried enabling hbase checksum yet but I'd like to be able to
>> >> verify that works too.
>> >>
>> >>
>> >> On Thu, Jan 31, 2013 at 9:55 PM, Anoop Sam John <anoopsj@huawei.com>
>> >> wrote:
>> >>
>> >> > You can check with HDFS level logs whether the checksum meta file is
>> >> > getting read to the DFS client? In the HBase handled checksum, this
>> >> should
>> >> > not happen.
>> >> > Have you noticed any perf gain when you configure the HBase handled
>> >> > checksum option?
>> >> >
>> >> > -Anoop-
>> >> > ________________________________________
>> >> > From: Jean-Marc Spaggiari [jean-marc@spaggiari.org]
>> >> > Sent: Friday, February 01, 2013 4:16 AM
>> >> > To: user
>> >> > Subject: HBase Checksum
>> >> >
>> >> > Hi,
>> >> >
>> >> > I have activated shortcircuit and checksum and I would like to get a
>> >> > confirmation that it's working fine.
>> >> >
>> >> > So I have activated short circuit first and saw a 40% improvement of
>> >> > the MR rowcount job. So I guess it's working fine.
>> >> >
>> >> > Now, I'm configuring the checksum option, and I'm wondering how I
>> >> > can
>> >> > do to validate that it's taken into consideration and used, or not.
>> >> > Is
>> >> > there a way to see that?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > JM
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Robert Dyer
>> > rdyer@iastate.edu
>> >
>>
>
>
>
> --
>
> Robert Dyer
> rdyer@iastate.edu
>