hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drew Dahlke <drew.dah...@bronto.com>
Subject Re: client-side caching
Date Tue, 05 Jul 2011 12:15:57 GMT
I've been down that road and did a memcache lookup within a mapper. My
9 node cluster managed to CPU and network spike the memcache server
hard enough I practically DOS attacked it.

I'd recommend taking your data (if it's not massive) and dumping it to
a hadoop map file. Think of it as a file based key value store with
indexed look-ups. As part of your job config you can use hadoop's
distributed cache mechanism to push the files out to all the nodes.
Then they'll be available from within your job locally. As an
optimization you might want to cache reads from the map file using a
weak hash map or something similar.

Hope that helps

On Tue, Jul 5, 2011 at 5:51 AM, Claudio Martella
<claudio.martella@tis.bz.it> wrote:
> Totally understand. As a matter of fact I didn't mention my table is
> read-only or insert-only (no data is modified), so no real invalidation
> necessary here. I guess this means i should go for my own memcache on
> the client side.
>
> Thanks!
>
>
> On 7/5/11 11:39 AM, Ryan Rawson wrote:
>> Caching sounds easy until you need to worry about invalidation. It's hard to
>> build efficient and correct invalidation.
>> On Jul 5, 2011 2:13 AM, "Claudio Martella" <claudio.martella@tis.bz.it>
>> wrote:
>>> I've seen that. But that's about caching on regionserver-side through
>>> memcache.
>>>
>>> You still have the network roundtrip. I'd like to avoid at all
>>> contacting the regionserver, when possible. So I was guessing whether
>>> the hbase-client would have some caching embedded, otherwise I'll
>>> implement it through memcache.
>>>
>>>
>>> On 7/4/11 7:03 PM, Ted Yu wrote:
>>>> See HBASE-4018
>>>>
>>>> On Mon, Jul 4, 2011 at 7:33 AM, Claudio Martella <
>> claudio.martella@tis.bz.it
>>>>> wrote:
>>>>> Hello list,
>>>>>
>>>>> i'm using hbase 0.90.3 on a 5 nodes cluster. I'm using a table as a
>>>>> string<->long map. As I'm using this map a lot, I was thinking
about
>>>>> installing memcache on the client side, as to avoid flooding hbase for
>>>>> the same value over and over.
>>>>>
>>>>> What is the best practice in these situations? some client-side caching
>>>>> already in hbase?
>>>>>
>>>>> Best,
>>>>>
>>>>> Claudio
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> Digital Technologies
>>>>> Unit Research & Development - Analyst
>>>>>
>>>>> TIS innovation park
>>>>> Via Siemens 19 | Siemensstr. 19
>>>>> 39100 Bolzano | 39100 Bozen
>>>>> Tel. +39 0471 068 123
>>>>> Fax +39 0471 068 129
>>>>> claudio.martella@tis.bz.it http://www.tis.bz.it
>>>>>
>>>>> Short information regarding use of personal data. According to Section
>> 13
>>>>> of Italian Legislative Decree no. 196 of 30 June 2003, we inform you
>> that we
>>>>> process your personal data in order to fulfil contractual and fiscal
>>>>> obligations and also to send you information regarding our services and
>>>>> events. Your personal data are processed with and without electronic
>> means
>>>>> and by respecting data subjects' rights, fundamental freedoms and
>> dignity,
>>>>> particularly with regard to confidentiality, personal identity and the
>> right
>>>>> to personal data protection. At any time and without formalities you
can
>>>>> write an e-mail to privacy@tis.bz.it in order to object the processing
>> of
>>>>> your personal data for the purpose of sending advertising materials and
>> also
>>>>> to exercise the right to access personal data and other rights referred
>> to
>>>>> in Section 7 of Decree 196/2003. The data controller is TIS Techno
>>>>> Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the
>>>>> complete information on the web site www.tis.bz.it.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Claudio Martella
>>> Free Software & Open Technologies
>>> Analyst
>>>
>>> TIS innovation park
>>> Via Siemens 19 | Siemensstr. 19
>>> 39100 Bolzano | 39100 Bozen
>>> Tel. +39 0471 068 123
>>> Fax +39 0471 068 129
>>> claudio.martella@tis.bz.it http://www.tis.bz.it
>>>
>>> Short information regarding use of personal data. According to Section 13
>> of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we
>> process your personal data in order to fulfil contractual and fiscal
>> obligations and also to send you information regarding our services and
>> events. Your personal data are processed with and without electronic means
>> and by respecting data subjects' rights, fundamental freedoms and dignity,
>> particularly with regard to confidentiality, personal identity and the right
>> to personal data protection. At any time and without formalities you can
>> write an e-mail to privacy@tis.bz.it in order to object the processing of
>> your personal data for the purpose of sending advertising materials and also
>> to exercise the right to access personal data and other rights referred to
>> in Section 7 of Decree 196/2003. The data controller is TIS Techno
>> Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the
>> complete information on the web site www.tis.bz.it.
>>>
>>>
>>>
>
>
> --
> Claudio Martella
> Free Software & Open Technologies
> Analyst
>
> TIS innovation park
> Via Siemens 19 | Siemensstr. 19
> 39100 Bolzano | 39100 Bozen
> Tel. +39 0471 068 123
> Fax  +39 0471 068 129
> claudio.martella@tis.bz.it http://www.tis.bz.it
>
> Short information regarding use of personal data. According to Section 13 of Italian
Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data
in order to fulfil contractual and fiscal obligations and also to send you information regarding
our services and events. Your personal data are processed with and without electronic means
and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with
regard to confidentiality, personal identity and the right to personal data protection. At
any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to
object the processing of your personal data for the purpose of sending advertising materials
and also to exercise the right to access personal data and other rights referred to in Section
7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street
n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.
>
>
>
>
>

Mime
View raw message