accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kepner, Jeremy - 0553 - MITLL" <kep...@ll.mit.edu>
Subject Re: Accumulo Cluster Sizing
Date Fri, 22 May 2015 19:33:52 GMT
77M records / 4 hours ~ 1.5B entries / 4 hours ~ 100K entries/sec

On May 22, 2015, at 3:28 PM, Fagan, Michael <Michael_Fagan@cable.comcast.com> wrote:

> Jeremy, 
> ~72 million records.
> 
> Regards,
> Mike Fagan
> 
> On 5/22/15, 1:12 PM, "Jeremy Kepner" <kepner@ll.mit.edu> wrote:
> 
>> How many records/entires is that?
>> 
>> On Fri, May 22, 2015 at 07:02:05PM +0000, Fagan, Michael wrote:
>>> Jeremy,
>>> 
>>> The data will age off daily so I plan to bulk load ~1TB every 4 hours.
>>> 
>>> Regards,
>>> Mike Fagan
>>> 
>>> 
>>> On 5/22/15, 12:09 PM, "Jeremy Kepner" <kepner@ll.mit.edu> wrote:
>>> 
>>>> 7TB -> 21TB (Hadoop replication), perhaps larger if you have index
>>>> tables, ...
>>>> 
>>>> 1M fetches / day ~ 10M entries / day ~ 1000 entries/sec
>>>> 
>>>> Typical Accumulo peak is 100K entries/sec/core so you should be fine on
>>>> query
>>>> 
>>>> How fast do you need to insert the data into Accumulo?
>>>> 
>>>> On Fri, May 22, 2015 at 03:46:20PM +0000, Fagan, Michael wrote:
>>>>> Josh,
>>>>> 
>>>>> Thanks, I would like use my performance requirements to derive my HW
>>>>> requirements.
>>>>> 
>>>>> For example: assume I have a raw 7TB dataset representing 500 million
>>>>> records with the expectation of 500K-1000K key fetches a day.
>>>>> 
>>>>> I remember there was a tuning webpage circulating around a several
>>> years
>>>>> back to help figure the HW sizing to meet performance benchmarks.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Mike Fagan
>>>>> 
>>>>> 
>>>>> 
>>>>> On 5/22/15, 8:55 AM, "Josh Elser" <josh.elser@gmail.com> wrote:
>>>>> 
>>>>>> Hi Mike,
>>>>>> 
>>>>>> We have some info in
>>>>>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_hardware
>>>>>> 
>>>>>> What's missing there? Let us know the types of questions you have
>>> and
>>>>> we
>>>>>> can expand on the document.
>>>>>> 
>>>>>> - Josh
>>>>>> 
>>>>>> Fagan, Michael wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Can someone point me to recommendations regarding cluster sizing?
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Mike Fagan
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


Mime
View raw message