hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YIMEN YIMGA Gael <gael.yimen-yi...@sgcib.com>
Subject RE: Need to evaluate a cluster
Date Thu, 10 Jul 2014 08:35:01 GMT
Hi,

To Mirko
The number of HDDs per datanodes is : 3 (3 disks of 1TB to 3TB)

I calculate the number of nodes using the following formulae

=======

-       Used space on the cluster by daily feed : <daily feed> * <replication factor>
= 720GB * 3 = 2160GB

-       Size of a disk for HDFS : <Size of a disk> * (1 – <booked space for each
disk out HDFS>) = 3TB * (1 – 30%) = 2.1TB

-       Number of datanodes in a year (without monthly data increasing) : <used space on
the cluster by daily feed> * 365 / <size of a disk for HDFS> = 2160GB * 365/1024*2.1
= 367 datanodes
=======

To Olivier

About compression of data, No, I assumed data will not be compressed.
How to use compression ratio in my calculation ?

Standing by

From: Olivier Renault [mailto:orenault@hortonworks.com]
Sent: Wednesday 9 July 2014 18:51
To: user@hadoop.apache.org
Subject: Re: Need to evaluate a cluster


Is your data already compressed? If it's not you can safely assume a compression ratio of
5.

Olivier
On 9 Jul 2014 17:10, "Mirko Kämpf" <mirko.kaempf@gmail.com<mailto:mirko.kaempf@gmail.com>>
wrote:
Hello,

if I follow your numbers I see one missing fact: What is the number of HDDs per DataNode?
Let's assume you use machines with 6 x 3TB HDDs per box, you would need about 60 DataNodes
per year (0.75 TB per day x 3 for replication x 1.3 for overhead / ( nr of HDDs per node x
capacity per HDD )).
With 12 HDD you would only need 30 servers per year.
How did you calculate the number of 367 datanodes?

Cheers,
Mirko

2014-07-09 17:59 GMT+02:00 YIMEN YIMGA Gael <gael.yimen-yimga@sgcib.com<mailto:gael.yimen-yimga@sgcib.com>>:
Hello Dear,

I made an estimation of a number of nodes of a cluster that can be supplied by 720GB of data/day.
My estimation gave me 367 datanodes in a year. I’m a bit afraid by that amount of datanodes.
The assumptions, I used are the followings :


-          Daily supply (feed) : 720GB

-          HDFS replication factor: 3

-          Booked space for each disk outside HDFS: 30%

-          Size of a disk: 3TB.

I have two questions.

First, I would like to know if my assumptions are well taken?
Secondly, could someone help me to evaluate that cluster, to let me be sure that my results
are not to excessive, please ?

Standing by for your feedback

Warm regard

*************************************************************************
This message and any attachments (the "message") are confidential, intended solely for the
addressee(s), and may contain legally privileged information.
Any unauthorised use or dissemination is prohibited. E-mails are susceptible to alteration.
Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the
message if altered, changed or
falsified.
Please visit http://swapdisclosure.sgcib.com for important information with respect to derivative
products.
                              ************
Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles
de contenir des informations couvertes
par le secret professionnel.
Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion
non autorisee est interdite.
Tout message electronique est susceptible d'alteration.
La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message
s'il a ete altere, deforme ou falsifie.
Veuillez consulter le site http://swapdisclosure.sgcib.com afin de recueillir d'importantes
informations sur les produits derives.
*************************************************************************


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.
Mime
View raw message