hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jagaran das <jagaran_...@yahoo.co.in>
Subject Re: Namenode Scalability
Date Wed, 10 Aug 2011 17:07:17 GMT
To be precise, the projected data is around 1 PB.
But the publishing rate is also around 1GBPS.

Please suggest.

From: jagaran das <jagaran_das@yahoo.co.in>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Sent: Wednesday, 10 August 2011 12:58 AM
Subject: Namenode Scalability

In my current project we  are planning to streams of data to Namenode (20 Node Cluster).
Data Volume would be around 1 PB per day.
But there are application which can publish data at 1GBPS.

Few queries:

1. Can a single Namenode handle such high speed writes? Or it becomes unresponsive when GC
cycle kicks in.
2. Can we have multiple federated Name nodes  sharing the same slaves and then we can
distribute the writes accordingly.
3. Can multiple region servers of HBase help us ??

Please suggest how we can design the streaming part to handle such scale of data. 

Jagaran Das 
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message