hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: Need Help hdfs -How to minimize access Time
Date Wed, 25 Mar 2009 08:25:54 GMT
Hey Snehal (removing the core-dev list; please only post to one at a  

The access time should be fine, but it depends on what you define as  
an acceptable access time.  If this is not acceptable, I'd suggest  
putting it behind a web cache like Squid.  The best way to find out is  
to use the system as a prototype and to evaluate it based on your  

Hadoop is useful for small data, but optimized and originally designed  
only for big data.  The primary downfall of the small files is that it  
may cost more per file in terms of memory.  Hadoop as a solution may  
be overkill, however, if your total storage size is never going to  
grow very large.

We currently use HDFS for mostly random access.


On Mar 25, 2009, at 6:10 AM, snehal nagmote wrote:

> Hello Sir,
> I am doing mtech in iiit  hyderabad , I am doing research project  
> whose aim
> is to develop the scalable storage system For esagu.
> The esagu is all about taking the crop images from the fields and  
> store it
> in the filesystem and then those images would be accessed by  
> agricultural
> scientist to detect the problem, So currently many fields in the  
> A.P. are
> using this system,it may go beyond A.Pso we require storage system
> 1)My problem is we are using hadoop for the storage, but hadoop  
> retrieves
> (reads/writes) in 64 mb chunk . these images stored would be very  
> small size
> say max 2 to 3 mb, So access time would be larger in case of accessing
> images, Can you suggest how this access time can be reduced.Is there
> anyother thing we could do to improve the performance like building  
> our own
> cache, To what extent it would be feasible or helpful in such kind of
> application.
> 2)Second is does hadoop would be useful for small small data like  
> this, if
> not  what tricks we could do to make it usable for such knid of  
> application
> Please help, Thanks in advance
> Regards,
> Snehal Nagmote
> IIIT Hyderabad

View raw message