hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Dinic <marko.di...@nissatech.com>
Subject Re: Smaller block size for more intense jobs
Date Wed, 13 May 2015 12:39:00 GMT
Dear Harshit,

Thank you very much for your answer.

Well, to be fully honest with you, I'm currently given just a sample of 
data (poor 35MB sample), so I could develop the processing, and 
hopefully there will be a lot more, I don't know how much.

I'm not really having the problem with large number of small files (not 
at the moment), but I'm expecting to have normal size files, or even 
read that data from a database such as Cassandra (I'm not sure if this 
is going to work and how).

The thing that I know is that there is some heavy processing of data, 
since we're talking about some machine learning (data mining) 
algorithms, such as clustering, where there may be a number of steps 
and iterations. In one of the mappers for example I'm calculating 
similarities between some signals, which may be intensive.

So, to wrap it up, I'm not sure how big the data will be, but I know 
that processing is intensive, and currently a bit slow in my opinion - 
the complete algorithm contains a number of steps and iterations, as I 
said, but it lasts for a couple of hours for the 35MB dataset, which 
worries me.

What do you, or anyone else willing to get into discussion, think?

Best regards,

On Wed 13 May 2015 06:17:58 AM CEST, Harshit Mathur wrote:
> Hi Marko,
> If your files are very small (less than the block size) then a lot of
> map tasks will get executed, but as the initialization and overheads
> degrades the overall performance, so it might appear that the single
> map is executing very fast but the overall job execution will take
> more time.
> I was having a similar problem where the data files were huge in
> number but the size of a single file was much lesser than the block
> size, and due to this a large number of maps were executed by the
> framework. This was taking a great amount of time in overall job
> execution, so to overcome this issue, we used Combined file input
> format, this handles the input split efficiently and an optimum number
> of maps are executed, and thus the overall job execution improves
> drastically.
> Can you give some info about the size of data and the logic for
> processing in the map function, it will help me understand your issue
> more.
> BR,
> Harshit Mathur
> On Wed, May 13, 2015 at 1:27 AM, <marko.dinic@nissatech.com
> <mailto:marko.dinic@nissatech.com>> wrote:
>     Hello,
>     I'm in doubt should I specify the block size to be smaller than
>     64MB in case that my mappers need to do intensive computations?
>     I know that it is better to have larger files, since the
>     replication and NameNode as a weak point, but I'm don't have that
>     much data, but the operations that need to be performed on it are
>     intensive.
>     It looks like it's better to have smaller block size (at least
>     until there is more data) so that multiple Mappers get
>     instantiated, so they could share the computations.
>     I'm currently talking about Hadoop 1, not YARN. But a heads up
>     about the same problem with YARN will be appreciated.
>     Thanks,
>     Marko
>     Sent with inky <http://inky.com?kme=signature>
> --
> Harshit Mathur

View raw message