hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@gmail.com>
Subject Re: Basic Hadoop Doubt
Date Tue, 18 May 2010 02:08:25 GMT

> I have some basic doubt on hadoop Input Data placement...
> Like, If i input some 30GB of data to hadoop program , it will place the
> 30gb into HDFS  into some set of files based on some input formats..

Conceptually, it would be more accurate to say that it splits the data
into 'blocks' that are managed in HDFS. Of course,
implementation-wise, these blocks do get stored on physical files on
the datanodes.

> I have 2 doubts here ..
> 1. Each time i run a program 30GB is placed into HDFS or how its going to
> Work

What program ? Are you talking about this 30GB as input to the program
or output from it ? Assuming Map/Reduce input, the answer is in
general, no. A typical M/R program takes an input path on DFS and this
can point to data that's already copied to DFS, independent of the
program itself.

> 2. Again if i want to run some other program on another 100Gb of data, where
> the above stated data and program is different. Then the previous 30GB is
> erased in HDFS or how its going to run..

Given that the program and its input are independent, the program will
not modify any existing data. In fact most Map/Reduce applications do
not overwrite output data as well. Rather, they will refuse to start
if the output directory already exists.


View raw message