hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: How multiple input files are processed by mappers
Date Mon, 14 Apr 2014 14:12:03 GMT
1. In real production environment do we copy these 10 files in hdfs under a
folder one by one. If this is the case then how many mappers do we specify
10 mappers. And do we use put command of hadoop to transfer this file.

Ans: This will depend on what you want to do with files. There is no rule
which says that all files need to go in one folder one.
While uploading files to hdfs via dfs clients (native hadoop cli or your
java dfs client), they do not need mappers. Its a file system operation.
Remember mappers will be involved only if you call mapreduce framework for
processing the files or writing the files. In normal file uploads, its only
dfs operations.

2. If the above is not the case then do we pre-process to merge these 10
files make it one file of size 10 TB and copy this in hdfs.

Ans: You do not need to merge the files outside and put them on hdfs as
long as individual files are of fair enough sized . When it goes to hdfs,
do you want to merge it that again depends on the purpose you want to use
it for.



On Mon, Apr 14, 2014 at 7:28 PM, Shashidhar Rao
<raoshashidhar123@gmail.com>wrote:

> Hi,
>
> Please can somebody clarify my doubts. Say. I have a cluster of 30 nodes
> and I want to put the files in HDFS. And all the files combine together the
> size is 10 TB but each file is roughly say 1GB  only and the total number
> of files 10 files
>
> 1. In real production environment do we copy these 10 files in hdfs under
> a folder one by one. If this is the case then how many mappers do we
> specify 10 mappers. And do we use put command of hadoop to transfer this
> file.
>
> 2. If the above is not the case then do we pre-process to merge these 10
> files make it one file of size 10 TB and copy this in hdfs .
>
> Regards
> Shashidhar
>
>
>


-- 
Nitin Pawar

Mime
View raw message