hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ulul <had...@ulul.org>
Subject Re: BLOCK and Split size question
Date Sun, 22 Feb 2015 08:12:40 GMT
Hi

As of Hadoop 2.6, default blocksize is 128 MB (look for dfs.blocksize)
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

About the 100 MB file for a 64 MB blocksize, as said there are 2 blocks 
of 64 and 36 MB each. There is no 28 MB thing, it's just a smaller 
block, HDFS doesn't "reserve" entire blocks as a traditional FS. That 
can be deceptive by the way. If you stuff your cluster with small files, 
you won't see it by your FS filling up but by your NN running out of 
memory or your job performance being bad (since unless you activate 
uberized jobs - which is not yet supported on HDP at least - many JVMs 
will be spawned to deal with too little splits).
In the case of small files you can use CombineFileInputFormat which 
gives an example of the input split not being aligned on block since 
multiple blocks are combined to make an input split.

To complete the non aligned block answer : mapreduce will download the 
missing record part for you from an other DN.

Cheers
Ulul

Le 22/02/2015 03:19, Ahmed Ossama a écrit :
> Hi,
>
> Answering the first question;
>
> What happens is that the client on the machine issuing the command 
> 'copyfromlocal' first creates a new instance of DistributedFileSystem, 
> which makes an RPC call to the NameNode to create a new file in the 
> filesystem's namespace, the NN performs various checks, if passed it 
> creates a record for the file. Then the DistributedFileSystem returns 
> FSDataOutputStream for the client which handles the communication to 
> the datanodes.
>
> The file is then split into packets (based on the block size) and 
> written to the data queue by DataStreamer, which asks the nn to 
> allocate new blocks and picking up a list of dn to store the replicas 
> on. The list of dn forms a pipeline with the number of replicas, the 
> DataStreamer streams the packet to the first datanode in the pipeline 
> which stores the packet and forward it to the second datanode in the 
> pipeline. The packet keeps forwarded until the replicas are written.
>
> When the client finish the stream it sends a confirmation to the NN.
>
> Second question; split size is used during processing of a file, in 
> other words the split size is consumed by the mappers not during HDFS 
> operations. The block is the one consumed during the read and write 
> operations by HDFS client. IIRC the default size of a block in 
> Hadoop-2.6 is 256MB.
>
> Third question; split is the logical representation of the data in a 
> block. Block is the physical representation of the data. As I said, 
> splits are consumed when processing the data, a mapper reads data from 
> a block through a split.
>
> Consider the following content of a file:
>
>     000
>     111
>     222
>     333
>     444
>     555
>     666
>     777
>     888
>     999
>     aaa
>     bbb
>     ccc
>     ddd
>     eee
>     fff
>
>
> The block representation of this file could look something like this
>
>     block-0
>
>         000
>         111
>         222
>         333
>         444
>         55
>
>     block-1
>
>         5
>         666
>         777
>         888
>         999
>         aa
>
>     block-2
>
>         a
>         bbb
>         ccc
>         ddd
>         eee
>         fff
>
> If you notice, the blocks aren't aligned with the record, this happen 
> because the fact that each record in a file isn't like the other. Here 
> comes the splits to put the records all together before being 
> processed by a mapper.
>
> Unlike traditional filesystems, HDFS will create files up to the size 
> of the HDFS block size; Note that there are 2 block size here, the 
> underlying filesystem block size and HDFS block size.
>
> On 02/21/2015 01:59 AM, SP wrote:
>> Hello Every one,
>>
>> I have couple of doubts can any one please point me in right direction.
>>
>> 1>What exactly happen when I want to copy 1TB file to Hadoop Cluster 
>> using copyfromlocal command
>>
>> 1> what will be the split size? will it be same as the block size?
>>
>> 2> What is a block and split?
>>
>> If we have 100 MB file and a block size of 64 MB, As we know it will 
>> be divided into 2 blocks of 64 MB and 36 MB the second block still 
>> has 28 MB of space left what will happen to that free space?
>> will the cluster have unequal block size or will it be occupied by 
>> other file?
>>
>>
>> 3) let’s say a 64MB block is on node A and replicated among 2 other 
>> nodes(B,C), and the input split size for the map-reduce program is 
>> 64MB, will this split just have location for node A? Or will it have 
>> locations for all the three nodes A,b,C?
>>
>>
>> 4) How is it handled if the Input Split size is greater or lesser 
>> than block size?
>>
>>
>> can any one please help?
>>
>> thanks
>>
>> SP
>>
>
> -- 
> Regards,
> Ahmed Ossama


Mime
View raw message