hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 臧冬松 <donal0...@gmail.com>
Subject Re: structured data split
Date Fri, 11 Nov 2011 14:12:44 GMT
Hi Bejoy,
I don't understand why it's impossible to have half of a line in one block,
since the file is split into  fixed size of blocks.
My scenario is that I have lots of files from High Energy Physics
These files are in binary format,about 2G each, but basically they are
composed by lots of "Event", each Event is independent with others. The
physicists use a C++ program called ROOT to analysis these files,and write
the output to a result file(use open(),read(),write()).
I'm considering how to store the files in HDFS, and use the Map-reduce to
analize them.
Tell me if that's not clear.


2011/11/11 <bejoy.hadoop@gmail.com>

> **
> Donal
> In hadoop that hardly happens so. When you are storing data in hdfs it
> would be split line to blocks depending on end of lines, in case of normal
> files. It won't be like you'd be having half of a line in one block and the
> rest in next one. You don't need to worry on that fact.
> The case you mentioned is like dependent data splits. Hadoop's massive
> parallel processing could be fully utilized only in case of independent
> data splits. When data splits are dependent on a file level as I pointed
> out you can go for WholeFileInputFormat.
> Please revert if you are still confused. Also if you have some specific
> scenario, please put that across so we may be able to help you understand
> better on the map reduce processing of the same.
> Hope it clarifies...
> Regards
> Bejoy K S
> ------------------------------
> *From: * 臧冬松 <donal0412@gmail.com>
> *Date: *Fri, 11 Nov 2011 20:46:54 +0800
> *To: *<hdfs-user@hadoop.apache.org>
> *ReplyTo: * hdfs-user@hadoop.apache.org
> *Subject: *Re: structured data split
> Thanks Bejoy!
> It's better to process the data blocks locally and separately.
> I just want to know how to deal with a structure (i.e. a word,a line) that
> is split into two blocks.
> Cheers,
> Donal
> 在 2011年11月11日 下午7:01,Bejoy KS <bejoy.hadoop@gmail.com>写道:
>> Hi Donal
>>       You can configure your map tasks the way you like to process your
>> input. If you have file of size 100 mb, it would be divided into two input
>> blocks and stored in hdfs ( if your dfs.block.size is default 64 Mb). It is
>> your choice on how you  process the same using map reduce
>> - With the default TextInputFormat the two blocks would be processed by
>> two different mappers. (under default split settings) If the blocks are in
>> two different data nodes then two different mappers mappers would be
>> spanned in each data node in beat case. ie They are data local map tasks
>>  - If you want one mapper to process the whole file,change your input
>> format to WholeFileInputFormat. There a mapper task would be triggred on
>> any one of the node where the blocks are located. (best case) If both the
>> blocks are not on the same node then one of the blocks would be transferred
>> to the map task location for processing.
>> Hope it helps!...
>> Thank You
>> Bejoy.K.S
>> 2011/11/11 臧冬松 <donal0412@gmail.com>
>>> Thanks Denny!
>>> So that means each map task will have to read from another DataNode
>>> inorder to read the end line of the previous block?
>>> Cheers,
>>> Donal
>>> 2011/11/11 Denny Ye <dennyy99@gmail.com>
>>>> hi
>>>>    Structured data is always being split into different blocks, likes a
>>>> word or line.
>>>>    MapReduce task read HDFS data with the unit - *line* - it will read
>>>> the whole line from the end of previous block to start of subsequent to
>>>> obtains that part of line record. So you does not worry about the
>>>> Incomplete structured data. HDFS do nothing for this mechanism.
>>>> -Regards
>>>> Denny Ye
>>>> On Fri, Nov 11, 2011 at 3:43 PM, 臧冬松 <donal0412@gmail.com> wrote:
>>>>> Usually large file in HDFS is split into bulks and store in different
>>>>> DataNodes.
>>>>> A map task is assigned to deal with that bulk, I wonder what if the
>>>>> Structured data(i.e a word) was split into two bulks?
>>>>> How MapReduce and HDFS deal with this?
>>>>> Thanks!
>>>>> Donal

View raw message