hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sudhakara st <sudhakara...@gmail.com>
Subject Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Date Fri, 17 Jan 2014 08:42:09 GMT
Have Look on
BlockReaderLocal.java : Enables local short circuited reads.
The client gets the path to the file where block is stored using
getBlockLocalPathInfo()
Used to read file BlockReaderLocal()

http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/


On Fri, Jan 17, 2014 at 1:24 PM, Atish Kathpal <atish.kathpal@gmail.com>wrote:

> Thanks Sudhakara. That explains the behavior I am observing.
>
> What I need: I would like files submitted as part of a MR job, to be asked
> from the underlying storage in a batched form immediately after submitting
> a job. This way the files that would be accessed as part of the MR job,
> will get cached in the underlying storage, leading to improved MR
> performance. I need to get this behavior out of MR as some of the files in
> the underlying storage (NFS mount point) I am using, are not available
> online and need to be fetched with some latency (~5-10secs or more). When
> all such offline files are asked from the underlying store by MR in one go,
> the underlying storage can make them available in parallel hence speeding
> up the entire process.
>
> Any clues on which classes I could start looking at in order to get this
> behavior from MR would be helpful.
>
> *TL;DR*
> How to make MR perform batch IO for files to be accessed as part of
> submitted job, before actual mappers have started processing those files?
>
> Thanks
> Atish
>
>
>
>
>
> On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <sudhakara.st@gmail.com>wrote:
>
>> Hello Atish,
>>          Number of splits parallel processed  at any instance is
>> dependance upon the your cluster capacity i,e. How many mapper and reduce
>> slots are configure in the cluster.
>> The default each tasktracker   run maximum 2 map tasks that will be run
>> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
>> single task tacker it can simultaneously two mapper and process to input
>> splits. In your case your processing >1GB file, it as more than 8 input
>> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
>> mapper slots, maximum will process 4 input split any instance. Hence it
>> looking like  sequential behavior.
>> For more info please have look on:
>>
>> mapred.tasktracker.map.tasks.maximum( for map slots)
>> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <atish.kathpal@gmail.com>wrote:
>>
>>> Basically what I really need is that if I am submitting a job (say
>>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>>> issue immediate IO for reads on each of these files, such that the
>>> underlying storage system can know that it has to cache those files and
>>> present them back to Hadoop for processing. What I observe through by logs
>>> currently is that, if a particular file is very large, then Hadoop first
>>> operates on all splits of this large file, before asking for the next file.
>>>
>>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>>> where I should make any changes to. Any other suggestions for this will be
>>> welcome too.
>>>
>>>
>>>
>>>
>>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <atish.kathpal@gmail.com>wrote:
>>>
>>>> I took logs at the underlying storage system and noticed the sequential
>>>> behavior across files. However, most of the files I am testing with are big
>>>> > 1GB in size, hence leading to several splits. From the logs I notice
that
>>>> File2 is not being accessed until all blocks of File1 have been read. It
is
>>>> possible that this is just the case as I have just 2 nodes and the files
>>>> being processed are huge, Hence I am observing a sequential behaviour
>>>> across files.
>>>>
>>>> Within the files however, the splits seems to be getting processed in
>>>> parallel as I see random offsets within the file being accessed. That said,
>>>> let me try logging at the NFS level to map IOs coming from different IP
>>>> addresses (nodes) and analyze the pattern.
>>>>
>>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>>> take a look and get back accordingly.
>>>>
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <java8964@hotmail.com> wrote:
>>>>
>>>>> When you said that the mappers seem to be accessing file sequentially,
>>>>> why do you think so?
>>>>>
>>>>> NFS maybe changes something, but mappers shouldn't access file
>>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>>> test to verify it.
>>>>>
>>>>> The class you want to check out is the
>>>>> org.apache.hadoop.mapred.FileInputFormat, especially method
>>>>> getSplits().
>>>>>
>>>>> The above code is the key how the split list is generated. If it
>>>>> doesn't performance well for your underline storage system, you can always
>>>>> write your own InputFormat to utilize your own storage system.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> From: atish.kathpal@gmail.com
>>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>>
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>>> Need guidance for 2. though!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>>
>>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>>> > wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>>> Thanks a lot for the advice.
>>>>>
>>>>> *Two issues though*:
>>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>>> output directory, but no part-r file containing the actual results of
>>>>> the wordcount job. However I am seeing the correct output on running
MR
>>>>> over HDFS. What is going wrong? Any place I can find logs for the MR
job. I
>>>>> see no errors on the console.
>>>>> Command used:
>>>>> hadoop jar
>>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>>
>>>>>
>>>>> 2. I am observing that the mappers seem to be accessing files
>>>>> sequentially, splitting the files across mappers, and then reading data
in
>>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>>> files themselves should be accessed in parallel, that is, if there are
10
>>>>> files to be MRed, then MR should ask for each of these files in parallel
in
>>>>> one go, and then work on the splits of these files in parallel.
>>>>> *Why do I need this?* Some of the data coming from the NFS mount
>>>>> point is coming from offline media (which takes ~5-10 seconds of time
>>>>> before first bytes are received). So I would like all required files
to be
>>>>> asked at the onset itself from the NFS mount point. This way several
>>>>> offline media will be spun up parallely and as the data from these media
>>>>> gets available MR can process them.
>>>>>
>>>>> Would be glad to get inputs on these points!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>> Tip for those who are trying similar stuff::
>>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>>> but I was able to rectify this with help from:
>>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>>
>>>>> As on the bottlenecking issue, I agree, but  I am trying to run few MR
>>>>> jobs on a traditional NAS server. I can live with a few bottlenecks,
so
>>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>>
>>>>>
>>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <chris.mawata@gmail.com>wrote:
>>>>>
>>>>>  Yong raises an important issue:  You have thrown out the I/O
>>>>> advantages of HDFS and also thrown out the advantages of data locality.
It
>>>>> would be interesting to know why you are taking this approach.
>>>>> Chris
>>>>>
>>>>>
>>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>>
>>>>> I believe the "-fs local" should be removed too. The reason is that
>>>>> even you have a dedicated JobTracker after removing "-jt local", but
with
>>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>>
>>>>>  "-fs local" will force the mapreducer run in "local" mode, which is
>>>>> really a test mode.
>>>>>
>>>>>  What you can do is to remove both "-fs local -jt local", but give
>>>>> the FULL URI of the input and output path, to tell Hadoop that they are
>>>>> local filesystem instead of HDFS.
>>>>>
>>>>>  "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///hduser/mount_point file:///results"
>>>>>
>>>>>  Keep in mind followings:
>>>>>
>>>>>  1) The NFS mount need to be available in all your Task Nodes, and
>>>>> mounted in the same way.
>>>>> 2) Even you can do that, but your sharing storage will be your
>>>>> bottleneck. NFS won't work well for scalability.
>>>>>
>>>>>  Yong
>>>>>
>>>>>  ------------------------------
>>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> From: dsuiter@rdx.com
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>> I think most of your problem is coming from the options you are
>>>>> setting:
>>>>>
>>>>>  "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount *-fs local -jt local* /hduser/mount_point/  /results"
>>>>>
>>>>>  You appear to be directing your namenode to run jobs in the *LOCAL*
job
>>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>>> is set up right. You don't need to do anything special to point Hadoop
>>>>> towards a NFS location, other than set up the NFS location properly and
>>>>> make sure if you are directing to it by name that it will resolve to
the
>>>>> right address. Hadoop doesn't care where it is, as long as it can read
from
>>>>> and write to it. The fact that you are telling it to read/write from/to
a
>>>>> NFS location that happens to be mounted as a local filesystem object
>>>>> doesn't matter - you could direct it to the local /hduser/ path and set
the
>>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>>> network location of the folder that you want, it shouldn't make a
>>>>> difference.
>>>>>
>>>>>  *Devin Suiter*
>>>>> Jr. Data Solutions Software Engineer
>>>>>   100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>>
>>>>>
>>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Hello
>>>>>
>>>>>  The picture below describes the deployment architecture I am trying
>>>>> to achieve.
>>>>> However, when I run the wordcount example code with the below
>>>>> configuration, by issuing the command from the master node, I notice
only
>>>>> the master node spawning map tasks and completing the submitted job.
Below
>>>>> is the command I used:
>>>>>
>>>>>  *hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount -fs local -jt local /hduser/mount_point/  /results*
>>>>>
>>>>>  *Question: How can I leverage both the hadoop nodes for running MR,
>>>>> while serving my data from the common NFS mount point running my filesystem
>>>>> at the backend? Has any one tried such a setup before?*
>>>>> [image: Inline image 1]
>>>>>
>>>>>  Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> ...Sudhakara.st
>>
>>
>
>


-- 

Regards,
...Sudhakara.st

Mime
View raw message