hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Curtis Ullerich <curtuller...@gmail.com>
Subject Re: Debugging YARN AM
Date Mon, 24 Jun 2013 16:17:34 GMT
For others looking for a solution, here's one way to do it.

You can add the arg directly to the code in hadoop-yarn-applications-
distributedshell/src/main/java/org/apache/hadoop/yarn/
applications/distributedshell/Client.java

Rather than referencing line numbers (as there are frequent commits to
this), find the variable 'vargs' inside Client.run(). The first item added
to that list is the qualified path to /bin/java.
vargs.add(Environment.JAVA_HOME.$() + "/bin/java");

Add this line below that:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1337");

(This will be dependent on how you're building/running)
>From the distributedshell directory, run mvn clean package -DskipTests; mvn
install -DskipTests. You may need to copy the jar from target into
hadoop-dist/... so it's properly available on Hadoop's classpath.

Run the jar like normal
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami

In Eclipse, set up a new debug configuration for a remote Java application.
Select hadoop-yarn-applications-distributedshell as the project. Use
localhost as the Host and 1337 as the port., with connection type of
Standard (Socket Attach). Hit 'Debug' and Eclipse should connect to the
running Application Master. Be sure to set a breakpoint inside main (or
wherever) in the distributedshell project, which must be present in your
Eclipse workspace. Now you can step through the AM.

Let me know of any confusion or questions.

Best,

Curtis



On Tue, Jun 18, 2013 at 3:10 PM, Curtis Ullerich <curtullerich@gmail.com>wrote:

> I'm under the impression that this will launch hadoop (so, the
> ResourceManager?) with those options, and not the ApplicationMaster. Is
> that incorrect?
>
>
> On Tue, Jun 18, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:
>
>> If distributed shell is running as an unmanaged AM then you should set
>> the debug flags for the 'hadoop jar' invocation, doing an 'export
>> HADOOP_OPTS=....' with the debug flags would do.
>>
>> Thx
>>
>>
>> On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <curtullerich@gmail.com
>> > wrote:
>>
>>> Update: It looks like I could add the flag at line 515 of
>>> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
>>> (package org.apache.hadoop.yarn.applications.distributedshell).
>>>
>>> I tried this:
>>>     vargs.add("-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>>>
>>> I recompiled the jar and launched it with
>>> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
>>> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
>>> whoami
>>>
>>> I added a logging statement to make sure the change took effect. It
>>> prints, but it doesn't seem to halt like the MR example (because of the
>>> suspend=y option) even after launching the debugger. Why might this be?
>>>
>>> Thanks!
>>> Curtis
>>>
>>>
>>>
>>> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <curtullerich@gmail.com
>>> > wrote:
>>>
>>>> Hi Devaraj,
>>>>
>>>> That's what I'm not sure how to do. I want to debug by connecting
>>>> remotely, and I'm not sure how to configure that in the code. I haven't
>>>> found anything telling in the docs/source. Can you point me in the right
>>>> direction?
>>>>
>>>> Thanks,
>>>> Curtis
>>>>
>>>>
>>>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <devaraj.k@huawei.com>wrote:
>>>>
>>>>>  Hi Curtis, ****
>>>>>
>>>>> ** **
>>>>>
>>>>>      "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>>>> MRAppMaster. It is not applicable for DistributedShell AM.  ****
>>>>>
>>>>> ** **
>>>>>
>>>>> If you want to dump out debug information then you can make use of the
>>>>> debug option of DistributedShell application. If you want to debug by
>>>>> connecting remotely, you need to update the DS application code accordingly.
>>>>> ****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks****
>>>>>
>>>>> Devaraj K****
>>>>>
>>>>> ** **
>>>>>
>>>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>>>> *Sent:* 18 June 2013 08:19
>>>>> *To:* user@hadoop.apache.org
>>>>> *Subject:* Debugging YARN AM****
>>>>>
>>>>> ** **
>>>>>
>>>>> Hi all, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>>>> mode by launching the pi estimator example with this command:****
>>>>>
>>>>> ** **
>>>>>
>>>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10**
>>>>> **
>>>>>
>>>>> ** **
>>>>>
>>>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>>>> work with the DistributedShell example, presumably because it's not
>>>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>>>> learning purposes, how can I debug the DistributedShell example (and
other
>>>>> AMs)?****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks!****
>>>>>
>>>>> ** **
>>>>>
>>>>> Curtis****
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>

Mime
View raw message