hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: How to debug hadoop(or YARN) locally?
Date Tue, 22 Dec 2015 03:01:55 GMT
If you want to change the yarn internal code, you can use MiniYarnCluster
for testing.

https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java

On Tue, Dec 22, 2015 at 10:00 AM, Allen Zhang <allenzhang010@126.com> wrote:

>
>
> so, does it to mean that, if I change or add some code, I have to
> re-tarball the whole project using "mvn clean package -Pdist -DskipTests
> -Dtar", and then, deploy it to somewhere to remote debug?  if yes, I think
> it is so inconvincence. if no, can you guys explain more in this way?
>
>
> Thanks,
> Allen
>
>
>
>
>
>
>
>
>
> At 2015-12-22 08:55:01, "Jeff Zhang" <zjffdu@gmail.com> wrote:
> >+1 for Chris, remote debug will help you.
> >
> >On Tue, Dec 22, 2015 at 1:54 AM, Chris Nauroth <cnauroth@hortonworks.com>
> >wrote:
> >
> >> If you're running the Hadoop daemons in pseudo-distributed mode (all the
> >> daemons running as separate processes, but on a single dev host), then
> >> another option is to launch the daemon's JVM with the JDWP arguments and
> >> attach a "remote" debugger.  This can be either the jdb CLI debugger
> that
> >> ships with the JDK or a fancier IDE like Eclipse or IntelliJ.
> >>
> >> Each daemon's JVM arguments are controlled with an environment variable
> >> suffixed with "_OPTS" defined in files named *-env.sh.  For example, in
> >> hadoop-env.sh, you could set something like this to enable remote
> >> debugging for the NameNode process:
> >>
> >> export
> >>
> HADOOP_NAMENODE_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,address=8
> >> 000,suspend=n $HADOOP_NAMENODE_OPTS"
> >>
> >>
> >> Then, you can run "jdb -attach localhost:8000" to attach the debugger,
> or
> >> do the equivalent in your IDE of choice.
> >>
> >> --Chris Nauroth
> >>
> >>
> >>
> >>
> >> On 12/21/15, 7:25 AM, "Daniel Templeton" <daniel@cloudera.com> wrote:
> >>
> >> >Your best bet is to find a test that includes all the bits you want and
> >> >execute that test in debug mode.  (You can also change an existing test
> >> >to include what you want, but in most cases it is easier to start with
> >> >an existing test than to start from scratch.)
> >> >
> >> >Daniel
> >> >
> >> >On 12/20/15 6:01 PM, Allen Zhang wrote:
> >> >> Hi all,
> >> >>
> >> >> I am reading hadoop-2.6.0 source code, mainly focusing on hadoop
> yarn.
> >> >> However i have some problems in reading or debugging the source
> >> >>code,can I debug it locally(I mean in my laptop locally with this
> source
> >> >>code I've downloaded, not remotely debug),
> >> >> because I need to track it execution flow stey by stey, and then I
> want
> >> >>to add a new feature or enhancement.
> >> >>
> >> >>
> >> >> So can anyone give some good suggestions or share your method or any
> >> >>wiki page?  Really appreciate!!
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Allen
> >> >
> >> >
> >>
> >>
> >
> >
> >--
> >Best Regards
> >
> >Jeff Zhang
>



-- 
Best Regards

Jeff Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message