Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of rmelick@hmc.edu designates
 134.173.53.165 as permitted sender)
Subject: RE: Debugging Hive
From: Russell Melick <rmelick@hmc.edu>
To: Sanjay Sharma <sanjay.sharma@impetus.co.in>
Cc: "dev@hive.apache.org" <dev@hive.apache.org>,
 "hive-dev@hadoop.apache.org" <hive-dev@hadoop.apache.org>,
 "jlym@hmc.edu" <jlym@hmc.edu>, "mwang@hmc.edu" <mwang@hmc.edu>
In-Reply-To: 
 <8419CE074608E24FB834DB9D270C78D60138E2A2B9FE@mail1.impetus.co.in>
References: <1289979231.2062.66.camel@fbook-clinic-r>
	 <8419CE074608E24FB834DB9D270C78D60138E2A2B9FE@mail1.impetus.co.in>
Content-Type: text/plain; charset="UTF-8"
Date: Fri, 19 Nov 2010 15:31:02 -0800
Message-ID: <1290209462.2062.74.camel@fbook-clinic-r>
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit

Thanks Sanjay,

I had a look, but I got some help doing it with eclipse, which worked
great.

I've updated the wiki with that process.

http://wiki.apache.org/hadoop/Hive/DeveloperGuide#Debugging_Hive_code

Russell

On Wed, 2010-11-17 at 13:16 +0530, Sanjay Sharma wrote:
> Russell,
> See if this can help you- http://indoos.wordpress.com/2010/06/24/hive-remote-debugging/
> 
> 
> You would have to tweak the old scripts however for your Hadoop and Hive versions
> 
> Regards,
> Sanjay Sharma
> 
> 
> -----Original Message-----
> From: Russell Melick [mailto:rmelick@hmc.edu]
> Sent: Wednesday, November 17, 2010 1:04 PM
> To: hive-dev@hadoop.apache.org
> Cc: jlym@hmc.edu; mwang@hmc.edu
> Subject: Debugging Hive
> 
> Hi all,
> 
> Apologies for the long email.  I'm attempting to connect to the hive
> code with a debugger to understand the optimizer.  In this example, I
> would like to pause at the beginning of the createIndex function in
> DDLTask.  After following the directions on the wiki, I'm stumped as to
> why connecting remotely with jdb is not working.  Any help is
> appreciated.  I've detailed my process below.
> 
> Thanks,
> Russell Melick
> 
> 
> 
> First, I check out a fresh copy of trunk:
> 
> > svn checkout http://svn.apache.org/repos/asf/hive/trunk hive
> 
> Then, I compile it for debugging, as I found at http://wiki.apache.org/hadoop/Hive/DeveloperGuide#Debugging_Hive_code:
> 
> > cd hive
> > ant -Djavac.debug=on package
> 
> Here are the important variables before I run hive:
> > echo $HIVE_DEBUG_PORT
> > 8000
> 
> > echo $HIVE_DEBUG
> > -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
> 
> > echo $HADOOP_OPTS
> > -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
> 
> I then try to run one of the unit tests:
> 
> > ant test -Dtestcase=TestCliDriver -Dqfile=index_compact_1.q
> 
> The test then waits for a debugger to attach:
> > [junit] Begin query: index_compact_1.q
> >     [junit] Listening for transport dt_socket at address: 8000
> 
> However, when I try to attach with jdb, it doesn't work.
> 
> > jdb -attach 8000
> > Set uncaught java.lang.Throwable
> > Set deferred uncaught java.lang.Throwable
> > Initializing jdb ...
> > >
> > VM Started: No frames on the current call stack
> >
> > main[1] stop at DDLTask:317
> > Deferring breakpoint DDLTask:317.
> > It will be set after the class is loaded.
> > main[1] resume
> > All threads resumed.
> > >
> > The application exited
> 
> The unit test does not seem to recognize that it received the connection.  After jdb prints "application exited", the test says it is listening again, but it didn't confirm the connection.
> 
> > [junit] Begin query: index_compact_1.q
> > [junit] Listening for transport dt_socket at address: 8000
> > [junit] Deleted file:/home/rmelick/hive/build/ql/test/data/warehouse/default__src_src_index__
> > [junit] Listening for transport dt_socket at address: 8000
> > [junit] Listening for transport dt_socket at address: 8000
> 
> I have also attempted to start hiveCli, but I have had no success getting the CLI to start after attaching the debugger.
> 
> This is the only output I get from the cli:
> > build/dist/bin/hive
> > Unable to determine Hadoop version information.
> > 'hadoop version' returned:
> 
> jdb doesn't seem to be connecting correctly
> > jdb -attach 8000
> > Set uncaught java.lang.Throwable
> > Set deferred uncaught java.lang.Throwable
> > Initializing jdb ...
> > >
> > VM Started: No frames on the current call stack
> >
> > main[1] stop at DDLTask:317
> > Deferring breakpoint DDLTask:317.
> > It will be set after the class is loaded.
> > main[1] resume
> > All threads resumed.
> > >
> > The application exited
> 
> I've also tried connecting with Eclipse, after generating the eclipse files, importing the project, and creating the MapRedTask configuration.  It either seems to similarly connect without the client recognizing it, or throws a connection error.
> 
> Again, thanks for any help.
> 
> 
> 
> Impetus is a proud sponsor for ASCI Tour 2010 (Agile Software Community of India) on Oct 30 in Noida, India.
> 
> Meet Impetus at the Cloud Computing Expo from Nov 1-4 in Santa Clara. Our Sr. Director of Engineering, Vineet Tyagi will be speaking about ‘Using Hadoop for Deriving Intelligence from Large Data’.
> 
> Click http://www.impetus.com/ to know more. Follow us on www.twitter.com/impetuscalling
> 
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.