hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akash Sharma <sharma...@hotmail.com>
Subject RE: Sqoop Issue
Date Wed, 27 Jun 2012 03:03:40 GMT

Thanks John and Minh for sharing your thoughts and hepling provide the direction to resolve
the issue.

I will add the connector as part of Hadoop Jar command. However there is one thought that
crossed my mind as I was reading these emails.

My initial problem was related to connecting with Oracle DB using the Oracle RAC environment.
I am still able to connect directly to the physical database however due to network issues
between Access Node and DB, a routing deamon is unable to route the requests appropriately.


Are we sure we are really addressing the issue at hand? I mean I could definately discover
if I can really use Oracle RAC connection URL as opposed to direct physical connection by
writing the Java program. How will this new implementation be different from say sqoop tool
which does not work when I source Oracle RAC url (proxy) instead of direct url. 

Any help is appreciated. 

-Akash

> From: mdnguyen@gmail.com
> Date: Tue, 26 Jun 2012 22:41:24 -0400
> Subject: Re: Sqoop Issue
> To: common-user@hadoop.apache.org
> 
> Akash,
> 
>    Instead of adding the connector jar to $HADOOP_HOME/lib, when
> running your map-reduce job using "hadoop jar" you can pass your
> connector jar using the "-libjars" flag.
> 
> For example: hadoop jar hadoop-examples.jar wordcount -files
> cachefile.txt -libjars mylib.jar input output
> 
>    ~ Minh
> 
> On Tue, Jun 26, 2012 at 9:52 PM, John Hancock <jhancock1975@gmail.com> wrote:
> > Akash,
> >
> > I forgot to mention you will need to make sure the connector for your
> > oracle db is available to Hadoop.  There are many ways to do this, but what
> > works for me is I copy the library to <hadoop-install>/lib in all nodes in
> > my cluster.
> >
> > I've written a similar program to do what you are asking about but using a
> > mysql db, so I have mysql-connector-java-5.0.8-bin.jar in the lib directory
> > on all my hadoop nodes.
> >
> > -John
> >
> > On Tue, Jun 26, 2012 at 8:38 AM, Akash Sharma <sharma101@hotmail.com> wrote:
> >
> >>
> >> Thanks for the response John,
> >>
> >> This is exactly what I was looking for. I will pursue this direction and
> >> create a Java program to connect with Oracle DB, extract relevant data and
> >> push it into HDFS.
> >>
> >> I am guessing the Hadoop/HDFS API's will contain all the relevant calls
> >> required to execute these tasks.
> >>
> >> Thanks,
> >>
> >> -Akash
> >>
> >> > Date: Tue, 26 Jun 2012 05:36:27 -0400
> >> > Subject: Re: Sqoop Issue
> >> > From: jhancock1975@gmail.com
> >> > To: common-user@hadoop.apache.org
> >> >
> >> > Akash,
> >> >
> >> > You can write a simple Java program that queries your Oracle DB and uses
> >> > whatever kind of file output object from java.io that you like to write
> >> the
> >> > data to a file.
> >> >
> >> > Compile the program and package it into a jar file.
> >> >
> >> > Then run the program using <path-to-hadoop>/'hadoop jar <jar-file-name>
> >> > <output-file-name> in your Hadoop cluster.
> >> >
> >> > The result will be in HDFS.
> >> >
> >> > -John
> >> >
> >> > On Mon, Jun 25, 2012 at 8:02 AM, Akash Sharma <sharma101@hotmail.com>
> >> wrote:
> >> >
> >> > >
> >> > >
> >> > > Hi,
> >> > >
> >> > > Seeking some advise/options on sqoop of data from Oracle DB to HDFS.
We
> >> > > are on Cloudera 3.
> >> > >
> >> > > Sqoop/JDBC connection to Oracle RAC fails within the Integration
> >> cluster.
> >> > > In the Oracle RAC there  is an additional layer between the Edge
> >> > > Node/Server and Database. Routing the connection occurs to  appropriate
> >> > > database listener. The specific issue is ORA-12516, TNS:listener could
> >> not
> >> > > find available handler with matching protocol stack. This issue is
> >> > > primarily a network configuration issue which is stopping the team
to
> >> > > progress further.
> >> > >
> >> > > I would like to know other than sqoop, what are the other  options
to
> >> > > bring the data into HDFS. Our primary goal is to land data from Oracle
> >> DB
> >> > > into HDFS. Since sqoop does not work, we are thinking of using Oracle
> >> > > unload, ftp, and then put command to import the data into HDFS. Please
> >> > > guide if there  is any other option other than this. Will it help
in
> >> anyway
> >> > > to code these options in java using HDFS/FTP Api within Java itself
> >> rather
> >> > > than using tools?
> >> > >
> >> > > Any help is appreciative.
> >> > >
> >> > > -Akash
> >> > >
> >> > >
> >> > >
> >>
> >>
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message