hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Lynch" <rly...@veoh.com>
Subject RE: hadoop in the ETL process
Date Wed, 02 Jul 2008 23:33:04 GMT
Hadoop is a great way to offload ETL jobs (especially aggregation) out
of the DB. More than likely you would want to use Hadoop as a mechanism
to create a file you can load into the database as a batch job (data
pump or sql loader with Oracle for example) outside of Hadoop entirely.
I would imagine establishing connections inside map/reduce jobs would
not be ideal.

Regards,
Ryan

-----Original Message-----
From: Chris K Wensel [mailto:chris@wensel.net] 
Sent: Wednesday, July 02, 2008 11:31 AM
To: core-user@hadoop.apache.org
Subject: Re: hadoop in the ETL process

If your referring to loading an RDBMS with data on Hadoop, this is  
doable. but you will need to write your own JDBC adapters to your  
tables.

But you might review what you are using the RDBMS for and see if those  
jobs would be better off running on Hadoop entirely, if not for most  
of the processing.

ckw

On Jul 2, 2008, at 10:51 AM, David J. O'Dell wrote:

> Is anyone using hadoop for any part of the ETL process?
>
> Given its ability to process large amounts of log files this seems  
> like
> a good fit.
>
> -- 
> David O'Dell
> Director, Operations
> e: dodell@videoegg.com
> t:  (415) 738-5152
> 180 Townsend St., Third Floor
> San Francisco, CA 94107
>

--
Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
http://www.cascading.org/







Mime
View raw message