spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Donahue <jdona...@adobe.com>
Subject RE: Save an RDD to a SQL Database
Date Thu, 07 Aug 2014 20:41:09 GMT
Depending on what you mean by "save," you might be able to use the Twitter Storehaus package
to do this.  There was a nice talk about this at a Spark meetup -- "Stores, Monoids and Dependency
Injection - Abstractions for Spark Streaming Jobs."  Video here: https://www.youtube.com/watch?v=C7gWtxelYNM&feature=youtu.be.


Jim Donahue
Adobe

-----Original Message-----
From: Ron Gonzalez [mailto:zlgonzalez@yahoo.com.INVALID] 
Sent: Wednesday, August 06, 2014 7:18 AM
To: Vida Ha
Cc: user@spark.incubator.apache.org
Subject: Re: Save an RDD to a SQL Database

Hi Vida,
  It's possible to save an RDD as a hadoop file using hadoop output formats. It might be worthwhile
to investigate using DBOutputFormat and see if this will work for you.
  I haven't personally written to a db, but I'd imagine this would be one way to do it.

Thanks,
Ron

Sent from my iPhone

> On Aug 5, 2014, at 8:29 PM, Vida Ha <vidaha@gmail.com> wrote:
> 
> 
> Hi,
> 
> I would like to save an RDD to a SQL database.  It seems like this would be a common
enough use case.  Are there any built in libraries to do it?
> 
> Otherwise, I'm just planning on mapping my RDD, and having that call a method to write
to the database.   Given that a lot of records are going to be written, the code would need
to be smart and do a batch insert after enough records have collected.  Does that sound like
a reasonable approach?
> 
> 
> -Vida
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail:
user-help@spark.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message