hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcos Ortiz <mlor...@uci.cu>
Subject Re: DBOutputFormat with one reducer
Date Wed, 25 May 2011 23:22:17 GMT
On 05/25/2011 04:27 PM, Giridhar Addepalli wrote:
> Hi,
> We have MapReduce program which writes data to mysql database using 
> DBOutputFormat.
> Our program has one reducer.
> I understand that all the inserts happen during the close() operation 
> of the reducer.
> Is it gauranteed that this operation is atomic ? i.e; what happens if 
> the writes fail in the middle of the operation.
> Does it mean that only partial number of rows get in to mysql database?
> What does it take to make the write operation atomic ?
> Any suggestions around our situation( alternative solutions ) are welcome.
> Thanks,
> Giridhar.
Sqoop is done for that kind of tasks.

  Sqoop is an open-source tool that allows users to extract data from a 
relational database into Hadoop for further processing.
  This processing can be done with MapReduce programs or other 
higher-level tools such  as Hive. When the final results of an analytic 
pipeline are available, Sqoop can export
  these results back to the database for consumption by other clients.

Source code


Marcos Luis Ortiz Valmaseda
  Software Engineer (Distributed Systems)

View raw message