hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Writing Reducer output to database
Date Fri, 04 Feb 2011 03:29:40 GMT
At least in cdh3b2, there are two DBOutputFormat.java:


You should be able to use the latter.

On Thu, Feb 3, 2011 at 2:45 PM, Adeel Qureshi <adeelmahmood@gmail.com>wrote:

> I had started a thread recently to ask questions about custom writable
> implementations which is basically similar to this .. but that was more of
> an understanding of the concept and here I wanted to ask my actual problem
> and get help on that.
> I want to be able to read text data line by line in my mapper ..
> create an instance of a custom writable class that holds some information
> parsed out of the line ..
> pass that custom writable along with its count to reducer
> reducer then simply need to insert every single entry into a database ..
> I am just trying to understand how to accomplish this. here is what I am
> thinking i need to do based on my little understanding of all this custom
> stuff
> 1. create a custom writable class that can hold my parsed records. in my
> mapper create a new instance of it using the text line and output the
> created instance
> 2. accept this custom writable in mapper
> 3. set reducer output to DBOutputFormat
>    I tried doing that and it seems like I am supposed to use JobConf class
> which is deprecated and the new configuration class where you are supposed
> to use the job object to set the input/output formats doesnt seems to work
> with DBOuputFormat .. doesnt this DBOutputFormat stuff works with hadoop
> new
> api
> 4. now in reducer I am confused wat to do .. i guess i need to convert my
> custom writable object to another custom dbwritable object .. that will
> then
> be written to the database .. any hints on how to accomplish this ..
> Sorry if the questions arent very clear .. I am just really confused about
> this stuff and it doesnt helps that there is literally NO useful
> information
> available anywhere on this writable and dbwritable stuff
> Thanks
> Adeel

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message