hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Vigeant <mark.vige...@riskmetrics.com>
Subject RE: Multiple Input Paths
Date Mon, 02 Nov 2009 15:31:29 GMT
Yes, the structure is similar. They're both XML log files documenting the same set of data,
just in different ways.

That's a really cool idea though, to combine them. How exactly would I go about doing that?

-----Original Message-----
From: L [mailto:architect@galatea.com] 
Sent: Monday, November 02, 2009 10:27 AM
To: common-user@hadoop.apache.org
Subject: Re: Multiple Input Paths


Is the structure of both files the same? It makes even more sense to 
combine the files, if you can, as I have seen a considerable speed up 
when I've done that (at least when I've had small files to deal with).


Mark Vigeant wrote:
> Hey, quick question:
> I'm writing a program that parses data from 2 different files and puts the data into
a table. Currently I have 2 different map functions and so I submit 2 separate jobs to the
job client. Would it be more efficient to add both paths to the same mapper and only submit
one job? Thanks a lot!
> Mark Vigeant
> RiskMetrics Group, Inc.


View raw message