hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Bieniosek <mich...@powerset.com>
Subject Re: Serializing code to nodes: no can do?
Date Wed, 18 Apr 2007 20:49:24 GMT
I'm not sure exactly what you're trying to do, but you can specify command
line parameters to hadoop -jar which you can interpret in your code.  Your
code can then write arbitrary config parameters before starting the
mapreduce.  Based on these configs, you can load specific jars in your
mapreduce tasks.  But I'm not sure why you'd need to do this, since you
should be able to include any new code in the job jar you submit to hadoop.


On 4/18/07 11:23 AM, "Pedro Guedes" <ps-guedes@criticalsoftware.com> wrote:

> I keep talking to myself... hope it doesn't annoy u too much!
> We thought of a solution to our problem in wich we build a new .job
> file, in accordance with our crawl configuration, and then pass it to
> hadoop for execution... Is there somewhere i can look for the
> specification of the .job format?
> Thanks again,
> Pedro
> I wrote:
>> Hi hadoopers,
>> I'm working on an enterprise search engine that works on an hadoop
>> cluster but is controlled form the outside. I managed to implement a
>> simple crawler much like Nutch's...
>> Now i have a new system's requirement: the crawl process must be
>> configurable outside hadoop. This means that I should be able to add
>> steps to the crawling process that the cluster would execute without
>> knowing before hand what they are... since serialization if not
>> possible, is there another way to achieve the same effect?
>> Using Writable means I need implementations to be on each node so they
>> can read the object data from HDFS... but then i just get the same
>> object and not a new implementation, right?
>> Any thoughts will be appreciated,
>> Pedro
> DISCLAIMER: This message may contain confidential information or privileged
> material and is intended only for the individual(s) named. If you are not a
> named addressee and mistakenly received this message you should not copy or
> otherwise disseminate it: please delete this e-mail from your system and
> notify the sender immediately. E-mail transmissions are not guaranteed to be
> secure or without errors as information could be intercepted, corrupted, lost,
> destroyed, arrive late or incomplete or contain viruses. Therefore, the sender
> does not accept liability for any errors or omissions in the contents of this
> message that arise as a result of e-mail transmissions. Please request a hard
> copy version if verification is required. Critical Software, SA.

View raw message