Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 209.17.190.200 is neither permitted
 nor denied by domain of enis.soz.nutch@gmail.com)
Message-ID: <48195DB5.7030305@gmail.com>
Date: Thu, 01 May 2008 09:05:41 +0300
From: Enis Soztutar <enis.soz.nutch@gmail.com>
User-Agent: Thunderbird 2.0.0.12 (X11/20080227)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: JobConf: How to pass List/Map
References: <e75c02ef0804300419n1f547726i36673b6a9975fae9@mail.gmail.com>
	 <481861D6.7080207@gmail.com>
 <e75c02ef0804300951r6ace9ff3x652a5cf79a9862f5@mail.gmail.com>
In-Reply-To: <e75c02ef0804300951r6ace9ff3x652a5cf79a9862f5@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------060902020805060006080905"

--------------060902020805060006080905
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Yes Stringifier was committed in 0.17. What you can do in 0.16 is to 
simulate DefaultStringifier. The key feature of the Stringifier is that 
it can convert/restore any object to string using base64 encoding on the 
binary form of the object. If your objects can be easily converted to 
and from strings, then you can directly store them in conf. The other 
obvious alternative would be to switch to 0.17, once it is out.

Tarandeep Singh wrote:
> On Wed, Apr 30, 2008 at 5:11 AM, Enis Soztutar <enis.soz.nutch@gmail.com> wrote:
>   
>> Hi,
>>
>>  There are many ways which you can pass objects using configuration.
>> Possibly the easiest way would be to use Stringifier interface.
>>
>>  you can for example :
>>
>>  DefaultStringifier.store(conf, variable ,"mykey");
>>
>>  variable = DefaultStringifier.load(conf, "mykey", variableClass );
>>     
>
> thanks... but I am using Hadoop-0.16 and Stringifier is a fix for 0.17 version -
> https://issues.apache.org/jira/browse/HADOOP-3048
>
> Any thoughts on how to do this in 0.16 version ?
>
> thanks,
> Taran
>
>   
>>  you should take into account that the variable you pass to configuration
>> should be serializable by the framework. That means it must implement
>> Writable of Serializable interfaces. In your particular case, you might want
>> to look at ArrayWritable and MapWritable classes.
>>
>>  That said, you should however not pass large objects via configuration,
>> since it can seriously effect job overhead. If the data you want to pass is
>> large, then you should use other alternatives(such as DistributedCache,
>> HDFS, etc).
>>
>>
>>
>>  Tarandeep Singh wrote:
>>
>>     
>>> Hi,
>>>
>>> How can I set a list or map to JobConf that I can access in
>>> Mapper/Reducer class ?
>>> The get/setObject method from Configuration has been deprecated and
>>> the documentation says -
>>> "A side map of Configuration to Object should be used instead."
>>> I could not follow this :(
>>>
>>> Can someone please explain to me how to do this ?
>>>
>>> Thanks,
>>> Taran
>>>
>>>
>>>
>>>       
>
>   

--------------060902020805060006080905--