hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Job, JobConf, and Configuration.
Date Sun, 08 Apr 2012 19:04:41 GMT
The Job class encapsulates the Configuration object and manages it for
you. You can also get its reference out via Job.getConfiguration() ->

Hence, when you do a: "Job job = new Job();", the internal
Configuration object is auto-created for you. This is how the
underlying constructor looks:

public Job() throws IOException {
    this(new Configuration());

On Mon, Apr 9, 2012 at 12:24 AM, JAX <jayunit100@gmail.com> wrote:
> Hi guys.  Just a theoretical question here : I notice in chapter 1 of the Hadoop orielly
book that the "new API" example has *no* Configuration object.
> Why is that?
> I thought the new API still uses / needs a Configuration class when running jobs.
> Jay Vyas
> On Apr 7, 2012, at 4:29 PM, Harsh J <harsh@cloudera.com> wrote:
>> MapReduce sets "mapred.child.tmp" for all tasks to be the Task
>> Attempt's WorkingDir/tmp automatically. This also sets the
>> -Djava.io.tmpdir prop for each task at JVM boot.
>> Hence you may use the regular Java API to create a temporary file:
>> http://docs.oracle.com/javase/6/docs/api/java/io/File.html#createTempFile(java.lang.String,%20java.lang.String)
>> These files would also be automatically deleted away after the task
>> attempt is done.
>> On Sun, Apr 8, 2012 at 2:14 AM, Ondřej Klimpera <klimpond@fit.cvut.cz> wrote:
>>> Hello,
>>> I would like to ask you if it is possible to create and work with a
>>> temporary file while in a map function.
>>> I suppose that map function is running on a single node in Hadoop cluster.
>>> So what is a safe way to create a temporary file and read from it in one
>>> map() run. If it is possible is there a size limit for the file.
>>> The file can not be created before hadoop job is created. I need to create
>>> and process the file inside map().
>>> Thanks for your answer.
>>> Ondrej Klimpera.
>> --
>> Harsh J

Harsh J

View raw message