hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ellis Miller <outlaw...@gmail.com>
Subject Re:
Date Mon, 11 Mar 2013 11:31:29 GMT
Sorry...was typing another email:)

Anyway, default config files across Master and Slave Nodes:


   - /src/core-site.xml, src/hdfs/hdfs-default.xml and the one you would
   want to be standard, of course, the src/mapred/mapred-default.xml


For each DataNode / Task Tracker across however many Virtual Machines could
/ would customize the mapred*.xml file in particular:


   - conf/core-site.xml, conf/hdfs-site.xml, and conf/mapred-site.xml


Again, using *log4j.properties* on the Slave sites under the conf/
directory very flexible in terms of metrics, etc. you would like to
custom-log say for each TaskTracker node in the same site-specific Slave
directory as the* /conf/hadoop-env.sh* which you can also modify on each
Slave Node for various options (both log4j.properties and the hadoop-env.sh
files provide fairly extensive means to log most of what you would be
interested in along with the memory settings, in particular, in the
conf/mapred-site.xml files.

However, simulated cluster on single node configuration is actually more
confusing in that regard in your case if it's a single node then extending
the *Task.java* source file or more specifically the package including that
file then compiling as .jar (Java archive) and uploading may arguably be
just as simple yet the other site-specific files for the DataNode and
TaskTracker along with the hadoop-env.sh are very, very flexible for
logging or otherwise setting specific options for Java Heap size, etc. (in
conjunction with the log4j.properties).

In short, would look at the /conf/mapred-site.xml, /conf/hadoop-env.sh, and
log4j.properties files first as think it could be a very quick
already-available fix in terms of the additional logging you want to
perform.

Thanks.

Ellis

On Mon, Mar 11, 2013 at 7:13 AM, Ellis Miller <outlawdba@gmail.com> wrote:

> Could use an UDF to extend the Task.java yet if it's a true cluster or
> appropriately simulated then basic standard:
>
> 1. Master is comprised of NameNode and JobTracker
> 2. All other are Slave (hosts) or Virtual Machines consisting of DataNodes
> and TaskTrackers in particular
>
> So, on Slaves Hosts could easily modify mapred.trasktracker.* parameters
> in the customized Slave VMs mapred-site.xml files for logging Memory
> Monitoring as (assuming) you are concerned with the TT memory usage which
> can cause a task to fail.
>
> In terms of logging Hadoop configuration for cluster:
>
> Default configurations which should be consistent (per documentation) are:
>
>
> In terms of the
>
> On Mon, Mar 11, 2013 at 4:43 AM, preethi ganeshan <
> preethiganeshan92@gmail.com> wrote:
>
>> Hi,
>>
>> I want to modify the Task.java so that it gives additional information in
>> the usrlogs files.
>> How do i go about the modification? I am new to Hadoop. Shall i simply
>> open the src .mapred . .... appropriate file in eclipse modify and save?
>> Will that help?
>>
>> Thank you
>>
>> Regards,
>> Preethi Ganeshan
>>
>
>
>
> --
> Ellis R. Miller
> 937.829.2380
>
>
> <http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
>
> Mundo Nulla Fides
>
>
>
> <http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
>
>
>
>
>


-- 
Ellis R. Miller
937.829.2380

<http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>

Mundo Nulla Fides


<http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>

Mime
View raw message