hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ellis Miller <outlaw...@gmail.com>
Subject Re:
Date Mon, 11 Mar 2013 11:31:29 GMT
Sorry...was typing another email:)

Anyway, default config files across Master and Slave Nodes:

   - /src/core-site.xml, src/hdfs/hdfs-default.xml and the one you would
   want to be standard, of course, the src/mapred/mapred-default.xml

For each DataNode / Task Tracker across however many Virtual Machines could
/ would customize the mapred*.xml file in particular:

   - conf/core-site.xml, conf/hdfs-site.xml, and conf/mapred-site.xml

Again, using *log4j.properties* on the Slave sites under the conf/
directory very flexible in terms of metrics, etc. you would like to
custom-log say for each TaskTracker node in the same site-specific Slave
directory as the* /conf/hadoop-env.sh* which you can also modify on each
Slave Node for various options (both log4j.properties and the hadoop-env.sh
files provide fairly extensive means to log most of what you would be
interested in along with the memory settings, in particular, in the
conf/mapred-site.xml files.

However, simulated cluster on single node configuration is actually more
confusing in that regard in your case if it's a single node then extending
the *Task.java* source file or more specifically the package including that
file then compiling as .jar (Java archive) and uploading may arguably be
just as simple yet the other site-specific files for the DataNode and
TaskTracker along with the hadoop-env.sh are very, very flexible for
logging or otherwise setting specific options for Java Heap size, etc. (in
conjunction with the log4j.properties).

In short, would look at the /conf/mapred-site.xml, /conf/hadoop-env.sh, and
log4j.properties files first as think it could be a very quick
already-available fix in terms of the additional logging you want to



On Mon, Mar 11, 2013 at 7:13 AM, Ellis Miller <outlawdba@gmail.com> wrote:

> Could use an UDF to extend the Task.java yet if it's a true cluster or
> appropriately simulated then basic standard:
> 1. Master is comprised of NameNode and JobTracker
> 2. All other are Slave (hosts) or Virtual Machines consisting of DataNodes
> and TaskTrackers in particular
> So, on Slaves Hosts could easily modify mapred.trasktracker.* parameters
> in the customized Slave VMs mapred-site.xml files for logging Memory
> Monitoring as (assuming) you are concerned with the TT memory usage which
> can cause a task to fail.
> In terms of logging Hadoop configuration for cluster:
> Default configurations which should be consistent (per documentation) are:
> In terms of the
> On Mon, Mar 11, 2013 at 4:43 AM, preethi ganeshan <
> preethiganeshan92@gmail.com> wrote:
>> Hi,
>> I want to modify the Task.java so that it gives additional information in
>> the usrlogs files.
>> How do i go about the modification? I am new to Hadoop. Shall i simply
>> open the src .mapred . .... appropriate file in eclipse modify and save?
>> Will that help?
>> Thank you
>> Regards,
>> Preethi Ganeshan
> --
> Ellis R. Miller
> 937.829.2380
> <http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
> Mundo Nulla Fides
> <http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>

Ellis R. Miller


Mundo Nulla Fides


View raw message