From user-return-3374-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 4 08:00:55 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E795EEB06 for ; Tue, 4 Dec 2012 08:00:55 +0000 (UTC) Received: (qmail 32024 invoked by uid 500); 4 Dec 2012 08:00:51 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 31628 invoked by uid 500); 4 Dec 2012 08:00:50 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 31595 invoked by uid 99); 4 Dec 2012 08:00:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Dec 2012 08:00:49 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [98.136.217.99] (HELO nm24-vm4.bullet.mail.gq1.yahoo.com) (98.136.217.99) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Dec 2012 08:00:38 +0000 Received: from [98.137.12.191] by nm24.bullet.mail.gq1.yahoo.com with NNFMP; 04 Dec 2012 08:00:16 -0000 Received: from [208.71.42.209] by tm12.bullet.mail.gq1.yahoo.com with NNFMP; 04 Dec 2012 08:00:16 -0000 Received: from [127.0.0.1] by smtp220.mail.gq1.yahoo.com with NNFMP; 04 Dec 2012 08:00:16 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1354608016; bh=VvqJ0r5n3mKklXkwuOSzaQkauMS/WKP8838CD0d2alc=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:From:To:References:In-Reply-To:Subject:Date:Message-ID:MIME-Version:Content-Type:X-Mailer:Thread-Index:Content-Language; b=UQ7x96o0io96aAI6QniE1R/G+y998BWT1iFWYqwwf+YrYH6wFrhcFiCi6Q5iwz0GLTBJfzJzQjlitX4QQNFf5Ui88vvXDh6v9x/83E56YvnvAfmx83gSeQ+PAy+dA8EiwnU8Bxeh3KoS03v0+Si9InLUqWXLwOZhlwWFxpAzyqw= X-Yahoo-Newman-Id: 703459.51811.bm@smtp220.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: H9FZl88VM1kXTNEunornxGLY4goSLWg.NJWS8OpEoQMS1di WjyDPl88B8XsG01oh79v8kb4h7ZWO9o__BnlGRoDU1lWbeBBj7D2N6KL0xIx uCfhQrgy2Tm_0xoC6y09NiYJTpTYWJrMVOqxHCg3CgRXR_ZF64xR1XYtdgfK nohdo8nbS7pCoUH1moMYfS59k334ExB6vXlZkDGaCGdcYxKJxZOT7yUrcV3f mnAl8WsZh_Z3XY0ypJkMTVXxB7VI9tEwJmx.DYXFX6fA3Fl8dsMqcx2lGgkL OCNYNkJwh6.U8lg54F9OKE9Lipd6J691uhaNzM7IvjEYUQfsBmffZWy7lZwx Qrn1Q7pMfGsqZ0xTEOpfpcLdoWtjb2El_.IXzA34ONsrvgVWIa4wBWdm7.8D NP6Ce3df__7rEoUVl9kVP_OoYUmYT1meTHe0MMq4CYs5dKC7Y_yHkyxtzjdJ XIs6OnSNfna7li_Yi9CUsapuRSUU6o9wI_LaJpwtZ_Pw2HoV9GJ5fiNlvrOC dWLq3S8D2GSEGnpugSbst8or60MPcbi2Unls7gH.dKfoQVprb52qgA2vBYEk 0SShKERreVYWQ8B8lhsA- X-Yahoo-SMTP: k2gD1GeswBAV_JFpZm8dmpTCwr4ufTKOyA-- Received: from sattelite (davidparks21@113.161.75.108 with login) by smtp220.mail.gq1.yahoo.com with SMTP; 04 Dec 2012 00:00:16 -0800 PST From: "David Parks" To: References: <17861B3B-BAF3-4563-B86A-CB882A5F82BE@hsk.hk> <22C72682-E38B-4943-B462-E7BEE0649C1E@hortonworks.com> In-Reply-To: <22C72682-E38B-4943-B462-E7BEE0649C1E@hortonworks.com> Subject: RE: [Bulk] Re: Failed To Start SecondaryNameNode in Secure Mode Date: Tue, 4 Dec 2012 15:00:07 +0700 Message-ID: <019601cdd1f5$61e25cf0$25a716d0$@yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0197_01CDD230.0E424660" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQFsl9gEBFVA1MbFme3ekc8kjT6FqQEYHDGcAjrdDYcCigU6Jpib0hYw Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_0197_01CDD230.0E424660 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I'm curious about profiling, I see some documentation about it (1.0.3 on AWS), but the references to JobConf seem to be for the "old api" and I've got everything running on the "new api". I've got a job to handle processing of about 30GB of compressed CSVs and it's taking over a day with 3 m1.medium boxes, more than I expected, so I'd like to see where the time is being spent. http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html#Profiling I've never set up any kind of profiling, so I don't really know what to expect here. Any pointers to help me set up what's suggested here? Am I correct in understanding that this doc is a little outdated? ------=_NextPart_000_0197_01CDD230.0E424660 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I’m curious about profiling, I see some documentation about it = (1.0.3 on AWS), but the references to JobConf seem to be for the = “old api” and I’ve got everything running on the = “new api”.

 

I’ve got a job to handle processing of about 30GB of compressed = CSVs and it’s taking over a day with 3 m1.medium boxes, more than = I expected, so I’d like to see where the time is being = spent.

 

http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html#Profiling

 

I’ve never set up any kind of profiling, so I don’t = really know what to expect here.

 

Any pointers to help me set up what’s suggested here? Am I = correct in understanding that this doc is a little = outdated?

------=_NextPart_000_0197_01CDD230.0E424660--