Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C27A2DCAE for ; Sat, 18 May 2013 03:16:46 +0000 (UTC) Received: (qmail 97426 invoked by uid 500); 18 May 2013 03:16:42 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 97077 invoked by uid 500); 18 May 2013 03:16:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 97055 invoked by uid 99); 18 May 2013 03:16:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 May 2013 03:16:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of publicnetworkservices@gmail.com designates 209.85.128.48 as permitted sender) Received: from [209.85.128.48] (HELO mail-qe0-f48.google.com) (209.85.128.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 May 2013 03:16:33 +0000 Received: by mail-qe0-f48.google.com with SMTP id 9so3073044qea.7 for ; Fri, 17 May 2013 20:16:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=hR/ETX+BkQfGXWGYKtTjqwHXLXBTjN5DWQmEVFH+ARw=; b=hS4CeDNMJ4mRLaT2nxWP4ah4mBKGanhTfFfYOX5DU0gD6yg4I7WJdumQRzPx97cOxn WTMkn4OorV4Ozt/J8a6Y0/T5S+3rHD/HCGqm2/smBQpgtmaBJsQFqOGESkeS76HtKiR1 Bfe1SQYuApABUoWWO14ju/1WRMjDamdTn3jMckZ9Hg1/xxHVYqZXCEVB38/+zl64As+w DtK58FeRvNso+u1Zvhl0RxlWB6gOSH3Ud98pM0xg/mYEfHo3Y9nIHtCoeBlvMTG7GOs4 eeO35zJAsGci/hAvMgTGGZ9UV/QGoZ9W+b3Cq5H6VHrsCNju/1Yo0APkz4gajBsHzoXB 2uzw== MIME-Version: 1.0 X-Received: by 10.229.59.71 with SMTP id k7mr16251164qch.50.1368846972986; Fri, 17 May 2013 20:16:12 -0700 (PDT) Received: by 10.49.61.226 with HTTP; Fri, 17 May 2013 20:16:12 -0700 (PDT) In-Reply-To: References: Date: Fri, 17 May 2013 20:16:12 -0700 Message-ID: Subject: Re: Passing values from InputFormat via the Configuration object From: Public Network Services To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c2a8e493cef004dcf5851b X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2a8e493cef004dcf5851b Content-Type: text/plain; charset=ISO-8859-1 I tested both options and the "hack" is the only one working. No counter can be created in the custom InputFormat. If anyone has any better alternative, please advise. On Fri, May 17, 2013 at 5:53 PM, Public Network Services < publicnetworkservices@gmail.com> wrote: > A more standard approach could be converting the JobContext parameter of > the getSplits() method, into a Job object, which then allows retrieving the > counters, e.g.: > > public List getSplits(JobContext job) throws IOException > { > ... > Job work = new Job(job.getConfiguration()); > Counters counters = work.getCounters(); > } > > Would that be correct? > > > On Fri, May 17, 2013 at 5:33 PM, Public Network Services < > publicnetworkservices@gmail.com> wrote: > >> Hi... >> >> I need to communicate some proprietary number (long) values from the >> getSplits() method of a custom InputFormat class to the Hadoop driver class >> (used to launch the job), but the JobContext object passed to the >> getSplits() method has no access to a Counters object. >> >> From the source code, it seems that the Configuration object of the >> launched job is passed around, so the JobContext object of getSplits() has >> direct access to it via getConfiguration(). >> >> So, what about using a loop like >> >> Job job = ... // The launched job >> Configuration conf = job.getConfiguration(); >> while (!job.isComplete()) { >> // Read the values from the configuration >> } >> >> from the driver class, which presumably runs in the same framework that >> creates the splits? >> >> The getSplits() method of the custom InputFormat would set each of the >> values once. >> >> All this does seem like a hack, so I would like some expert advice before >> starting implementation. That is, >> >> 1. Will it work? >> 2. Is there a better method? >> >> Thanks! >> >> > --001a11c2a8e493cef004dcf5851b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I tested both options and the "hack" is the only= one working. No counter can be created in the custom InputFormat.

=
If anyone has any better alternative, please advise.=A0


On Fri,= May 17, 2013 at 5:53 PM, Public Network Services <publicnet= workservices@gmail.com> wrote:
A more standard approach co= uld be converting the JobContext parameter of the getSplits() method, into = a Job object, which then allows retrieving the counters, e.g.:

public Lis= t<InputSplit> getSplits(JobContext job) throws IOException
{
...
Job work =3D new Job(job.getConfiguration());
Counters counters =3D wor= k.getCounters();
= }

Would that be correct?


On Fri, May 17, 2013 at 5:33 PM, Public Network Services <publicnetworkservices@gmail.com> wrote:
Hi...

I = need to communicate some proprietary number (long) values from the getSplit= s() method of a custom InputFormat class to the Hadoop driver class (used t= o launch the job), but the JobContext object passed to the getSplits() meth= od has no access to a Counters object.

From the source code, it seems that the Configuration o= bject of the launched job is passed around, so the JobContext object of=A0g= etSplits()=A0has direct access to it via getConfiguration().

So, what about using a loop like

=A0 =A0 =A0 =A0 Job job =3D ... // The launched job
=A0 =A0 = =A0 =A0 Configuration conf =3D job.getConfiguration();
=A0 =A0 = =A0 =A0 while (!job.isComplete()) {
=A0 =A0 =A0 =A0 // Read t= he values from the configuration
=A0 =A0 =A0 =A0 }

=
from the driver class, which presumably runs in the same framewo= rk that creates the splits?

The getSplits() method of the=A0custom InputFormat=A0wo= uld set each of the values once.

All this does see= m like a hack, so I would like some expert advice before starting implement= ation. That is,
  1. Will it work?
  2. Is there a better method?
= Thanks!



--001a11c2a8e493cef004dcf5851b--