Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Date: Tue, 19 Apr 2016 19:37:25 +0000 (UTC)
From: =?utf-8?Q?Cl=C3=A9ment_MATHIEU_=28JIRA=29?= <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12915201.1448237164000.270749.1461094645507@Atlassian.JIRA>
In-Reply-To: <JIRA.12915201.1448237164000@Atlassian.JIRA>
References: <JIRA.12915201.1448237164000@Atlassian.JIRA>
 <JIRA.12915201.1448237164337@arcas>
Subject: [jira] [Commented] (CRUNCH-579) Support Counters from Custom
 RecordWriters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CRUNCH-579?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1524=
8468#comment-15248468 ]=20

Cl=C3=A9ment MATHIEU commented on CRUNCH-579:
----------------------------------------

I was facing the same issue last week. Downloaded the patch, back-ported it=
 to crunch-0.11.0-cdh5.4.2 / Hadoop 2. It works fine and we are now using i=
t in production.

It would be great if this patch could be merged. I don't see why it would n=
ot work on master, but can give it a try if really needed.

> Support Counters from Custom RecordWriters
> ------------------------------------------
>
>                 Key: CRUNCH-579
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-579
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Micah Whitacre
>            Assignee: Josh Wills
>         Attachments: CRUNCH-579.patch
>
>
> A consumer mentioned this on the mailing list:
> {quote}
> So we wrote a custom Crunch Target to write data to a particular location=
 which included providing RecordWriters, OutputFormat, OutCommitters and su=
ch. Now we  wanted to add Counters to get a count of how much data our redu=
cers are writing. A most obvious design was to use the TaskAttemptContext t=
hat gets passed to the RecordWriter and use the getCounter() method on it t=
o manipulate the counters. However, that did not work as expected and even =
though the counters were getting incremented, they did not show up on the R=
esource Manager UI. On further investigation I found that, cruch mangles th=
is context object to add in a named output via the CrunchOutputs class[1] w=
hich basically makes the counters useless within the recordwriter class.=20
> Would it be a feasible enhancement for the CrunchOutputs to pass in the o=
riginal base context object along with the modified one with named outputs =
so that it can be used for counters? Any other suggestions are most welcome=
 as well.
> {quote}
> http://mail-archives.apache.org/mod_mbox/crunch-user/201511.mbox/%3CCAHnp=
etQpcSqFhWFZ9ZJg6DkN02jeC%3DLpvJ0%2BVSP%2BoA%2B8c0DK%2Bw%40mail.gmail.com%3=
E


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)