Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Date: Mon, 23 Nov 2015 00:14:11 +0000 (UTC)
From: "Josh Wills (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12915201.1448237164000.153671.1448237651026@Atlassian.JIRA>
In-Reply-To: <JIRA.12915201.1448237164000@Atlassian.JIRA>
References: <JIRA.12915201.1448237164000@Atlassian.JIRA>
 <JIRA.12915201.1448237164337@arcas>
Subject: [jira] [Commented] (CRUNCH-579) Support Counters from Custom
 RecordWriters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021260#comment-15021260 ] 

Josh Wills commented on CRUNCH-579:
-----------------------------------

Seems like an obviously good idea.

> Support Counters from Custom RecordWriters
> ------------------------------------------
>
>                 Key: CRUNCH-579
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-579
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Micah Whitacre
>            Assignee: Josh Wills
>         Attachments: CRUNCH-579.patch
>
>
> A consumer mentioned this on the mailing list:
> {quote}
> So we wrote a custom Crunch Target to write data to a particular location which included providing RecordWriters, OutputFormat, OutCommitters and such. Now we  wanted to add Counters to get a count of how much data our reducers are writing. A most obvious design was to use the TaskAttemptContext that gets passed to the RecordWriter and use the getCounter() method on it to manipulate the counters. However, that did not work as expected and even though the counters were getting incremented, they did not show up on the Resource Manager UI. On further investigation I found that, cruch mangles this context object to add in a named output via the CrunchOutputs class[1] which basically makes the counters useless within the recordwriter class. 
> Would it be a feasible enhancement for the CrunchOutputs to pass in the original base context object along with the modified one with named outputs so that it can be used for counters? Any other suggestions are most welcome as well.
> {quote}
> http://mail-archives.apache.org/mod_mbox/crunch-user/201511.mbox/%3CCAHnpetQpcSqFhWFZ9ZJg6DkN02jeC%3DLpvJ0%2BVSP%2BoA%2B8c0DK%2Bw%40mail.gmail.com%3E


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)