Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANb5z2JOC7O5SPb_d1Wyd1rFbeJdSatNuy3qo1TiENB2sGEw9w@mail.gmail.com>
References: 
 <CAHnpetQpcSqFhWFZ9ZJg6DkN02jeC=LpvJ0+VSP+oA+8c0DK+w@mail.gmail.com>
	<CANb5z2JOC7O5SPb_d1Wyd1rFbeJdSatNuy3qo1TiENB2sGEw9w@mail.gmail.com>
Date: Sun, 22 Nov 2015 18:09:04 -0600
Message-ID: 
 <CADH-_NKpDgM5wUE6emVsYooq230KjbO6bAMySKzGyiHmMwna-Q@mail.gmail.com>
Subject: Re: Custom Crunch Target and Counters
From: Micah Whitacre <mkwhitacre@gmail.com>
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a113cfee682d0b205252a081d

--001a113cfee682d0b205252a081d
Content-Type: text/plain; charset=UTF-8

Logged this request here to track progress...
https://issues.apache.org/jira/browse/CRUNCH-579

On Fri, Nov 20, 2015 at 12:44 PM, Josh Wills <josh.wills@gmail.com> wrote:

> Yeah, that does seem sort of crappy. Let me see what I can do.
>
> On Thu, Nov 19, 2015 at 10:39 PM, kulkarni.swarnim@gmail.com <
> kulkarni.swarnim@gmail.com> wrote:
>
>> Hello,
>>
>> So we wrote a custom Crunch Target to write data to a particular location
>> which included providing RecordWriters, OutputFormat, OutCommitters and
>> such. Now we  wanted to add Counters to get a count of how much data our
>> reducers are writing. A most obvious design was to use the
>> TaskAttemptContext that gets passed to the RecordWriter and use the
>> getCounter() method on it to manipulate the counters. However, that did not
>> work as expected and even though the counters were getting incremented,
>> they did not show up on the Resource Manager UI. On further investigation I
>> found that, cruch mangles this context object to add in a named output via
>> the CrunchOutputs class[1] which basically makes the counters useless
>> within the recordwriter class.
>>
>> Would it be a feasible enhancement for the CrunchOutputs to pass in the
>> original base context object along with the modified one with named outputs
>> so that it can be used for counters? Any other suggestions are most welcome
>> as well.
>>
>> Thanks,
>> Swarnim
>>
>> [1]
>> https://github.com/cloudera/crunch/blob/cdh5.4.8-release/crunch-core/src/main/java/org/apache/crunch/io/CrunchOutputs.java#L210-L232
>>
>
>

--001a113cfee682d0b205252a081d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Logged this request here to track progress...<div><a href=
=3D"https://issues.apache.org/jira/browse/CRUNCH-579">https://issues.apache=
.org/jira/browse/CRUNCH-579</a><br></div></div><div class=3D"gmail_extra"><=
br><div class=3D"gmail_quote">On Fri, Nov 20, 2015 at 12:44 PM, Josh Wills =
<span dir=3D"ltr">&lt;<a href=3D"mailto:josh.wills@gmail.com" target=3D"_bl=
ank">josh.wills@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex"><div dir=3D"ltr">Yeah, that does seem sort of crappy. Let me see wha=
t I can do.</div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmai=
l_extra"><br><div class=3D"gmail_quote">On Thu, Nov 19, 2015 at 10:39 PM, <=
a href=3D"mailto:kulkarni.swarnim@gmail.com" target=3D"_blank">kulkarni.swa=
rnim@gmail.com</a> <span dir=3D"ltr">&lt;<a href=3D"mailto:kulkarni.swarnim=
@gmail.com" target=3D"_blank">kulkarni.swarnim@gmail.com</a>&gt;</span> wro=
te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hello,<div><br></div=
><div>So we wrote a custom Crunch Target to write data to a particular loca=
tion which included providing RecordWriters, OutputFormat, OutCommitters an=
d such. Now we =C2=A0wanted to add Counters to get a count of how much data=
 our reducers are writing. A most obvious design was to use the TaskAttempt=
Context that gets passed to the RecordWriter and use the getCounter() metho=
d on it to manipulate the counters. However, that did not work as expected =
and even though the counters were getting incremented, they did not show up=
 on the Resource Manager UI. On further investigation I found that, cruch m=
angles this context object to add in a named output via the CrunchOutputs c=
lass[1] which basically makes the counters useless within the recordwriter =
class.=C2=A0</div><div><br></div><div>Would it be a feasible enhancement fo=
r the CrunchOutputs to pass in the original base context object along with =
the modified one with named outputs so that it can be used for counters? An=
y other suggestions are most welcome as well.</div><div><br></div><div>Than=
ks,</div><div>Swarnim</div><div><br></div><div>[1]=C2=A0<a href=3D"https://=
github.com/cloudera/crunch/blob/cdh5.4.8-release/crunch-core/src/main/java/=
org/apache/crunch/io/CrunchOutputs.java#L210-L232" target=3D"_blank">https:=
//github.com/cloudera/crunch/blob/cdh5.4.8-release/crunch-core/src/main/jav=
a/org/apache/crunch/io/CrunchOutputs.java#L210-L232</a>
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a113cfee682d0b205252a081d--