hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From edward choi <mp2...@gmail.com>
Subject Re: how to figure out the range of a split that failed?
Date Tue, 06 Jul 2010 05:07:45 GMT
Thanks for the tip.
I actually already have tried your method. The command I wrote is like below

cerr << "reporter:counter:SkippingTaskCounters,MapProcessedRecords,1\n";

This actually produced some skipped records in skip folder. But the problem
is that the skipped records' text was all messed up. So I couldn't recycle
them. The broken text is at the end of this mail.
I don't know the reason. Maybe it's because I wrote some other information
on error stream(such as document ID. The command below is the one)

cerr << "Processing: " << docID << endl;

Anyway if I happen to have any progress, I will update through this post.

Broken Text:
SEQ!org.apache.hadoop.io.LongWritableorg.apache.hadoop.io.Text*org.apache.hadoop.io.compress.DefaultCodec�9/S庇4z����9/S庇4z��
x��x�`[0텬�,wx�c�c]r8�$x�U]oW}i�H�Hi��Z�b+X�r�y접�"H���\��탈�y��
6�%�b��EMLの]`[}Ø�}�則�霙�綴�愁P�<�q�賽���\��5~庾X�xp�c≠�=��<�嵬p|3:�B�A?-8�勁쫑�2��_�@���F#A�6
G�}殘�蛛甦Q�$潾�,8��0鉛=�츈^曠��G�└&?日���;=~兪����묽��씔�nl�o]�판��M�d橫罹�|잣��C�噫�0����p`���i�U���z�4t^d芚Mc}�qf��뎬�����_^朽c���V��}S�V콸�Q�%κ쐴*�O_�����s������:�kn��(����b��RX�+ohS��8���梁=㈍哥�0%DGw當�.
�3V���y�분�!A�0%�;���瀏��茁�'&?
�N�jP����$��� %�m��밑!c��ls����<``% �����廣�>Y�i���璽��Fb賃$x좃_�W�L
�\F�gi�Ix8���廟別<�毫�<u9�BI(5S��u��後�
#��~��弛欒��鯖�`h�����|�9�c�高�:課Y�Q��������銳�占�쇽��p
-�cK��qY%�OR���55�pJ��潁~��Qt�>HywK�\Dz觴���U��(����/����渾�Re'%�s0k咫키L�%{K$��U+���1�B5�7j�-�f~��~頃�K�`c\�G�+t-��dJc|s��b�vWA�2�荳�f�X�却"��M�����W�.����D=O~��/���$
s��헷1M�e����긺 ^�9��1�揷�]奸��;���0-�굇���
�w욜=l泊�踰=c��u��瓣��S��*�����奈y1룡bM]��X�씬�7h짼Zx�\琅윰'��n?-�s�?��q�弘벧MV�*距�已희�岸侖:N�����刃2�


2010/7/6 Sharad Agarwal <sharadag@yahoo-inc.com>

> to be precise you have to write on error stream ->
> for map:
> reporter:counter:SkippingTaskCounters,MapProcessedRecords,<count>
>
> for reduce:
> reporter:counter:SkippingTaskCounters,ReduceProcessedGroups,<count>
>
>
> edward choi wrote:
>
>> Thanks for the response. I went to the web page you told me and several
>> other pages that I found.
>> I am still not sure if I got it right.
>> If I am trying to increment COUNTER_MAP_PROCESS_RECORDS using Hadoop
>> Streaming, is the example below the way to do it? (assuming that I am using
>> c++)
>>
>> example:
>> cerr << "reporter:counter:counters,linecount,1" << endl;
>>
>>
>>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message