Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of xeonmailinglist@gmail.com
 designates 209.85.212.178 as permitted sender)
Message-ID: <5501C30A.2050308@gmail.com>
Date: Thu, 12 Mar 2015 16:47:06 +0000
From: xeonmailinglist-gmail <xeonmailinglist@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Fwd: Re: Prune out data to a specific reduce task
References: <72830A6F-963F-400B-B898-231454F6EE22@gmail.com>
In-Reply-To: <72830A6F-963F-400B-B898-231454F6EE22@gmail.com>
Content-Type: multipart/alternative;
 boundary="------------090105020805050803000000"

This is a multi-part message in MIME format.
--------------090105020805050803000000
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

If I use the partitioner, I must be able to tell map reduce to not 
execute values from a certain reduce tasks.

The method |public int getPartition(K key, V value, int numReduceTasks)| 
must always return a partition. I can’t return -1. Thus, I don’ t know 
how to tell Mapreduce to not execute data from a partition. Any suggestion?

———— Forwarded Message ————

Subject: Re: Prune out data to a specific reduce task

Date: Thu, 12 Mar 2015 12:40:04 -0400

From: Fei Hu hufei68@gmail.com <http://mailto:hufei68@gmail.com>

Reply-To: user@hadoop.apache.org

To: user@hadoop.apache.org

Maybe you could use Partitioner.class to solve your problem.

> On Mar 11, 2015, at 6:28 AM, xeonmailinglist-gmail 
> <xeonmailinglist@gmail.com <mailto:xeonmailinglist@gmail.com>> wrote:
>
> Hi,
>
> I have this job that has 3 map tasks and 2 reduce tasks. But, I want 
> to excludes data that will go to the reduce task 2. This means that, 
> only reducer 1 will produce data, and the other one will be empty, or 
> even it doesn't execute.
>
> How can I do this in MapReduce?
>
> <ExampleJobExecution.png>
>
>
> Thanks,
>
> -- 
> --

​

--------------090105020805050803000000
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div style="" class="markdown-here-wrapper"
      data-md-url="Thunderbird">
      <p style="margin: 1.2em 0px ! important;">If I use the
        partitioner, I must be able to tell map reduce to not execute
        values from a certain reduce tasks.</p>
      <p style="margin: 1.2em 0px ! important;">The method <code
          style="font-size: 0.85em; font-family:
          Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
          padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
          rgb(234, 234, 234); background-color: rgb(248, 248, 248);
          border-radius: 3px; display: inline;">public int
          getPartition(K key, V value, int numReduceTasks)</code> must
        always return a partition. I can’t return -1. Thus, I don’ t
        know how to tell Mapreduce to not execute data from a partition.
        Any suggestion?</p>
      <p style="margin: 1.2em 0px ! important;">———— Forwarded Message
        ————</p>
      <p style="margin: 1.2em 0px ! important;">Subject: Re: Prune out
        data to a specific reduce task</p>
      <p style="margin: 1.2em 0px ! important;">Date: Thu, 12 Mar 2015
        12:40:04 -0400</p>
      <p style="margin: 1.2em 0px ! important;">From: Fei Hu <a
          href="http://mailto:hufei68@gmail.com">hufei68@gmail.com</a></p>
      <p style="margin: 1.2em 0px ! important;">Reply-To:
        <a class="moz-txt-link-abbreviated" href="mailto:user@hadoop.apache.org">user@hadoop.apache.org</a></p>
      <p style="margin: 1.2em 0px ! important;">To:
        <a class="moz-txt-link-abbreviated" href="mailto:user@hadoop.apache.org">user@hadoop.apache.org</a></p>
      <p style="margin: 1.2em 0px ! important;">Maybe you could use
        Partitioner.class to solve your problem.</p>
      <p style="margin: 1.2em 0px ! important;"></p>
      <div class="markdown-here-exclude">
        <p></p>
        <blockquote type="cite" class="">
          <div class="">On Mar 11, 2015, at 6:28 AM,
            xeonmailinglist-gmail &lt;<a moz-do-not-send="true"
              href="mailto:xeonmailinglist@gmail.com" class="">xeonmailinglist@gmail.com</a>&gt;
            wrote:</div>
          <br class="Apple-interchange-newline">
          <div class="">
            <meta http-equiv="content-type" content="text/html;
              charset=UTF-8" class="">
            <div bgcolor="#FFFFFF" text="#000000" class=""> Hi,<br
                class="">
              <br class="">
              I have this job that has 3 map tasks and 2 reduce tasks.
              But, I want to excludes data that will go to the reduce
              task 2. This means that, only reducer 1 will produce data,
              and the other one will be empty, or even it doesn't
              execute.<br class="">
              <br class="">
              How can I do this in MapReduce?<br class="">
              <br class="">
              <span id="cid:part1.00040306.02030506@gmail.com" class="">&lt;ExampleJobExecution.png&gt;</span><br
                class="">
              <br class="">
              <br class="">
              Thanks,<br class="">
              <br class="">
              <pre class="moz-signature" cols="72">-- 
--</pre>
            </div>
          </div>
        </blockquote>
        <p></p>
      </div>
      <p style="margin: 1.2em 0px ! important;"></p>
      <div
title="MDH:SWYgSSB1c2UgdGhlIHBhcnRpdGlvbmVyLCBJIG11c3QgYmUgYWJsZSB0byB0ZWxsIG1hcCByZWR1Y2UgdG8gbm90IGV4ZWN1dGUgdmFsdWVzIGZyb20gYSBjZXJ0YWluIHJlZHVjZSB0YXNrcy48YnI+
PGJyPlRoZSBtZXRob2QgYHB1YmxpYyBpbnQgZ2V0UGFydGl0aW9uKEsga2V5LCBWIHZhbHVlLCBp
bnQgbnVtUmVkdWNlVGFza3MpYCBtdXN0IGFsd2F5cyByZXR1cm4gYSBwYXJ0aXRpb24uIEkgY2Fu
J3QgcmV0dXJuIC0xLiBUaHVzLCBJIGRvbicgdCBrbm93IGhvdyB0byB0ZWxsIE1hcHJlZHVjZSB0
byBub3QgZXhlY3V0ZSBkYXRhIGZyb20gYSBwYXJ0aXRpb24uIEFueSBzdWdnZXN0aW9uPzxicj48
ZGl2IGNsYXNzPSJtb3otZm9yd2FyZC1jb250YWluZXIiPjxicj48YnI+LS0tLS0tLS0gRm9yd2Fy
ZGVkIE1lc3NhZ2UgLS0tLS0tLS08dGFibGUgY2xhc3M9Im1vei1lbWFpbC1oZWFkZXJzLXRhYmxl
IiBib3JkZXI9IjAiIGNlbGxwYWRkaW5nPSIwIiBjZWxsc3BhY2luZz0iMCI+PHRib2R5Pjx0cj48
dGggYWxpZ249IlJJR0hUIiBub3dyYXA9IiIgdmFsaWduPSJCQVNFTElORSI+U3ViamVjdDogPC90
aD48dGQ+UmU6IFBydW5lIG91dCBkYXRhIHRvIGEgc3BlY2lmaWMgcmVkdWNlIHRhc2s8L3RkPjwv
dHI+PHRyPjx0aCBhbGlnbj0iUklHSFQiIG5vd3JhcD0iIiB2YWxpZ249IkJBU0VMSU5FIj5EYXRl
OiA8L3RoPjx0ZD5UaHUsIDEyIE1hciAyMDE1IDEyOjQwOjA0IC0wNDAwPC90ZD48L3RyPjx0cj48
dGggYWxpZ249IlJJR0hUIiBub3dyYXA9IiIgdmFsaWduPSJCQVNFTElORSI+RnJvbTogPC90aD48
dGQ+RmVpIEh1ICZsdDtodWZlaTY4QGdtYWlsLmNvbSZndDs8L3RkPjwvdHI+PHRyPjx0aCBhbGln
bj0iUklHSFQiIG5vd3JhcD0iIiB2YWxpZ249IkJBU0VMSU5FIj5SZXBseS1UbzogPC90aD48dGQ+
dXNlckBoYWRvb3AuYXBhY2hlLm9yZzwvdGQ+PC90cj48dHI+PHRoIGFsaWduPSJSSUdIVCIgbm93
cmFwPSIiIHZhbGlnbj0iQkFTRUxJTkUiPlRvOiA8L3RoPjx0ZD51c2VyQGhhZG9vcC5hcGFjaGUu
b3JnPC90ZD48L3RyPjwvdGJvZHk+PC90YWJsZT4KPGJyPjxicj48bWV0YSBodHRwLWVxdWl2PSJD
b250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbCBjaGFyc2V0PXVzLWFzY2lpIj48bWV0YSBo
dHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbCBjaGFyc2V0PXVzLWFz
Y2lpIj5NYXliZSB5b3UgY291bGQgdXNlJm5ic3A7PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBN
b25hY287IGZvbnQtc2l6ZTogMTFweDsiIGNsYXNzPSIiPlBhcnRpdGlvbmVyLmNsYXNzIDwvc3Bh
bj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMXB4OyIgY2xhc3M9IiI+dG8gc29sdmUgeW91ciZu
YnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMXB4OyIgY2xhc3M9IiI+cHJvYmxl
bS48L3NwYW4+PGRpdiBjbGFzcz0iIj48Zm9udCBjbGFzcz0iIiBmYWNlPSJNb25hY28iPjxzcGFu
IHN0eWxlPSJmb250LXNpemU6IDExcHg7IiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+PC9zcGFuPjwv
Zm9udD48ZGl2IGNsYXNzPSIiPjxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIiPjxkaXYg
Y2xhc3M9IiI+T24gTWFyIDExLCAyMDE1LCBhdCA2OjI4IEFNLCB4ZW9ubWFpbGluZ2xpc3QtZ21h
aWwgJmx0OzxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIgaHJlZj0ibWFpbHRvOnhlb25tYWlsaW5n
bGlzdEBnbWFpbC5jb20iIGNsYXNzPSIiPnhlb25tYWlsaW5nbGlzdEBnbWFpbC5jb208L2E+Jmd0
OyB3cm90ZTo8L2Rpdj48YnIgY2xhc3M9IkFwcGxlLWludGVyY2hhbmdlLW5ld2xpbmUiPjxkaXYg
Y2xhc3M9IiI+CiAgCgogICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50
PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiIGNsYXNzPSIiPgogIAogIDxkaXYgYmdjb2xvcj0i
I0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCIgY2xhc3M9IiI+CiAgICBIaSw8YnIgY2xhc3M9IiI+CiAg
ICA8YnIgY2xhc3M9IiI+CiAgICBJIGhhdmUgdGhpcyBqb2IgdGhhdCBoYXMgMyBtYXAgdGFza3Mg
YW5kIDIgcmVkdWNlIHRhc2tzLiBCdXQsIEkgd2FudAogICAgdG8gZXhjbHVkZXMgZGF0YSB0aGF0
IHdpbGwgZ28gdG8gdGhlIHJlZHVjZSB0YXNrIDIuIFRoaXMgbWVhbnMgdGhhdCwKICAgIG9ubHkg
cmVkdWNlciAxIHdpbGwgcHJvZHVjZSBkYXRhLCBhbmQgdGhlIG90aGVyIG9uZSB3aWxsIGJlIGVt
cHR5LAogICAgb3IgZXZlbiBpdCBkb2Vzbid0IGV4ZWN1dGUuPGJyIGNsYXNzPSIiPgogICAgPGJy
IGNsYXNzPSIiPgogICAgSG93IGNhbiBJIGRvIHRoaXMgaW4gTWFwUmVkdWNlPzxiciBjbGFzcz0i
Ij4KICAgIDxiciBjbGFzcz0iIj4KICAgIDxzcGFuIGlkPSJjaWQ6cGFydDEuMDAwNDAzMDYuMDIw
MzA1MDZAZ21haWwuY29tIiBjbGFzcz0iIj4mbHQ7RXhhbXBsZUpvYkV4ZWN1dGlvbi5wbmcmZ3Q7
PC9zcGFuPjxiciBjbGFzcz0iIj4KICAgIDxiciBjbGFzcz0iIj4KICAgIDxiciBjbGFzcz0iIj4K
ICAgIFRoYW5rcyw8YnIgY2xhc3M9IiI+CiAgICA8YnIgY2xhc3M9IiI+CiAgICA8cHJlIGNsYXNz
PSJtb3otc2lnbmF0dXJlIiBjb2xzPSI3MiI+LS0gCi0tPC9wcmU+CiAgPC9kaXY+Cgo8L2Rpdj48
L2Jsb2NrcXVvdGU+PC9kaXY+PGJyIGNsYXNzPSIiPjwvZGl2Pgo8YnI+PC9kaXY+PGJyPg=="
style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0;">​</div>
    </div>
  </body>
</html>

--------------090105020805050803000000--