Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6842710BB1 for ; Sat, 30 Nov 2013 23:33:03 +0000 (UTC) Received: (qmail 72290 invoked by uid 500); 30 Nov 2013 23:32:58 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 72033 invoked by uid 500); 30 Nov 2013 23:32:58 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 72026 invoked by uid 99); 30 Nov 2013 23:32:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Nov 2013 23:32:57 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kawa.adam@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Nov 2013 23:32:52 +0000 Received: by mail-ie0-f176.google.com with SMTP id at1so18637235iec.35 for ; Sat, 30 Nov 2013 15:32:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=uoh9yXz8DhTvhye6QFNgl5UCh/Al/IS2i1SJoPodQQc=; b=TLM9vRAAsV5OINYN3loGEUlB9KrtqbI1QMutkTaYESlzc8v5IDsVqjULrtOyoir2em Fo/mszmIYmoWOweMVDnqzspu/ozzWvK9JmzZ67VQUwSuCwH5/2xhGxABFhfloLiQM+vx wx9xwoSqrx5qtRPV5dy3Z+LHMtPebUBIf0h8UCt4v5cbEgbIy4pvonDgCBYox/XGlx5m 7e9QBvrv4M33JAsoiAucsklKR6AQlQI11XXP8WoGc8YMdHR4BfiuLCoHQipzY99fV2sH eO9nXA4TqZ1l8lXmLXudOpxNx4ma48TPdCYZV5A7eybv57KWT65kMZtNzPDHDih6rVfF IrtA== MIME-Version: 1.0 X-Received: by 10.50.41.106 with SMTP id e10mr11911950igl.34.1385854351326; Sat, 30 Nov 2013 15:32:31 -0800 (PST) Received: by 10.42.153.136 with HTTP; Sat, 30 Nov 2013 15:32:31 -0800 (PST) In-Reply-To: <5E71DA64-30FD-4CD3-A468-CBAC6D91B6A2@uvic.ca> References: <5E71DA64-30FD-4CD3-A468-CBAC6D91B6A2@uvic.ca> Date: Sun, 1 Dec 2013 01:32:31 +0200 Message-ID: Subject: Re: Can't get a streaming job to use a custom partitioner From: Adam Kawa To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0122f0965243f804ec6d5cd8 X-Virus-Checked: Checked by ClamAV on apache.org --089e0122f0965243f804ec6d5cd8 Content-Type: text/plain; charset=ISO-8859-1 1) Could you print the output of: $ jar tf ./NumericPartitioner.jar 2) Could you try: $ export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:./NumericPartitioner.jar" $ ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \ -libjars ./NumericPartitioner.jar \ -input /input -output /output/keys -mapper "map_threeJoin.py" -reducer "keycount.py" \ -partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file "keycount.py" 2013/11/18 Ben K > I need help. No matter what I do I can't seem to get Hadoop to find my > custom partitioner. > Here is the command I am running: > > ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \ > -libjars ./NumericPartitioner.jar \ > -input /input -output /output/keys -mapper "map_threeJoin.py" -reducer > "keycount.py" \ > -partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file > "keycount.py" > > (The code of NumericPartitioner is very simple, and is here: > http://pastebin.com/ZEK7N1RN) > But no matter what I do, it gives: > > -partitioner : class not found : newjoin.NumericPartitioner > > Does anyone have any idea why it might be going wrong? > > Ben K > > > --089e0122f0965243f804ec6d5cd8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
1) Could you print the output of:
$ jar tf=A0./Numeri= cPartitioner.jar

2) Could you try:
$ export HADOOP= _CLASSPATH=3D"$HADOOP_CLASSPATH:./NumericPartitioner.jar"

$ ../bin/hadoop jar ../contrib/streaming/hadoop-streami= ng-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -out= put /output/keys -mapper "map_threeJoin.py" -reducer "keycou= nt.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" = -file "keycount.py"



2013/11= /18 Ben K <blk@uvic.ca>
I need help. No matter what I do I can't seem to get Hadoop to find my = custom partitioner.
Here is the command I am running:

../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -output /output/keys -mapper "map_threeJoin.py" -re= ducer "keycount.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" = -file "keycount.py"

(The code of NumericPartitioner is very simple, and is here: http://pastebin.com/ZEK7N1RN)
But no matter what I do, it gives:

-partitioner : class not found : newjoin.NumericPartitioner

Does anyone have any idea why it might be going wrong?

Ben K



--089e0122f0965243f804ec6d5cd8--