Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BBEB6176EC for ; Sat, 21 Feb 2015 07:57:22 +0000 (UTC) Received: (qmail 91948 invoked by uid 500); 21 Feb 2015 07:57:17 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 91826 invoked by uid 500); 21 Feb 2015 07:57:17 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 91809 invoked by uid 99); 21 Feb 2015 07:57:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Feb 2015 07:57:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of unmeshabiju@gmail.com designates 209.85.223.174 as permitted sender) Received: from [209.85.223.174] (HELO mail-ie0-f174.google.com) (209.85.223.174) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Feb 2015 07:56:47 +0000 Received: by iecar1 with SMTP id ar1so13123142iec.11 for ; Fri, 20 Feb 2015 23:56:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=PCuHWyFU59QuY+xB5oWqv0hgxObT4qWnr2benkGp0yM=; b=HpjR0a8cDC16ryjfyvtMD+Y8u27Jj3wimW9LSoFRSYdedNCrAH2GRvAanP3hmkgOWM qc8dGiOrcRURDNsHtD8yHDrfKZkXCATsnDLWCnZZgQoA1UCmRMqFad1YZ2zbxZaQbT+o HrH/eDK4uA9TZT/fovMZfntxRsjMBt/6JZYwdUD126l0xsfEiJj/QAvDj2guxPBlV7wA pUgYsJ9ELRAn8EnWL9eSFF35NVNq6F5QbFTGkEXQs3D0FPfJWMtomQzVlLMmUsqw0f3b grLV4ytR7DT8LKQM86AboiQ+KbVe4wP3R1UQvk7bHEdEuiye583qg9QztaPhVjq9kSXc 8uXg== X-Received: by 10.42.121.84 with SMTP id i20mr2010688icr.20.1424505360030; Fri, 20 Feb 2015 23:56:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.36.106.3 with HTTP; Fri, 20 Feb 2015 23:55:19 -0800 (PST) From: unmesha sreeveni Date: Sat, 21 Feb 2015 13:25:19 +0530 Message-ID: Subject: Get method in Writable To: User Hadoop Content-Type: multipart/alternative; boundary=20cf301b65c3f748be050f947fca X-Virus-Checked: Checked by ClamAV on apache.org --20cf301b65c3f748be050f947fca Content-Type: text/plain; charset=UTF-8 Am I able to get the values from writable of a previous job. ie I have 2 MR jobs *MR 1:* I need to pass 3 element as values from reducer and the key is NullWritable. So I created a custom writable class to achieve this. * public class TreeInfoWritable implements Writable{* * DoubleWritable entropy;* * IntWritable sum;* * IntWritable clsCount;* * ..* *}* *MR 2:* I need to access MR 1 result in MR2 mapper setup function. And I accessed it as distributed cache (small file). Is there a way to get those values using get method. *while ((setupData = bf.readLine()) != null) {* * System.out.println("Setup Line "+setupData);* * TreeInfoWritable info = //something i can pass to TreeInfoWritable and get values* * DoubleWritable entropy = info.getEntropy();* * System.out.println("entropy: "+entropy);* *}* Tried to convert writable to gson format. *MR 1* *Gson gson = new Gson();* *String emitVal = gson.toJson(valEmit);* *context.write(out, new Text(emitVal));* But parsing canot be done in *MR2*. *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);* *Error: Type mismatch: cannot convert from String to TreeInfoWritable* Once it is changed to string we cannot get values. Am I able to get a workaround for the same. or to use just POJO classes instaed of Writable. I'm afraid if that becomes slower as we are depending on Java instaed of hadoop 's serializable classes --20cf301b65c3f748be050f947fca Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
= Am I able to get the values = from writable of a previous job.
ie I have 2 MR jobs
MR 1:
= =C2=A0I need to pass 3 element as values from reducer and the key is NullWr= itable. So I created a custom writable class to achieve this.
<= div class=3D"gmail_default">=C2=A0pub= lic class TreeInfoWritable implements Writable{

DoubleWritable entropy;<= /div>
IntWritable sum;
=
IntWritable clsCount;
=
..
}
MR 2:<= /div>
=C2=A0= I need to access MR 1 result in MR2 mapper setup function. And I accessed i= t as distributed cache (small file).
=C2=A0Is there a way to get those val= ues using get method.
=C2=A0while ((setupData =3D bf.readLine()) !=3D= null) {
System.out.= println("Setup Line "+setupData);
TreeInfoWritable info =3D //something i can pass to= TreeInfoWritable and get values
DoubleWritable entropy =3D info.getEntropy();
=
System.out.println("entropy: &q= uot;+entropy);
}

Tried to convert writable to gson forma= t.
MR 1
Gson gson =3D new Gson();
String emitVal =3D gs= on.toJson(valEmit);
context.write(out, new Text(emitVal));
=C2=A0But parsing canot be done in MR2.
TreeInfoWritable info= =3D gson.toJson(setupData, TreeInfoWritable.class);

E= rror: Type mismatch: cannot convert from String to TreeInfoWritable
On= ce it is changed to string we cannot get values.

Am I able to get a = workaround for the same. or to use just POJO classes instaed of Writable. I= 'm afraid if that becomes slower as we are depending on Java instaed of= hadoop 's serializable classes

<= div dir=3D"ltr">


=
--20cf301b65c3f748be050f947fca--