Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jim.twensky@gmail.com
 designates 209.85.219.41 as permitted sender)
MIME-Version: 1.0
Date: Sun, 12 May 2013 13:24:48 -0500
Message-ID: 
 <CALUCP4FxhURXDzer6hOF6-FswjmMHEqgV0wo-5bmBgy7z3t_-g@mail.gmail.com>
Subject: Wrapping around BitSet with the Writable interface
From: Jim Twensky <jim.twensky@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0112d0daeb12bc04dc8983a4

--089e0112d0daeb12bc04dc8983a4
Content-Type: text/plain; charset=ISO-8859-1

I have large java.util.BitSet objects that I want to bitwise-OR using a
MapReduce job. I decided to wrap around each object using the Writable
interface. Right now I convert each BitSet to a byte array and serialize
the byte array on disk.

Converting them to byte arrays is a bit inefficient but I could not find a
work around to write them directly to the DataOutput. Is there a way to
skip this and serialize the object directly? Here is what my current
implementation looks like:

public class BitSetWritable implements Writable {

  private BitSet bs;

  public BitSetWritable() {
    this.bs = new BitSet();
  }

  @Override
  public void write(DataOutput out) throws IOException {

    ByteArrayOutputStream bos = new ByteArrayOutputStream(bs.size()/8);
    ObjectOutputStream oos = new ObjectOutputStream(bos);
    oos.writeObject(bs);
    byte[] bytes = bos.toByteArray();
    oos.close();
    out.writeInt(bytes.length);
    out.write(bytes);

  }

  @Override
  public void readFields(DataInput in) throws IOException {

    int len = in.readInt();
    byte[] bytes = new byte[len];
    in.readFully(bytes);

    ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
    ObjectInputStream ois = new ObjectInputStream(bis);
    try {
      bs = (BitSet) ois.readObject();
    } catch (ClassNotFoundException e) {
      throw new IOException(e);
    }

    ois.close();
  }

}

--089e0112d0daeb12bc04dc8983a4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I have large java.util.BitSet objects that I want to bitwi=
se-OR using a MapReduce job. I decided to wrap around each object using the=
 Writable interface. Right now I convert each BitSet to a byte array and se=
rialize the byte array on disk. <br>
<br>Converting them to byte arrays is a bit inefficient but I could not fin=
d a work around to write them directly to the DataOutput. Is there a way to=
 skip this and serialize the object directly? Here is what my current imple=
mentation looks like:<br>
<br>public class BitSetWritable implements Writable {<br>=A0 <br>=A0 privat=
e BitSet bs;<br>=A0 <br>=A0 public BitSetWritable() {<br>=A0=A0=A0 <a href=
=3D"http://this.bs">this.bs</a> =3D new BitSet();<br>=A0 }<br><br>=A0 @Over=
ride<br>=A0 public void write(DataOutput out) throws IOException {<br>
=A0=A0=A0 <br>=A0=A0=A0 ByteArrayOutputStream bos =3D new ByteArrayOutputSt=
ream(bs.size()/8);<br>=A0=A0=A0 ObjectOutputStream oos =3D new ObjectOutput=
Stream(bos);<br>=A0=A0=A0 oos.writeObject(bs); <br>=A0=A0=A0 byte[] bytes =
=3D bos.toByteArray();<br>=A0=A0=A0 oos.close();=A0=A0=A0 <br>
=A0=A0=A0 out.writeInt(bytes.length);<br>=A0=A0=A0 out.write(bytes);=A0=A0=
=A0 <br>=A0=A0=A0 <br>=A0 }<br><br>=A0 @Override<br>=A0 public void readFie=
lds(DataInput in) throws IOException {<br>=A0=A0=A0 <br>=A0=A0=A0 int len =
=3D in.readInt();<br>=A0=A0=A0 byte[] bytes =3D new byte[len];<br>
=A0=A0=A0 in.readFully(bytes);<br>=A0=A0=A0 <br>=A0=A0=A0 ByteArrayInputStr=
eam bis =3D new ByteArrayInputStream(bytes);<br>=A0=A0=A0 ObjectInputStream=
 ois =3D new ObjectInputStream(bis);<br>=A0=A0=A0 try {<br>=A0=A0=A0=A0=A0 =
bs =3D (BitSet) ois.readObject();<br>=A0=A0=A0 } catch (ClassNotFoundExcept=
ion e) {<br>
=A0=A0=A0=A0=A0 throw new IOException(e);<br>=A0=A0=A0 }<br>=A0=A0=A0 <br>=
=A0=A0=A0 ois.close();<br>=A0 }<br><br>}<br></div>

--089e0112d0daeb12bc04dc8983a4--