hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wiley <kwi...@keithwiley.com>
Subject Re: Hadoop Binary File
Date Fri, 28 Jan 2011 01:28:10 GMT
On Jan 25, 2011, at 21:47 , F.Ozgur Catak wrote:

> Can you give me a simple example/source code for this project.
> On Tue, Jan 25, 2011 at 10:13 PM, Keith Wiley <kwiley@keithwiley.com> wrote:
>> I'm also doing binary image processing on Hadoop.  Where relevant, my Key
>> and Value types are a WritableComparable class of my own creation which
>> contains as members a BytesWritable object, obviously read from the file
>> itself directly into memory.  I also keep the path in my class so I know
>> where the file came from later.
>> On Jan 25, 2011, at 11:46 , F.Ozgur Catak wrote:
>>> Hi,
>>> I'm trying to develop an image processing application with hadoop. All
>> image
>>> files are in HDFS.  But I don't know how to read this files with
>> binary/byte
>>> stream. What is correct decleration of Mapper<K,V,K,V> and
>> Reducer<K,V,K,V>
>>> Class.

Hmmm, "simple" you say...not even remotely.  Our system has grown into quite the behemoth.

Let's see...here's my Writable class that I use to pass images around Hadoop as keys and values:

public class FileWritable
implements WritableComparable<FileWritable> {
	private Path			filePath_ = null;
	private BytesWritable	fileContents_;

	public FileWritable() {
		set(null, new BytesWritable());

	public FileWritable(Path filePath, BytesWritable fileContents) {
		set(filePath, fileContents);

	public void set(Path filePath, BytesWritable fileContents) {
		filePath_ = filePath;
		fileContents_ = fileContents;

	public Path getPath() {
		return filePath_;

	 * The key is the filename, i.e., the last component of the full path
	 * @return
	public Text getKey() {
		return new Text(filePath_.getName());

	public BytesWritable getContents() {
		return fileContents_;
	public void write(DataOutput out) throws IOException {
		new Text(filePath_.getName()).write(out);

	public void readFields(DataInput in) throws IOException {
		Text filePath = new Text();
		filePath_ = new Path(filePath.toString());

	// If we ever use this class as a key, might want to do this a little better.
	public int hashCode() {
		return fileContents_.hashCode();

	public boolean equals(Object o) {
		if (o instanceof FileWritable) {
			FileWritable f = (FileWritable) o;
			//Is the second half of this comparison *really* necessary?!
			return filePath_.equals(f.filePath_)
				&& fileContents_.equals(f.fileContents_);
		return false;

	public int compareTo(FileWritable f) {
		//Is the second half of this comparison *really* necessary?!
		int cmp = filePath_.compareTo(f.filePath_);
		if (cmp != 0)
			return cmp;
		return fileContents_.compareTo(f.fileContents_);

Does that help?

Keith Wiley               kwiley@keithwiley.com               www.keithwiley.com

"Luminous beings are we, not this crude matter."
  -- Yoda

View raw message