hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wiley <kwi...@keithwiley.com>
Subject Re: Hadoop Binary File
Date Fri, 28 Jan 2011 01:28:10 GMT
On Jan 25, 2011, at 21:47 , F.Ozgur Catak wrote:

> Can you give me a simple example/source code for this project.
> 
> On Tue, Jan 25, 2011 at 10:13 PM, Keith Wiley <kwiley@keithwiley.com> wrote:
> 
>> I'm also doing binary image processing on Hadoop.  Where relevant, my Key
>> and Value types are a WritableComparable class of my own creation which
>> contains as members a BytesWritable object, obviously read from the file
>> itself directly into memory.  I also keep the path in my class so I know
>> where the file came from later.
>> 
>> On Jan 25, 2011, at 11:46 , F.Ozgur Catak wrote:
>> 
>>> Hi,
>>> 
>>> I'm trying to develop an image processing application with hadoop. All
>> image
>>> files are in HDFS.  But I don't know how to read this files with
>> binary/byte
>>> stream. What is correct decleration of Mapper<K,V,K,V> and
>> Reducer<K,V,K,V>
>>> Class.

Hmmm, "simple" you say...not even remotely.  Our system has grown into quite the behemoth.

Let's see...here's my Writable class that I use to pass images around Hadoop as keys and values:

public class FileWritable
implements WritableComparable<FileWritable> {
	
	private Path			filePath_ = null;
	private BytesWritable	fileContents_;

	public FileWritable() {
		set(null, new BytesWritable());
	}

	public FileWritable(Path filePath, BytesWritable fileContents) {
		set(filePath, fileContents);
	}

	public void set(Path filePath, BytesWritable fileContents) {
		filePath_ = filePath;
		fileContents_ = fileContents;
	}

	public Path getPath() {
		return filePath_;
	}

	/**
	 * The key is the filename, i.e., the last component of the full path
	 * @return
	 */
	public Text getKey() {
		return new Text(filePath_.getName());
	}

	public BytesWritable getContents() {
		return fileContents_;
	}
	
	public void write(DataOutput out) throws IOException {
		new Text(filePath_.getName()).write(out);
		fileContents_.write(out);
	}

	public void readFields(DataInput in) throws IOException {
		Text filePath = new Text();
		filePath.readFields(in);
		filePath_ = new Path(filePath.toString());
		
		fileContents_.readFields(in);
	}

	// If we ever use this class as a key, might want to do this a little better.
	@Override
	public int hashCode() {
		return fileContents_.hashCode();
	}

	@Override
	public boolean equals(Object o) {
		if (o instanceof FileWritable) {
			FileWritable f = (FileWritable) o;
			//Is the second half of this comparison *really* necessary?!
			return filePath_.equals(f.filePath_)
				&& fileContents_.equals(f.fileContents_);
		}
		return false;
	}

	public int compareTo(FileWritable f) {
		//Is the second half of this comparison *really* necessary?!
		int cmp = filePath_.compareTo(f.filePath_);
		if (cmp != 0)
			return cmp;
		return fileContents_.compareTo(f.fileContents_);
	}
}

Does that help?

________________________________________________________________________________
Keith Wiley               kwiley@keithwiley.com               www.keithwiley.com

"Luminous beings are we, not this crude matter."
  -- Yoda
________________________________________________________________________________




Mime
View raw message