Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=FXVePvTVnxcjCfpdru47Y7j9gVmWPgGE3OdR45gaC292mcliOOWYuyWDFC923wsa
Message-Id: <D3B7A070-125B-4E84-9454-92F5FD6C72B7@yahoo-inc.com>
From: Hong Tang <htang@yahoo-inc.com>
To: common-user@hadoop.apache.org
In-Reply-To: <67C22614-089B-4547-9777-B8841E6B10E5@dataxu.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Efficiently Stream into Sequence Files?
Date: Fri, 12 Mar 2010 11:15:23 -0800
References: <67C22614-089B-4547-9777-B8841E6B10E5@dataxu.com>

Have you looked at TFile?

On Mar 12, 2010, at 5:22 AM, Scott Whitecross wrote:

> Hi -
>
> I'd like to create a job that pulls small files from a remote server  
> (using FTP, SCP, etc.) and stores them directly to sequence files on  
> HDFS.  Looking at the sequence file APi, I don't see an obvious way  
> to do this.  It looks like what I have to do is pull the remote file  
> to disk, then read the file into memory to place in the sequence  
> file.  Is there a better way?
>
> Looking at the API, am I forced to use the append method?
>
>            FileSystem hdfs =  
> FileSystem.get(context.getConfiguration());
>            FSDataOutputStream outputStream = hdfs.create(new  
> Path(outputPath));
>            writer =  
> SequenceFile.createWriter(context.getConfiguration(), outputStream,  
> Text.class, BytesWritable.class, null, null);
> 		
> 	   // read in file to remotefilebytes
>
>            writer.append(filekey, remotefilebytes);
>
>
> The alternative would be to have one job pull the remote files, and  
> a secondary job write them into sequence files.
>
> I'm using the latest Cloudera release, which I believe is Hadoop 20.1
>
> Thanks.
>
>
>
>