hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Ernster <jon.erns...@gmail.com>
Subject Re: Question appending data using WebHDFS
Date Fri, 15 Apr 2016 15:45:08 GMT
Your file will be split up across numerous data nodes assuming the
file you're storing is larger than the configured block size.
Therefore you'll have file / blocksize across X data nodes.  This is
all handled automatically so you don't really have the ability to say
"I want this entire file on this data node".

On Fri, Apr 15, 2016 at 6:49 AM, David Robison
<david.robison@psgglobal.net> wrote:
> I am trying to use WebHDFS to append a large amount of data to a file. From
> the documentation I see that I send the op=APPEND to the NameNode first and
> then it sends me the location of the DataNode. Then I can start sending data
> to the DataNode using the op=APPEND. My question is, can I continue to write
> data to the DataNode or do I need to periodically recheck with the NameNode
> to see if it wants to send the new data to another DataNode? In other words,
> once I have received the DataNode’s location can I continue to send as much
> data as I want to that DataNode or will I need at some point need to start
> sending data to another DataNode for the same file? I hope this is clear as
> I am new to HDFS. Thanks, David

To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org

View raw message