hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: response time increase??
Date Thu, 06 Jan 2011 16:01:35 GMT
Well, technically, 1st_arrival_time is O(number of network hops between
racks) which is O(#racks) and obviously #racks is a function of the number
of datanodes. But you're talking an extra 0.1ms or so there per network hop,
and there isn't any system which could avoid it.

But HDFS-wise, the data structures are mostly hashtable based.

-Todd

On Thu, Jan 6, 2011 at 2:01 AM, KevinKuei <kkuei@above.net.tw> wrote:

>  Dear Todd,
>
> Thanks for your reply.
> Let me clarify my "response time" definition.
>
> *start_time: time of request send
> *1st_arrival_time: the 1st block of data been received
> *completed_time: the final block of data been received
>
> response time = 1st_arrival_time = start_time
>
> Are you still confirmed that  there is NO  "response_time = O(number of
> datanode)" ?
> If so, HDFS will be great for our application.  Thanks!!
>
>
> --
> Kevin Kuei
>
>
> 於 2011/1/6 下午 04:23, Todd Lipcon 提到:
>
> Hi Kevin,
>
>  No, there is no O(number of datanodes) factor in performance.
>
>  -Todd
>
> On Wed, Jan 5, 2011 at 8:04 PM, KevinKuei <kkuei@above.net.tw> wrote:
>
>> Hi,
>>
>> I'm planing for a youtube-like video online site and looking for a
>> suitable file system.
>> The high performance and reliability of HDFS seems to be the great
>> candidate.
>>
>> But somebody told me that the response time will be linear increased with
>> more data node.
>> I've no enough hardwares to do the test for now.
>>
>> It will be very appreciated if anyone can provide such information.
>>
>> Thanks!!
>>
>> --
>> Kevin Kuei
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message