hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba Borthakur" <dhr...@yahoo-inc.com>
Subject RE: HDFS File Read
Date Mon, 19 Nov 2007 07:02:57 GMT
One simple way to do it is to make the writer write to a temporary file.
When all data to the temporary file has been written and the temporary
file is closed, rename to the temporary file to the real file name.

Thanks,
dhruba

-----Original Message-----
From: j2eeiscool [mailto:siddiqut@yahoo.com] 
Sent: Friday, November 16, 2007 6:32 PM
To: hadoop-user@lucene.apache.org
Subject: RE: HDFS File Read


Hi Dhruba,

For my test I do have a Reader and Writer thread. The Reader blocks till
the
InputStream is available:

The Reader gets the following exception till the Writer is done :

org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot open
filename /hadoopdata0.txt
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:269)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

        at org.apache.hadoop.ipc.Client.call(Client.java:470)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
        at org.apache.hadoop.dfs.$Proxy0.open(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
cationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
Handler.java:59)
        at org.apache.hadoop.dfs.$Proxy0.open(Unknown Source)
        at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:8
64)
        at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.<init>(DFSClient.java:856
)
        at org.apache.hadoop.dfs.DFSClient.open(DFSClient.java:277)
        at
org.apache.hadoop.dfs.DistributedFileSystem.open(DistributedFileSystem.j
ava:122)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:244)
        at HadoopDSMStore.select(HadoopDSMStore.java:44)
        at HadoopDSMStore$ReaderThread.run(HadoopDSMStore.java:174)
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot open
filename /hadoopdata0.txt
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:269)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

        at HadoopDSMStore.select(HadoopDSMStore.java:44)
        at HadoopDSMStore$ReaderThread.run(HadoopDSMStore.java:174)


1.Is there an api (like isFileAvailable(fileName)) the Reader needs to
check
before starting ?

2.Should there be a delay between Writer end and Reader start ?

Thanx,
Taj



Dhruba Borthakur wrote:
> 
> This could happen if one of your threads was reading a file when
another
> thread deleted the file and created another new file with the same
name.
> The first reader wants to fetch more blocks for the file but detects
> that the file has a different blocklist.
> 
> One option for you is to re-open the file when you get this error.
> 
> Thanks,
> dhruba
> 
> -----Original Message-----
> From: j2eeiscool [mailto:siddiqut@yahoo.com] 
> Sent: Friday, November 16, 2007 1:39 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: HDFS File Read
> 
> 
> Thanx for your reply Ted,
> 
> I get this in the middle of a file read (towards the end actually). 
> No change to the cluster config during this operation.
> 
> Programatically what would be the best way to recover from this :
> 
> Open the inputstream again and seek to the failure position ?
> 
> Thanx,
> Taj
> 
> 
> 
> Ted Dunning-3 wrote:
>> 
>> 
>> Run hadoop fsck /
>> 
>> It sounds like you have some blocks that have been lost somehow.
This
> is
>> pretty easy to do as you reconfigure a new cluster.
>> 
>> 
>> On 11/16/07 12:21 PM, "j2eeiscool" <siddiqut@yahoo.com> wrote:
>> 
>>> 
>>> Raghu/Ted,
>>> 
>>> This turned out to be a sub-optimal network pipe between client and
>>> data-node.
>>> 
>>> Now the average read time is around 35 secs (for 68 megs ).
>>> 
>>> On to the next issue:
>>> 
>>> 07/11/16 20:05:37 WARN fs.DFSClient: DFS Read: java.io.IOException:
>>> Blocklist for /hadoopdata0.txt has changed!
>>>         at
>>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:8
> 71)
>>>         at
>>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.
> java:1
>>> 161)
>>>         at
>>> 
>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.jav
> a:1004>
>> )
>>>         at
>>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1107)
>>>         at java.io.DataInputStream.read(DataInputStream.java:80)
>>>         at HadoopDSMStore$ReaderThread.run(HadoopDSMStore.java:187)
>>> 
>>> java.io.IOException: Blocklist for /hadoopdata0.txt has changed!
>>>         at
>>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:8
> 71)
>>>         at
>>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.
> java:1
>>> 161)
>>>         at
>>> 
>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.jav
> a:1004>
>> )
>>>         at
>>>
>
org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1107)
>>>         at java.io.DataInputStream.read(DataInputStream.java:80)
>>>         at HadoopDSMStore$ReaderThread.run(HadoopDSMStore.java:187)
>>> 07/11/16 20:05:37 INFO fs.DFSClient: Could not obtain block
>>> blk_1990972671947672118 from any node:  java.io.IOException: No live
>>> nodes
>>> contain current block
>>> 07/11/16 20:05:40 INFO fs.DFSClient: Could not obtain block
>>> blk_1990972671947672118 from any node:  java.io.IOException: No live
>>> nodes
>>> contain current block
>>> 
>>> 
>>> This happens during the read.
>>> 
>>> I get this error from time to time and specially when i run the
> client in
>>> multithreaded mode.
>>> 
>>> Could this be an instability on the dataNode side ?
>>> 
>>> Thanx much,
>>> Taj
>>> 
>>> 
>>> 
>>> Raghu Angadi wrote:
>>>> 
>>>> To simplify, read rate should be faster than write speed.
>>>> 
>>>> Raghu.
>>>> 
>>>> Raghu Angadi wrote:
>>>>> 
>>>>> Normally, Hadoop read saturates either disk b/w or network b/w on
>>>>> moderate hardware. So if you have one modern IDE disk and 100mbps
>>>>> ethernet, you should expect around 10MBps read rate for a simple
> read
>>>>> from client on different machine.
>>>>> 
>>>>> Raghu.
>>>>> 
>>>>> j2eeiscool wrote:
>>>>>> Hi Raghu,
>>>>>> 
>>>>>> Just to give me something to compare with: how long should this
> file
>>>>>> read
>>>>>> (68 megs) take on a good set-up
>>>>>> 
>>>>>> (client and data node on same network, one hop).
>>>>>> 
>>>>>> Thanx for your help,
>>>>>> Taj
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Raghu Angadi wrote:
>>>>>>> Taj,
>>>>>>> 
>>>>>>> Even 4 times faster (400 sec for 68MB) is not very fast. First
> try to
>>>>>>> scp a similar sized file between the hosts involved. If this
> transfer
>>>>>>> is slow, first fix this issue. Try to place the test file on
the
> same
>>>>>>> partition where HDFS data is stored.
>>>>>>> 
>>>>>>> With tcpdump, first make sure amount of data transfered matches
>>>>>>> around 68MB that you expect.. and check for any large gaps in
> data
>>>>>>> packets comming to the client. Also when the client is reading,
> check
>>>>>>> netstat on both client and the datanode.. note the send buffer
on
>>>>>>> datanode and recv buffer on the client. If datanodes send buffer
> is
>>>>>>> non-zero most of the time, then you have some network issue,
if
> recv
>>>>>>> buffer on client is full, then client is reading slow for some
>>>>>>> reason... etc.
>>>>>>> 
>>>>>>> hope this helps.
>>>>>>> 
>>>>>>> Raghu.
>>>>>>> 
>>>>>>> j2eeiscool wrote:
>>>>>>>> Hi Raghu,
>>>>>>>> 
>>>>>>>> Good catch, thanx. totalBytesRead  is not used for any decision
> etc.
>>>>>>>> 
>>>>>>>> I ran the client from another m/c and read was about 4 times
> faster.
>>>>>>>> I have the tcpdump from the original client m/c.
>>>>>>>> This is probably asking too much but anything in particular
I
> should
>>>>>>>> be
>>>>>>>> looking in the tcpdump.
>>>>>>>> 
>>>>>>>> Is (tcpdump) about 16 megs in size.
>>>>>>>> 
>>>>>>>> Thanx,
>>>>>>>> Taj
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Raghu Angadi wrote:
>>>>>>>>> Thats too long.. buffer size does not explain it. Only
small
>>>>>>>>> problem I see in your code:
>>>>>>>>> 
>>>>>>>>>> totalBytesRead += bytesReadThisRead;
>>>>>>>>>> fileNotReadFully = (bytesReadThisRead != -1);
>>>>>>>>> 
>>>>>>>>> totalBytesRead is off by 1. Not sure where totalBytesRead
is
> used.
>>>>>>>>> 
>>>>>>>>> If you can, try to check tcpdump on your client machine
(for
>>>>>>>>> datanode port 50010)
>>>>>>>>> 
>>>>>>>>> Raghu.
>>>>>>>>> 
>>>>>>>>> j2eeiscool wrote:
>>>>>>>>>> Hi Raghu,
>>>>>>>>>> 
>>>>>>>>>> Many thanx for your reply:
>>>>>>>>>> 
>>>>>>>>>> The write takes approximately:  11367 millisecs.
>>>>>>>>>> 
>>>>>>>>>> The read takes approximately: 1610565 millisecs.
>>>>>>>>>> 
>>>>>>>>>> File size is  68573254 bytes and hdfs block size
is 64 megs.
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 
>> 
> 
> -- 
> View this message in context:
> http://www.nabble.com/HDFS-File-Read-tf4773580.html#a13802096
> Sent from the Hadoop Users mailing list archive at Nabble.com.
> 
> 
> 

-- 
View this message in context:
http://www.nabble.com/HDFS-File-Read-tf4773580.html#a13805430
Sent from the Hadoop Users mailing list archive at Nabble.com.


Mime
View raw message