hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Divya R <avyakr...@gmail.com>
Subject Exception in mid of reading files.
Date Fri, 25 Oct 2013 11:53:28 GMT
Hi Guys,

   I'm indexing data (~50 -100GB per day) from hadoop. Hadoop is Running in
cluster mode (having 2 dataNaodes currently). After every two or three
hours I'm getting this exception. But both Data nodes are up and running.
Can any one please guide me as to what I should do or  If I'm doing wrong.

Code Snippet:
public InitHadoop()  {

        configuration = new Configuration();
        configuration.set("fs.default.name", "hdfs://<<namenode
IP>>:54310"); // Is this write to specify on namenode IP.?
        configuration.set("mapred.job.tracker", "hdfs://<<namenode
IP>>:54311");

        try {
            fileSystem = FileSystem.get(configuration);
        } catch (IOException e) {
            e.printStackTrace();
        }
}
private void indexDocument(FSDataInputStream file) {

            Scanner scanner = new Scanner(file);

            while (scanner.hasNext() != null) {
                  //   Indexing code
            }
      }
}

Logs:

2013-10-25 09:37:57 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:57 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:57 INFO  DFSClient:2432 - Could not obtain block
blk_-8795538519317154213_432897 from any node: java.io.IOException: No live
nodes contain current block. Will get new block locations from namenode and
retry...
2013-10-25 09:37:58 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:58 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:58 INFO  DFSClient:2432 - Could not obtain block
blk_-5974673190155585497_432671 from any node: java.io.IOException: No live
nodes contain current block. Will get new block locations from namenode and
retry...
2013-10-25 09:37:59 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:59 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:59 INFO  DFSClient:2432 - Could not obtain block
blk_-1662761320365439855_431653 from any node: java.io.IOException: No live
nodes contain current block. Will get new block locations from namenode and
retry...
2013-10-25 09:37:59 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:59 WARN  DFSClient:2266 - Failed to connect to
/<<IP>>:50010, add to deadNodes and continuejava.net.BindException: Cannot
assign requested address
2013-10-25 09:37:59 WARN  DFSClient:2400 - DFS Read: java.io.IOException:
Could not obtain block: blk_8826777676488299245_432528
file=/flume/<<File.Name>>.1382639351042
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2426)
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2218)
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2381)
    at java.io.DataInputStream.read(DataInputStream.java:149)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)

Regards,
-Divya

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message