hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jagaran das <jagaran_...@yahoo.co.in>
Subject Re: java.io.IOException: config()
Date Sat, 06 Aug 2011 07:16:18 GMT
I am accessing through threads in parallel.

What is the concept of Lease in HDFS??

Regards,
JD


________________________________
From: Harsh J <harsh@cloudera.com>
To: jagaran das <jagaran_das@yahoo.co.in>
Sent: Friday, 5 August 2011 11:37 PM
Subject: Re: java.io.IOException: config()


How long are you keeping it open for?


On 06-Aug-2011, at 10:14 AM, jagaran das wrote:

Hi,
>
>
>I am using CDH3.
>I need to stream huge amount of data from our application to hadoop.
>I am opening a connection like
> 
>config.set("fs.default.name",hdfsURI);
>FileSystem dfs = FileSystem.get(config);
>String path = hdfsURI + connectionKey;
>Path destPath = new Path(path);
>logger.debug("Path -- " + destPath.getName());
>outStream = dfs.create(destPath);
>and keeping the outStream open for some time and writing continuously through it and
then closing it.
>But it is throwing 
>
>
> 
>5Aug2011 21:36:48,550 DEBUG [LeaseChecker@DFSClient[clientName=DFSClient_218151655, ugi=jagarandas]:
java.lang.Throwable: for testing
>at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.toString(DFSClient.java:1181)
>at org.apache.hadoop.util.Daemon.<init>(Daemon.java:38)
>at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.put(DFSClient.java:1094)
>at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:547)
>at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:219)
>at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:584)
>at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:565)
>at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:472)
>at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:464)
>at com.apple.ireporter.common.persistence.ConnectionManager.createConnection(ConnectionManager.java:66)
>at com.apple.ireporter.common.persistence.HDPPersistor.writeToHDP(HDPPersistor.java:93)
>at com.apple.ireporter.datatransformer.translator.HDFSTranslator.persistData(HDFSTranslator.java:41)
>at com.apple.ireporter.datatransformer.adapter.TranslatorAdapter.processData(TranslatorAdapter.java:61)
>at com.apple.ireporter.datatransformer.DefaultMessageListener.persistValidatedData(DefaultMessageListener.java:276)
>at com.apple.ireporter.datatransformer.DefaultMessageListener.onMessage(DefaultMessageListener.java:93)
>at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:506)
>at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:463)
>at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
>at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
>at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:260)
>at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
>at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
>at java.lang.Thread.run(Thread.java:680)
>] (RPC.java:230) - Call: renewLease 4
>05Aug2011 21:36:48,550 DEBUG [listenerContainer-1] (DFSClient.java:3274) - DFSClient writeChunk
allocating new packet seqno=0, src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819,
packetSize=65557, chunksPerPacket=127, bytesCurBlock=0
>05Aug2011 21:36:48,551 DEBUG [Thread-11] (DFSClient.java:2499) - Allocating new block
>05Aug2011 21:36:48,552 DEBUG [sendParams-0] (Client.java:761) - IPC Client (47) connection
to localhost/127.0.0.1:8020 from jagarandas sending #3
>05Aug2011 21:36:48,553 DEBUG [IPC Client (47) connection to localhost/127.0.0.1:8020 from
jagarandas] (Client.java:815) - IPC Client (47) connection to localhost/127.0.0.1:8020 from
jagarandas got value #3
>05Aug2011 21:36:48,556 DEBUG [Thread-11] (RPC.java:230) - Call: addBlock 4
>05Aug2011 21:36:48,557 DEBUG [Thread-11] (DFSClient.java:3094) - pipeline = 127.0.0.1:50010
>05Aug2011 21:36:48,557 DEBUG [Thread-11] (DFSClient.java:3102) - Connecting to 127.0.0.1:50010
>05Aug2011 21:36:48,559 DEBUG [Thread-11] (DFSClient.java:3109) - Send buf size 131072
>05Aug2011 21:36:48,635 DEBUG [DataStreamer for file /home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819
block blk_-5183404460805094255_1042] (DFSClient.java:2533) - DataStreamer block blk_-5183404460805094255_1042
wrote packet seqno:0 size:1522 offsetInBlock:0 lastPacketInBlock:true
>05Aug2011 21:36:48,638 DEBUG [ResponseProcessor for block blk_-5183404460805094255_1042]
(DFSClient.java:2640) - DFSClient Replies for seqno 0 are SUCCESS
>05Aug2011 21:36:48,639 DEBUG [DataStreamer for file /home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819
block blk_-5183404460805094255_1042] (DFSClient.java:2563) - Closing old block blk_-5183404460805094255_1042
>05Aug2011 21:36:48,645 DEBUG [sendParams-0] (Client.java:761) - IPC Client (47) connection
to localhost/127.0.0.1:8020 from jagarandas sending #4
>05Aug2011 21:36:48,647 DEBUG [IPC Client (47) connection to localhost/127.0.0.1:8020 from
jagarandas] (Client.java:815) - IPC Client (47) connection to localhost/127.0.0.1:8020 from
jagarandas got value #4
>05Aug2011 21:36:48,648 DEBUG [listenerContainer-1] (RPC.java:230) - Call: complete 3
>
>
>Please help as it a production enhancement for us.
>
>
>Regards
>Jagaran 
>
>
>
>________________________________
>From: Harsh J <harsh@cloudera.com>
>To: user@pig.apache.org; jagaran das <jagaran_das@yahoo.co.in>
>Sent: Friday, 5 August 2011 8:54 PM
>Subject: Re: java.io.IOException: config()
>
>Could you explain how/where you're stuck?
>
>That DEBUG log doesn't even seem like a valid throw; its just to get a
>strace I believe.
>
>On Sat, Aug 6, 2011 at 8:52 AM, jagaran das <jagaran_das@yahoo.co.in> wrote:
>> Hi,
>>
>> I have been struck with this exception:
>>
>> java.io.IOException: config()
>> at org.apache.hadoop.conf.Configuration.(Configuration.java:211)
>> at
 org.apache.hadoop.conf.Configuration.(Configuration.java:198)
>> at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:99)
>> at test.TestApp.main(TestApp.java:19)
>>
>>
>>
>> 05Aug2011 20:08:53,303 DEBUG [LeaseChecker@DFSClient[clientName=DFSClient_-1591195062,
ugi=jagarandas,staff,com.apple.sharepoint.group.1,_developer,_lpoperator,_lpadmin,_appserveradm,admin,_appserverusr,localaccounts,everyone,fmsadmin,com.apple.access_screensharing,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3]:
java.lang.Throwable: for testing
>>
>>
>> 05Aug2011 20:08:53,315 DEBUG [listenerContainer-1] (DFSClient.java:3012) - DFSClient
writeChunk allocating new packet seqno=0, src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_222812011-08-05-20-08-52,
packetSize=65557, chunksPerPacket=127, bytesCurBlock=0
>>
>> I saw the source code :
>>
>>  public
 Configuration(boolean loadDefaults) {
>>     this.loadDefaults = loadDefaults;
>>     if (LOG.isDebugEnabled()) {
>>       LOG.debug(StringUtils.stringifyException(new IOException("config()")));
>>     }
>>     synchronized(Configuration.class) {
>>       REGISTRY.put(this, null);
>>     }
>>   }
>>
>> Log is in debug mode.
>>
>> Can anyone please help me on this??
>>
>> Regards,
>> JD
>
>
>
>-- 
>Harsh J
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message