Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2B3B819DBA for ; Sat, 30 Apr 2016 04:18:58 +0000 (UTC) Received: (qmail 96554 invoked by uid 500); 30 Apr 2016 04:18:54 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 96431 invoked by uid 500); 30 Apr 2016 04:18:53 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 96421 invoked by uid 99); 30 Apr 2016 04:18:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Apr 2016 04:18:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4BED11804D6 for ; Sat, 30 Apr 2016 04:18:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.798 X-Spam-Level: ** X-Spam-Status: No, score=2.798 tagged_above=-999 required=6.31 tests=[FSL_HELO_BARE_IP_2=1.499, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id ge7I4aGu6det for ; Sat, 30 Apr 2016 04:18:49 +0000 (UTC) Received: from relayvx11b.securemail.intermedia.net (relayvx11b.securemail.intermedia.net [64.78.52.184]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 987655F247 for ; Sat, 30 Apr 2016 04:18:48 +0000 (UTC) Received: from securemail.intermedia.net (localhost [127.0.0.1]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-1.localdomain (Postfix) with ESMTPS id 4B6DC53F17; Fri, 29 Apr 2016 21:18:41 -0700 (PDT) Subject: Re: [HDFS-inotify] "IOException: The client is stopped" after reading file MIME-Version: 1.0 x-echoworx-msg-id: 9c7fc890-31d7-42a0-bc61-fcec57f36842 x-echoworx-emg-received: Fri, 29 Apr 2016 21:18:41.248 -0700 x-echoworx-message-code-hashed: 51e78049cb3709d0d828ac34bde9acabd14efe27c2bf2fa625d560d6e2553f8d x-echoworx-action: delivered Received: from 10.254.155.14 ([10.254.155.14]) by emg-ca-1-1 (JAMES SMTP Server 2.3.2) with SMTP ID 790; Fri, 29 Apr 2016 21:18:41 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (unknown [10.224.117.102]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-1.localdomain (Postfix) with ESMTPS id 0328553F17; Fri, 29 Apr 2016 21:18:41 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) with Microsoft SMTP Server (TLS) id 15.0.1130.7; Fri, 29 Apr 2016 21:18:40 -0700 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1130.005; Fri, 29 Apr 2016 21:18:40 -0700 From: Chris Nauroth To: Cazen Lee , "user@hadoop.apache.org" Thread-Topic: [HDFS-inotify] "IOException: The client is stopped" after reading file Thread-Index: AQHRobG0wbiDy5ycvE2V6N7U7M5RSJ+h61iA Date: Sat, 30 Apr 2016 04:18:39 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [73.254.66.69] x-source-routing-agent: Processed Content-Type: multipart/alternative; boundary="_000_D3497D0541D61cnaurothhortonworkscom_" --_000_D3497D0541D61cnaurothhortonworkscom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello Cazen, This looks to me like this is suffering from an unintended side effect of c= losing the FileSystem object. Hadoop internally caches instances of the Fi= leSystem class, and the same instance can be returned to multiple call site= s. Even after one call site closes it, it's possible that other call sites= still hold a reference to that same FileSystem instance. Closing the File= System instance makes it unusable. HdfsAdmin#getInotifyEventStream is likely using the same FileSystem instanc= e that your own FileSystem.get call returns. By closing it (using try-with= -resources), that FileSystem instance is made invalid for the subsequent ca= lls to retrieve inotify events. The FileSystem cache is a fairly common source of confusion. However, its = current behavior is considered by design. For reasons of backwards-incompa= tibility, we can't easily change its behavior to help with confusing situat= ions like this. (Sorry!) A few suggestions to try: 1. Just don't close the FileSystem. Even if you don't close it explicitly,= it will be closed at process teardown via a shutdown hook. This definitel= y looks wrong from a resource management perspective, but a lot of applicat= ions work this way. 2. Call FileSystem#newInstance instead of FileSystem#get. The newInstance = method is guaranteed to return an instance unique to that call site, not a = shared instance potentially in use by other call sites. If you use newInst= ance, then you must guarantee it gets closed to avoid a leak with a long-te= rm impact. 3. You can disable the FileSystem cache for specific file system types by e= diting core-site.xml and setting property fs..impl.disabl= e.cache to true, e.g. fs.hdfs.impl.disable.cache. In general, disabling th= e cache is not desirable, because the performance benefits of the cache are= noticeable. Sometimes this is a helpful workaround for specific applicati= ons though. --Chris Nauroth From: Cazen Lee > Date: Thursday, April 28, 2016 at 5:53 PM To: "user@hadoop.apache.org" > Subject: [HDFS-inotify] "IOException: The client is stopped" after reading = file Good day this is Cazen Could I kindly ask about something weird situation when reading file in hdf= s with inotify polling - Env : MacOS, EMR, Linux(standalone) - same problem - Version : Hadoop 2.7.2 1. I would like to write down a code that read file under particular locati= on when it created(with using inotify) So I modify sample code based on "hdfs-inotify-example" in github https://github.com/onefoursix/hdfs-inotify-example/blob/master/src/main/jav= a/com/onefoursix/HdfsINotifyExample.java 2. I've changed code with read and print line to console when it renamed https://github.com/onefoursix/hdfs-inotify-example/commit/82485881c5da85a46= dd1741c2d8420c7c4e81f93 case RENAME: Event.RenameEvent renameEvent =3D (Event.RenameEvent) event; Configuration conf =3D new Configuration(); conf.set("fs.defaultFS", defaultFS); System.out.println(renameEvent.getDstPath() + " " + inputPath.getPath()= ); if (renameEvent.getDstPath().startsWith(inputPath.getPath())) { //Try to read file try (FileSystem fs =3D FileSystem.get(conf)) { Path filePath =3D new Path(defaultFS + renameEvent.getDstPath()= ); BufferedReader br =3D new BufferedReader(new InputStreamReader(= fs.open(filePath))); String line; line =3D br.readLine(); while (line !=3D null) { System.out.println(line); line =3D br.readLine(); } br.close(); } } 3. It works. But I encountered IOException in next eventStream.take() after= file read. It doesn't happen if I do not read file on hdfs. -------------CODE------------- DFSInotifyEventInputStream eventStream =3D admin.getInotifyEventStream(); EventBatch batch =3D eventStream.take(); -------------LOG------------- Cazens-MacBook-Pro:hdfs-inotify-example Cazen$ java -jar target/hdfs-inotif= y-example-uber.jar hdfs://localhost:8032/cazen/ lastReadTxid =3D 0 log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.S= hell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for mo= re info. TxId =3D 3134 event type =3D CREATE path =3D /cazen/test2.txt._COPYING_ owner =3D Cazen ctime =3D 1461850245559 TxId =3D 3138 event type =3D CLOSE TxId =3D 3139 event type =3D RENAME /cazen/test2.txt /cazen/ --------------------File Start Input File Text Sample LOL --------------------File END Exception in thread "main" java.io.IOException: The client is stopped at org.apache.hadoop.ipc.Client.getConnection(Client.java:1507) at org.apache.hadoop.ipc.Client.call(Client.java:1451) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine= .java:229) at com.sun.proxy.$Proxy9.getEditsFromTxid(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.get= EditsFromTxid(ClientNamenodeProtocolTranslatorPB.java:1511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav= a:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor= Impl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo= cationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation= Handler.java:102) at com.sun.proxy.$Proxy10.getEditsFromTxid(Unknown Source) at org.apache.hadoop.hdfs.DFSInotifyEventInputStream.poll(DFSInotifyEventIn= putStream.java:111) at org.apache.hadoop.hdfs.DFSInotifyEventInputStream.take(DFSInotifyEventIn= putStream.java:224) at com.onefoursix.HdfsINotifyExample.main(HdfsINotifyExample.java:40) There is possibility that I may have written the wrong code. If anyone alre= ady know about this situation, could I ask the reason? Any advice would be appreciated. Thank you Have a good day :) -- Cazen.lee@gmail.com Cazen.lee@samsung.com http://www.cazen.co.kr --_000_D3497D0541D61cnaurothhortonworkscom_ Content-Type: text/html; charset="us-ascii" Content-ID: <5D31BE5EEE8A8045855543BE1B099989@exch080.serverpod.net> Content-Transfer-Encoding: quoted-printable
Hello Cazen,

This looks to me like this is suffering from an unintended side effect= of closing the FileSystem object.  Hadoop internally caches instances= of the FileSystem class, and the same instance can be returned to multiple= call sites.  Even after one call site closes it, it's possible that other call sites still hold a reference to t= hat same FileSystem instance.  Closing the FileSystem instance makes i= t unusable.

HdfsAdmin#getInotifyEventStream is likely using the same FileSystem in= stance that your own FileSystem.get call returns.  By closing it (usin= g try-with-resources), that FileSystem instance is made invalid for the sub= sequent calls to retrieve inotify events.

The FileSystem cache is a fairly common source of confusion.  How= ever, its current behavior is considered by design.  For reasons of ba= ckwards-incompatibility, we can't easily change its behavior to help with c= onfusing situations like this.  (Sorry!)

A few suggestions to try:

1. Just don't close the FileSystem.  Even if you don't close it e= xplicitly, it will be closed at process teardown via a shutdown hook.  = ;This definitely looks wrong from a resource management perspective, but a = lot of applications work this way.

2. Call FileSystem#newInstance instead of FileSystem#get.  The ne= wInstance method is guaranteed to return an instance unique to that call si= te, not a shared instance potentially in use by other call sites.  If = you use newInstance, then you must guarantee it gets closed to avoid a leak with a long-term impact.

3. You can disable the FileSystem cache for specific file system types= by editing core-site.xml and setting property fs.<file system type>.= impl.disable.cache to true, e.g. fs.hdfs.impl.disable.cache.  In gener= al, disabling the cache is not desirable, because the performance benefits of the cache are noticeable.  Sometimes this= is a helpful workaround for specific applications though.

--Chris Nauroth

From: Cazen Lee <cazen.lee@gmail.com>
Date: Thursday, April 28, 2016 at 5= :53 PM
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: [HDFS-inotify] "IOExc= eption: The client is stopped" after reading file

Good day this is Cazen
Could I kindly ask about something weird situation when reading file in hdf= s with inotify polling

- Env : MacOS, EMR, Linux(standalone) - same problem
- Version : Hadoop 2.7.2

1. I would like to write down a code that read file under pa= rticular location when it created(with using inotify)
    So I modify sample code based on "hdfs-inotify-exam= ple" in github
https://github.com/one= foursix/hdfs-inotify-example/blob/master/src/main/java/com/onefoursix/HdfsI= NotifyExample.java

2. I've changed code with read and print line to console whe= n it renamed
https://github.com/onefoursix/hdfs-inot= ify-example/commit/82485881c5da85a46dd1741c2d8420c7c4e81f93

case RENAME:
    Event.RenameEvent renameEvent =3D (Event.RenameEvent) ev= ent;
    Configuration conf =3D new Configuration();
    conf.set("fs.defaultFS", defaultFS);
    System.out.println(renameEvent.getDstPath() + "= " + inputPath.getPath());
    if (renameEvent.getDstPath().startsWith(inputPath.getPat= h())) {
        //Try to read file
        try (FileSystem fs =3D FileSyste= m.get(conf)) {
            Path fil= ePath =3D new Path(defaultFS + renameEvent.getDstPath());
            Buffered= Reader br =3D new BufferedReader(new InputStreamReader(fs.open(filePath)));=
            String l= ine;
            line =3D= br.readLine();
            while (l= ine !=3D null) {
            &nb= sp;   System.out.println(line);
            &nb= sp;   line =3D br.readLine();
            }
            br.close= ();
        }
    }

3. It works. But I encountered IOException in next eventStre= am.take() after file read. It doesn't happen if I do not read file on hdfs.
-------------CODE-------------
DFSInotifyEventInputStream eventStream =3D admin.getInotifyEventStream(); EventBatch batch =3D eventStream.take();

-------------LOG-------------
Cazens-MacBook-Pro:hdfs-inotify-example Cazen$ java -jar target/hdfs-inotif= y-example-uber.jar hdfs://localhost:8032/cazen/
lastReadTxid =3D 0
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.S= hell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more in= fo.
TxId =3D 3134
event type =3D CREATE
  path =3D /cazen/test2.txt._COPYING_
  owner =3D Cazen
  ctime =3D 1461850245559
TxId =3D 3138
event type =3D CLOSE
TxId =3D 3139
event type =3D RENAME
/cazen/test2.txt /cazen/
--------------------File Start
Input File Text Sample LOL
--------------------File END
Exception in thread "main" java.io.IOException: The client is sto= pped
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1507)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine= .java:229)
at com.sun.proxy.$Proxy9.getEditsFromTxid(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.get= EditsFromTxid(ClientNamenodeProtocolTranslatorPB.java:1511)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav= a:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor= Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo= cationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation= Handler.java:102)
at com.sun.proxy.$Proxy10.getEditsFromTxid(Unknown Source)
at org.apache.hadoop.hdfs.DFSInotifyEventInputStream.poll(DFSInotifyEventIn= putStream.java:111)
at org.apache.hadoop.hdfs.DFSInotifyEventInputStream.take(DFSInotifyEventIn= putStream.java:224)
at com.onefoursix.HdfsINotifyExample.main(HdfsINotifyExample.java:40)

There is possibility that I may have written the wrong code.= If anyone already know about this situation, could I ask the reason?
Any advice would be appreciated.
Thank you Have a good day :)

--
Cazen.lee@gmail.com
Cazen.lee@samsung.com
http://www.cazen.co.kr

 --_000_D3497D0541D61cnaurothhortonworkscom_--