flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishnanand Khambadkone <kkhambadk...@yahoo.com>
Subject Re: part files written to HDFS with .pending extension
Date Sat, 02 Sep 2017 00:46:05 GMT
 BTW, I am using a BucketingSink and a DateTimeBucketer.  Do I need to set any other property
to move the files from .pending state.
BucketingSink<String> sink = new BucketingSink<String>("hdfs://localhost:8020/flinktwitter/");sink.setBucketer(new
DateTimeBucketer<String>("yyyy-MM-dd--HHmm"));
    On Friday, September 1, 2017, 5:03:46 PM PDT, Krishnanand Khambadkone <kkhambadkone@yahoo.com>
wrote:  
 
 This message is eligible for Automatic Cleanup! (kkhambadkone@yahoo.com) Add cleanup rule
| More info
 Hi,  I have written a small program that uses a Twitter input stream and a HDFS output sink.
  When the files are written to HDFS each part file in the directory has a .pending extension.
 I am able to cat the file and see the tweet text.  Is this normal for the part files to
have .pending extension.

-rw-r--r--   3 user  supergroup      46399 2017-09-01 16:35 /flinktwitter/2017-09-01--1635/_part-0-95.pending

-rw-r--r--   3 user supergroup      54861 2017-09-01 16:35 /flinktwitter/2017-09-01--1635/_part-0-96.pending

-rw-r--r--   3 user supergroup      41878 2017-09-01 16:35 /flinktwitter/2017-09-01--1635/_part-0-97.pending

-rw-r--r--   3  user supergroup      42813 2017-09-01 16:35 /flinktwitter/2017-09-01--1635/_part-0-98.pending

-rw-r--r--   3  user supergroup      42887 2017-09-01 16:35 /flinktwitter/2017-09-01--1635/_part-0-99.pending


Mime
View raw message