flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Natkins <na...@streamsets.com>
Subject Re: "Hit max consecutive under-replication rotations" Error
Date Tue, 15 Jul 2014 22:35:47 GMT
Hi Sanjay,

Is this just a single-node test cluster? Playing with replication configs
is probably a little bit dangerous, since it means that your blocks will
have no replicas, and if you lose a disk, you're going to end up with no
way to recover the blocks. If this is a cluster you actually care about,
I'd strongly recommend setting your dfs.replication to 3, and looking at
HDFS to determine why your blocks are not getting replicated in the first
place.

My guess is that what is happening is that your Flume agent is a mismatch
in configs. The HDFS sink has a parameter called minBlockReplicas, which
informs it as to how many block replicas are necessary to have, and if not
specified, it pulls that parameter from the default HDFS configuration
file. My guess is that, somehow, it's getting a different value for the
dfs.replication or for dfs.namenode.replication.min.

You can probably circumvent this error by modifying your Flume config with
this:

a1.sinks.k1.hdfs.minBlockReplicas = 1

Thanks,
Natty


On Tue, Jul 15, 2014 at 3:17 PM, Sanjay Ramanathan <
sanjay.ramanathan@lucidworks.com> wrote:

>  Hi all,
>
>
>  While trying to input data from flume to HDFS sink, I'm getting this
> error
>
>
>  "
>  [ERROR -
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:566)] Hit
> max consecutive under-replication rotations (30); will not continue rolling
> files under this path due to under-replication
>
> "
>
>
>
>  I looked up the error online and it said to make the below
> modification(dfs.replication). I did that and the problem still persists.
>
> My hadoop configuration hdfs-site.xml has the property
>  "
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> "
>
>  I also get this message 30 times before the above error message:
>  "Block Under-replication detected. Rotating file."
>
>  My flume conf file has the configuration:
>  a1.sinks.k1.channel = c1
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = hdfs://xx.xx.xx.xx:8020/input1/event/%y-%m-%d/%H%M
> a1.sinks.k1.hdfs.useLocalTimeStamp = true
> a1.sinks.k1.hdfs.round = true
> a1.sinks.k1.hdfs.roundValue = 10
> a1.sinks.k1.hdfs.roundUnit = minute
> a1.sinks.k1.hdfs.writeFormat = Text
> a1.sinks.k1.hdfs.fileType = DataStream
> #a1.sinks.k1.hdfs.filePrefix = events-
> a1.sinks.k1.hdfs.rollCount = 1000
> a1.sinks.k1.hdfs.batchSize = 10000
> a1.sinks.k1.hdfs.rollSize = 0
>  a1.sinks.k1.hdfs.rollInterval = 30
>
>  ‚ÄčKindly let me know what is it, that I'm doing wrong.
>
>
>  Sincerely,
>  Sanjay Ramanathan
>
>
>

Mime
View raw message