hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samir Ahmic <ahmic.sa...@gmail.com>
Subject Re: constantly adding snapshot info
Date Mon, 28 Dec 2015 08:58:18 GMT
Based on what i have found in code there are two processes related to
snapshots that are triggered every 5 minutes by default
SnapshotHFileCleaner and SnapshotLogCleaner but none of them can explain
why do you see new snapshots created. Have you been restoring snapshots,
that can explain new snapshots that you see ? I would suggest that you
delete existing snapshots and recreate them if that is acceptable option
for you. Also you may try to restart master.

Regards
Samir

On Mon, Dec 28, 2015 at 2:25 AM, Brian Jeltema <bdjeltema@gmail.com> wrote:

> > Hi Brian,
> > Did you have created snapshots on cluster while you were upgrading to new
> > hbase version ?
>
> There were existing snapshots when I upgraded. It looks like a bunch of
> new snapshot also appeared as part of the process, with names like
> ‘foo-ru-yyyymmdd’
>
> > Based on your logs there are some snapshots operations
> > running on your cluster. Can you dig your logs to find what is
> > triggering SnapshotManifestLoader there should some more info just above
> > this lines you have pasted here.
>
> I didn’t see anything in the logs significantly different than what I
> included
> in the previous post.
>
> Brian
>
> >
> > Regards
> > Samir
> >
> > On Sun, Dec 27, 2015 at 6:34 PM, Brian Jeltema <bdjeltema@gmail.com>
> wrote:
> >
> >> I recently upgraded to Hadoop 2.7.1 and HBase 1.1.2.
> >>
> >> Since the upgrade, the HBase master logs have been filling and rolling
> >> over about
> >> every 5 minutes, filled with variations of the following (modified to
> >> obscure internal details):
> >>
> >> 2015-12-27 12:24:52,481 DEBUG [SnapshotManifestLoader-pool878551-t3]
> >> snapshot.SnapshotManifestV1: Creating references for hfiles
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8]
> >> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8]
> >> snapshot.SnapshotManifestV1: Creating references for hfiles
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6]
> >> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6]
> >> snapshot.SnapshotManifestV1: Creating references for hfiles
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5]
> >> snapshot.SnapshotManifestV1: Adding snapshot references for
> >>
> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>]
> >> hfiles
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5]
> >> snapshot.SnapshotManifestV1: Adding reference for file (1/1):
> >>
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7]
> >> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7]
> >> snapshot.SnapshotManifestV1: Creating references for hfiles
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4]
> >> snapshot.SnapshotManifestV1: Adding snapshot references for
> >>
> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>,
> >>
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>,
> >>
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]>
> >> hfiles
> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4]
> >> snapshot.SnapshotManifestV1: Adding reference for file (1/3):
> >>
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>
> >> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4]
> >> snapshot.SnapshotManifestV1: Adding reference for file (2/3):
> >>
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>
> >> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4]
> >> snapshot.SnapshotManifestV1: Adding reference for file (3/3):
> >>
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4>
> >> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t3]
> >> snapshot.SnapshotManifestV1: Adding snapshot references for
> >>
> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3
> >>
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3>
> >>
> >> This is also causing my HDFS audit logs to grow at a rate of about
> >> 1.5MB/s. Has anyone observed
> >> this behavior or know a fix?
> >>
> >> Thanks
> >> Brian
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message