hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samir Ahmic <ahmic.sa...@gmail.com>
Subject Re: constantly adding snapshot info
Date Sun, 27 Dec 2015 19:22:58 GMT
Hi Brian,
Did you have created snapshots on cluster while you were upgrading to new
hbase version ? Based on your logs there are some snapshots operations
running on your cluster. Can you dig your logs to find what is
triggering SnapshotManifestLoader there should some more info just above
this lines you have pasted here.

Regards
Samir

On Sun, Dec 27, 2015 at 6:34 PM, Brian Jeltema <bdjeltema@gmail.com> wrote:

> I recently upgraded to Hadoop 2.7.1 and HBase 1.1.2.
>
> Since the upgrade, the HBase master logs have been filling and rolling
> over about
> every 5 minutes, filled with variations of the following (modified to
> obscure internal details):
>
> 2015-12-27 12:24:52,481 DEBUG [SnapshotManifestLoader-pool878551-t3]
> snapshot.SnapshotManifestV1: Creating references for hfiles
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8]
> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8]
> snapshot.SnapshotManifestV1: Creating references for hfiles
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6]
> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6]
> snapshot.SnapshotManifestV1: Creating references for hfiles
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5]
> snapshot.SnapshotManifestV1: Adding snapshot references for
> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>]
> hfiles
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5]
> snapshot.SnapshotManifestV1: Adding reference for file (1/1):
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7]
> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7]
> snapshot.SnapshotManifestV1: Creating references for hfiles
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4]
> snapshot.SnapshotManifestV1: Adding snapshot references for
> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>,
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>,
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]>
> hfiles
> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4]
> snapshot.SnapshotManifestV1: Adding reference for file (1/3):
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>
> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4]
> snapshot.SnapshotManifestV1: Adding reference for file (2/3):
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>
> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4]
> snapshot.SnapshotManifestV1: Adding reference for file (3/3):
> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4>
> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t3]
> snapshot.SnapshotManifestV1: Adding snapshot references for
> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3
> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3>
>
> This is also causing my HDFS audit logs to grow at a rate of about
> 1.5MB/s. Has anyone observed
> this behavior or know a fix?
>
> Thanks
> Brian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message