hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Jeltema <bdjelt...@gmail.com>
Subject Re: constantly adding snapshot info
Date Mon, 28 Dec 2015 01:25:15 GMT
> Hi Brian,
> Did you have created snapshots on cluster while you were upgrading to new
> hbase version ?

There were existing snapshots when I upgraded. It looks like a bunch of
new snapshot also appeared as part of the process, with names like ‘foo-ru-yyyymmdd’

> Based on your logs there are some snapshots operations
> running on your cluster. Can you dig your logs to find what is
> triggering SnapshotManifestLoader there should some more info just above
> this lines you have pasted here.

I didn’t see anything in the logs significantly different than what I included
in the previous post.

Brian

> 
> Regards
> Samir
> 
> On Sun, Dec 27, 2015 at 6:34 PM, Brian Jeltema <bdjeltema@gmail.com> wrote:
> 
>> I recently upgraded to Hadoop 2.7.1 and HBase 1.1.2.
>> 
>> Since the upgrade, the HBase master logs have been filling and rolling
>> over about
>> every 5 minutes, filled with variations of the following (modified to
>> obscure internal details):
>> 
>> 2015-12-27 12:24:52,481 DEBUG [SnapshotManifestLoader-pool878551-t3]
>> snapshot.SnapshotManifestV1: Creating references for hfiles
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8]
>> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8]
>> snapshot.SnapshotManifestV1: Creating references for hfiles
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6]
>> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6]
>> snapshot.SnapshotManifestV1: Creating references for hfiles
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5]
>> snapshot.SnapshotManifestV1: Adding snapshot references for
>> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>]
>> hfiles
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5]
>> snapshot.SnapshotManifestV1: Adding reference for file (1/1):
>> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7]
>> snapshot.SnapshotManifestV1: Storing region-info for snapshot.
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7]
>> snapshot.SnapshotManifestV1: Creating references for hfiles
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4]
>> snapshot.SnapshotManifestV1: Adding snapshot references for
>> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>,
>> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>,
>> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]>
>> hfiles
>> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4]
>> snapshot.SnapshotManifestV1: Adding reference for file (1/3):
>> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>
>> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4]
>> snapshot.SnapshotManifestV1: Adding reference for file (2/3):
>> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>
>> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4]
>> snapshot.SnapshotManifestV1: Adding reference for file (3/3):
>> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4>
>> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t3]
>> snapshot.SnapshotManifestV1: Adding snapshot references for
>> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3
>> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3>
>> 
>> This is also causing my HDFS audit logs to grow at a rate of about
>> 1.5MB/s. Has anyone observed
>> this behavior or know a fix?
>> 
>> Thanks
>> Brian


Mime
View raw message