hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chathuri Wimalasena <kamalas...@gmail.com>
Subject Re: Unusual log generation in Hadoop namenode logs
Date Sat, 18 Jun 2016 17:13:50 GMT
Hi Ted,

Similar error appears for different blocks. When I check the region-server
logs for one of such region server, I can see a log like this. I have no
idea what this means. Similar log appears in each region server logs. I
restarted the server last night during this time and these logs showed
after restart.

2016-06-17 20:12:08,234 WARN org.apache.hadoop.ipc.HBaseServer:
(responseTooSlow): {"processingtimems":13826,"call":"openRegions([{NAME =>
'dailyTweetCountTable,,1463586976452.5bb7e255c84055fc22777aa5669d84e5.',
STARTKEY => '', ENDKEY => '', ENCODED =>
5bb7e255c84055fc22777aa5669d84e5,}, {NAME =>
'geoIndexTable-2010-05,,1448647271977.ac9197ecfba5ee2079ff48603f847050.',
STARTKEY => '', ENDKEY => '', ENCODED =>
ac9197ecfba5ee2079ff48603f847050,}, {NAME =>
'geoIndexTable-2010-06,,1448647263331.5ae58506ae72a7505fc8a40921b9ca01.',
STARTKEY => '', ENDKEY => '', ENCODED =>
5ae58506ae72a7505fc8a40921b9ca01,}, {NAME =>
'geoIndexTable-2010-07,,1448647255029.8648eee2b4225a2ea98c8ea61a0a44a0.',
STARTKEY => '', ENDKEY => '', ENCODED =>
8648eee2b4225a2ea98c8ea61a0a44a0,}, {NAME =>
'geoIndexTable-2010-08,,1448647246755.c8ba053fc36f27a9ac2c178679f0488e.',
STARTKEY => '', ENDKEY => '', ENCODED =>
c8ba053fc36f27a9ac2c178679f0488e,}, {NAME =>
'geoIndexTable-2010-09,,1448647238153.d8fe74e350cacf53fc03a89900c68470.',
STARTKEY => '', ENDKEY => '', ENCODED =>
d8fe74e350cacf53fc03a89900c68470,}, {NAME =>
'geoIndexTable-2010-10,,1448647229607.66137956b612a13c4836533f8bf7901b.',
STARTKEY => '', ENDKEY => '', ENCODED =>
66137956b612a13c4836533f8bf7901b,}, {NAME =>
'geoIndexTable-2010-11,,1448647216117.edb26d17ad97461787129a08798b73f9.',
STARTKEY => '', ENDKEY => '', ENCODED =>
edb26d17ad97461787129a08798b73f9,}, {NAME =>
'geoIndexTable-2010-12,,1448647206835.f491a9d545f0071b7d9a7523bcc0eb0d.',
STARTKEY => '', ENDKEY => '', ENCODED =>
f491a9d545f0071b7d9a7523bcc0eb0d,}, {NAME =>
'geoIndexTable-2011-01,,1448334295446.9b5b7fb57f2ba5b964596cb9d5e7dc53.',
STARTKEY => '', ENDKEY => '', ENCODED =>
9b5b7fb57f2ba5b964596cb9d5e7dc53,}, {NAME =>
'geoIndexTable-2011-02,,1448334286105.f63660b053d21c44c570e0ea0b40fda9.',
STARTKEY => '', ENDKEY => '', ENCODED =>
f63660b053d21c44c570e0ea0b40fda9,}, {NAME =>
'geoIndexTable-2011-03,,1448334276625.15ca34fb9bf26742a81bb3059903df85.',
STARTKEY => '', ENDKEY => '', ENCODED =>
15ca34fb9bf26742a81bb3059903df85,}, {NAME =>
'geoIndexTable-2011-04,,1448334253107.46c25193de28f3729eb5c82b02d57d1e.',
STARTKEY => '', ENDKEY => '', ENCODED =>
46c25193de28f3729eb5c82b02d57d1e,}, {NAME =>
'geoIndexTable-2011-05,,1448334241005.e1b70b296cd1b6d458b46fde8562b99e.',
STARTKEY => '', ENDKEY => '', ENCODED =>
e1b70b296cd1b6d458b46fde8562b99e,}, {NAME =>
'geoIndexTable-2011-06,,1448334231051.6b0624afcd76d17b8bd89d75343486e2.',
STARTKEY => '', ENDKEY => '', ENCODED =>
6b0624afcd76d17b8bd89d75343486e2,}, {NAME =>
'geoIndexTable-2011-07,,1448334221528.a44521ed13506716f9c70e774596137a.',
STARTKEY => '', ENDKEY => '', ENCODED =>
a44521ed13506716f9c70e774596137a,}, {NAME =>
'geoIndexTable-2011-08,,1448334211917.a2051bbac9578fabf21bfb071c7fcfd3.',
STARTKEY => '', ENDKEY => '', ENCODED =>
a2051bbac9578fabf21bfb071c7fcfd3,}, {NAME =>
'geoIndexTable-2011-09,,1448334201977.d1ec93239fdc616911da9a3802907462.',
STARTKEY => '', ENDKEY => '', ENCODED =>
d1ec93239fdc616911da9a3802907462,}, {NAME =>
'geoIndexTable-2011-10,,1448334189415.969b69327d39e2e27d27afbbea867ebb.',
STARTKEY => '', ENDKEY => '', ENCODED =>
969b69327d39e2e27d27afbbea867ebb,}, {NAME =>
'geoIndexTable-2011-11,,1448334179915.fa2093ff09eeb2b80728e28c3cc6dc52.',
STARTKEY => '', ENDKEY => '', ENCODED =>
fa2093ff09eeb2b80728e28c3cc6dc52,}, {NAME =>
'geoIndexTable-2011-12,,1448334170540.1c8da5e32729c3d950a86f2b4c77b1f3.',
STARTKEY => '', ENDKEY => '', ENCODED =>
1c8da5e32729c3d950a86f2b4c77b1f3,}, {NAME =>
'geoIndexTable-2012-01,,1447981747677.e798784dac6d307db89b924a93239bae.',
STARTKEY => '', ENDKEY => '', ENCODED =>
e798784dac6d307db89b924a93239bae,}, {NAME =>
'geoIndexTable-2012-02,,1447981738161.6f3a5c1ba77bb4cab7fcc7298e566e6b.',
STARTKEY => '', ENDKEY => '', ENCODED =>
6f3a5c1ba77bb4cab7fcc7298e566e6b,}, {NAME =>
'geoIndexTable-2012-03,,1447981728637.f92fa7dda5ad4b6910cdd8575ecba3f2.',
STARTKEY => '', ENDKEY => '', ENCODED =>
f92fa7dda5ad4b6910cdd8575ecba3f2,}, {NAME =>
'geoIndexTable-2012-04,,1447981718427.5aca6cedd937eb8fa7ca667652ec363a.',
STARTKEY => '', ENDKEY => '', ENCODED =>
5aca6cedd937eb8fa7ca667652ec363a,}, {NAME =>
'geoIndexTable-2012-05,,1447981709307.4043f99b2c2b1da50dc9c5366b52550f.',
STARTKEY => '', ENDKEY => '', ENCODED =>
4043f99b2c2b1da50dc9c5366b52550f,}, {NAME =>
'geoIndexTable-2012-06,,1447981700265.fb6694ced4c3662abe48c2c12d7b31e3.',
STARTKEY => '', ENDKEY => '', ENCODED =>
fb6694ced4c3662abe48c2c12d7b31e3,}, {NAME =>
'geoIndexTable-2012-07,,1447981689467.32072590c0d6ebf70a264e4ca0f1f28a.',
STARTKEY => '', ENDKEY => '', E

On Sat, Jun 18, 2016 at 11:37 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Can you find out which region server hostsed
> d654d01588e8a46d7050852978f8eaf9
> and examine its log to see if there was some clue ?
>
> Thanks
>
> On Sat, Jun 18, 2016 at 8:29 AM, Chathuri Wimalasena <kamalasini@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > I'm using HBase 0.94-23 with Hadoop 2.7.2. In my hadoop namenode log
> file,
> > I'm getting below log info message over and over again. No jobs are
> running
> > still seeing the same log appearing. When I run a hadoop job, some jobs
> > failed with timeout. The cluster has 10 data nodes with 200TB of data.
> > When I stop HBase, logs no longer appear.
> >
> > 16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
> > updated: 10.10.2.1:44949 is added to
> > blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> > primaryNodeIndex=-1,
> >
> > replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:
> 10.10.2.6:44949
> > |RBW],
> >
> > ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:
> 10.10.2.1:44949
> > |RBW],
> >
> > ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:
> 10.10.2.3:44949
> > |RBW]]}
> > size 0
> > 16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
> > updated: 10.10.2.6:44949 is added to
> > blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> > primaryNodeIndex=-1,
> >
> > replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:
> 10.10.2.6:44949
> > |RBW],
> >
> > ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:
> 10.10.2.1:44949
> > |RBW],
> >
> > ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:
> 10.10.2.3:44949
> > |RBW]]}
> > size 0
> > 16/06/18 11:23:51 INFO hdfs.StateChange: BLOCK* allocate
> > blk_1077143674_3402999{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> > primaryNodeIndex=-1,
> >
> > replicas=[ReplicaUC[[DISK]DS-8af58951-0eb7-4244-b76a-9356ad832a93:NORMAL:
> 10.10.2.6:44949
> > |RBW],
> >
> > ReplicaUC[[DISK]DS-6095b306-487e-48d7-948e-84ac5ef16aa4:NORMAL:
> 10.10.2.9:44949
> > |RBW],
> >
> > ReplicaUC[[DISK]DS-9b2f64f6-bbac-4ed5-a1f9-9366129d196b:NORMAL:
> 10.10.2.1:44949
> > |RBW]]}
> > for
> >
> >
> /hbase/tweetTable-2016-02/d654d01588e8a46d7050852978f8eaf9/.tmp/14bc58b73bd74cf28cf7e43b2304e4d6
> >
> > Appreciate any insight on this.
> >
> > Regards,
> > Chathuri
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message