hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Loading of coprocessors on a region multiple times
Date Tue, 12 Sep 2017 10:48:12 GMT
loadTableCoprocessors() is called from RegionCoprocessorHost ctor.
In HRegion ctor, you would find:

      this.coprocessorHost = new RegionCoprocessorHost(this, rsServices,
conf);

Meaning RegionCoprocessorHost is instantiated per region.
This can explain why you saw the log multiple times.

On Tue, Sep 12, 2017 at 2:34 AM, Steen Manniche <boxunbox@gmail.com> wrote:

> Hi list.
>
> I have a coprocessor which has been loaded on a table 'plot'. The
> table has 2697 regions across 8 nodes. Recently, I have observed some
> strangeness in the log-files:
>
> 2017-09-12 11:00:28,461 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:00:29,080 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:00:29,648 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:00:30,464 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:00:31,204 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:00:31,874 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:00:32,253 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:01:52,703 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:02:08,403 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:02:18,215 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:02:37,779 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:02:49,227 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:03:01,271 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:09:10,639 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
> 2017-09-12 11:14:08,939 INFO  [RS_OPEN_REGION-sild03:16020-16]
> regionserver.RegionCoprocessorHost: Loaded coprocessor
> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> protein_ee successfully.
>
> All in all, There are thousand of these log-lines on the regionserver
> logs. The coprocessor is quite simple
> (https://github.com/manniche/hbase-table-event-signaler/
> blob/develop/src/main/java/net/nzcorp/hbase/tableevent_
> signaler/TableEventSignaler.java)
>
> I would assume to find this line exactly once per region, but it seems
> that the `start()` method on the coprocessor is called multiple times?
>
> the hbase describe table shows:
>
> hbase(main):007:0> describe 'plot'
> Table plot is ENABLED
> plot, {TABLE_ATTRIBUTES => {coprocessor$5 =>
> 'hdfs:///user/hbase/nzcorp-tableevent-signaler-0.10.0.
> jar|net.nzcorp.hbase.tableevent_signaler.TableEventSignaler|5|
> destination_table=plot,secondary_index_table=plot_index,seco
> ndary_index_cf=p,source_column_family=e,target_column_
> family=e,amq_address=amqp://*******:******@rmq-server1:5672/
> hbase_events,send_value=true,filter_qualifiers=dna_
> accession_number|feature_accession_number'}
> [... info on column families ...]
> 9 row(s) in 0.3650 seconds
>
> We are running hbase version 1.1.2
>
> Best regards,
> Steen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message