hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Loading of coprocessors on a region multiple times
Date Tue, 12 Sep 2017 14:08:27 GMT
Here is how ExecutorType forms the name:

  String getExecutorName(String serverName) {

    return this.toString() + "-" + serverName.replace("%", "%%");

Then Executor is created:

    Executor hbes = new Executor(name, maxThreads, this.
eventHandlerListeners);

In your sample, 16 was the thread number.

Cheers

On Tue, Sep 12, 2017 at 4:31 AM, Steen Manniche <boxunbox@gmail.com> wrote:

> Hi Ted,
>
> thanks for your quick answer. It would indeed. But since we have in
> the vicinity of 6K such log-lines for the table in a single
> regionserver, should I then interpret the the number coming after the
> RS_OPEN_REGION-{servername:port}- as the thread id? E.g.
> RS_OPEN_REGION-sild03:16020-16 as thread number 16 that operates on
> the region?
>
> Best regards,
> Steen
>
> On Tue, Sep 12, 2017 at 12:48 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > loadTableCoprocessors() is called from RegionCoprocessorHost ctor.
> > In HRegion ctor, you would find:
> >
> >       this.coprocessorHost = new RegionCoprocessorHost(this, rsServices,
> > conf);
> >
> > Meaning RegionCoprocessorHost is instantiated per region.
> > This can explain why you saw the log multiple times.
> >
> > On Tue, Sep 12, 2017 at 2:34 AM, Steen Manniche <boxunbox@gmail.com>
> wrote:
> >
> >> Hi list.
> >>
> >> I have a coprocessor which has been loaded on a table 'plot'. The
> >> table has 2697 regions across 8 nodes. Recently, I have observed some
> >> strangeness in the log-files:
> >>
> >> 2017-09-12 11:00:28,461 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:00:29,080 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:00:29,648 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:00:30,464 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:00:31,204 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:00:31,874 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:00:32,253 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:01:52,703 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:02:08,403 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:02:18,215 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:02:37,779 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:02:49,227 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:03:01,271 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:09:10,639 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >> 2017-09-12 11:14:08,939 INFO  [RS_OPEN_REGION-sild03:16020-16]
> >> regionserver.RegionCoprocessorHost: Loaded coprocessor
> >> net.nzcorp.hbase.tableevent_signaler.TableEventSignaler from HTD of
> >> protein_ee successfully.
> >>
> >> All in all, There are thousand of these log-lines on the regionserver
> >> logs. The coprocessor is quite simple
> >> (https://github.com/manniche/hbase-table-event-signaler/
> >> blob/develop/src/main/java/net/nzcorp/hbase/tableevent_
> >> signaler/TableEventSignaler.java)
> >>
> >> I would assume to find this line exactly once per region, but it seems
> >> that the `start()` method on the coprocessor is called multiple times?
> >>
> >> the hbase describe table shows:
> >>
> >> hbase(main):007:0> describe 'plot'
> >> Table plot is ENABLED
> >> plot, {TABLE_ATTRIBUTES => {coprocessor$5 =>
> >> 'hdfs:///user/hbase/nzcorp-tableevent-signaler-0.10.0.
> >> jar|net.nzcorp.hbase.tableevent_signaler.TableEventSignaler|5|
> >> destination_table=plot,secondary_index_table=plot_index,seco
> >> ndary_index_cf=p,source_column_family=e,target_column_
> >> family=e,amq_address=amqp://*******:******@rmq-server1:5672/
> >> hbase_events,send_value=true,filter_qualifiers=dna_
> >> accession_number|feature_accession_number'}
> >> [... info on column families ...]
> >> 9 row(s) in 0.3650 seconds
> >>
> >> We are running hbase version 1.1.2
> >>
> >> Best regards,
> >> Steen
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message