hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
Date Sun, 22 Oct 2017 03:38:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214095#comment-16214095
] 

stack edited comment on HBASE-18846 at 10/22/17 3:37 AM:
---------------------------------------------------------

bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go
through sometime and read it with a tool that can ignore those.

Sorry about that. Let me try and remove some. This stuff needs a refactor badly. I can't help
myself.

bq.  I guess we still check if the handler is enabled, but it looks less clean than the other
options we're exposing.

Replication startup is complicated (I've yet to figure it out). Default in hbase is to start
Replication. In the hollowed-out HRegionServer, we need Replication so we put up a Sink to
receive Replication. On enable/disable of Replication, I'd like to do that in another issue.
We have a million services/chores/managers and they all work different. Ideally, they'd all
work the same and the flag to stop/start it would be same pattern no matter what the Service;
but that is TODO I'd day.

Idea behind this issue is to get the hbase-indexer folks something they can work with; my
guess is they will come back with other stuff they'd like to be handled in subsequent issues
before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing
HRegionServer in Isolation.

bq. On the health check, can we get into a false positive state where a thread is null but
shouldn't be? Is it better to check the configuration source of truth there instead?

Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on.

bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either
a finally block or a @After method so that we don't pollute other tests if we fail.

Done. Added more tests too... for some of the other configs introduced here.





was (Author: stack):
bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go
through sometime and read it with a tool that can ignore those.

Sorry about that. Let me try and remove some. This stuff needs a refactor. I can't help myself.

bq.  I guess we still check if the handler is enabled, but it looks less clean than the other
options we're exposing.

On Replication startup, I moved it under general startServices for now -- it'll be off in
hollow-HRegionServer deploy ... Moved webui OUT of startServices because that already has
its own enable/disable (Idea behind this issue is to get the hbase-indexer folks something
they can work with; my guess is they will come back with other stuff they'd like to be handled
in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll
be useful testing HRegionServer in Isolation.  And whole world of start/stop services/managers/etc.
is in bad need of cleanup. TODO).

bq. On the health check, can we get into a false positive state where a thread is null but
shouldn't be? Is it better to check the configuration source of truth there instead?

Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on.

bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either
a finally block or a @After method so that we don't pollute other tests if we fail.

Done. Added more tests too... for some of the other configs introduced here.




> Accommodate the hbase-indexer/lily/SEP consumer deploy-type
> -----------------------------------------------------------
>
>                 Key: HBASE-18846
>                 URL: https://issues.apache.org/jira/browse/HBASE-18846
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 2.0.0-alpha-4
>
>         Attachments: HBASE-18846.master.001.patch, HBASE-18846.master.002.patch, HBASE-18846.master.003.patch,
HBASE-18846.master.004.patch, IndexerConnection.java, hbase-site.xml, javadoc.txt
>
>
> This is a follow-on from HBASE-10504, Define a Replication Interface. There we defined
a new, flexible replication endpoint for others to implement but it did little to help the
case of the lily hbase-indexer. This issue takes up the case of the hbase-indexer.
> The hbase-indexer poses to hbase as a 'fake' peer cluster (For why hbase-indexer is implemented
so, the advantage to having the indexing done in a separate process set that can be independently
scaled, can participate in the same security realm, etc., see discussion in HBASE-10504).
The hbase-indexer will start up a cut-down "RegionServer" processes that are just an instance
of hbase RpcServer hosting an AdminProtos Service. They make themselves 'appear' to the Replication
Source by hoisting up an ephemeral znode 'registering' as a RegionServer. The source cluster
then streams WALEdits to the Admin Protos method:
> {code}
>  public ReplicateWALEntryResponse replicateWALEntry(final RpcController controller,
>       final ReplicateWALEntryRequest request) throws ServiceException {
> {code}
> The hbase-indexer relies on other hbase internals like Server so it can get a ZooKeeperWatcher
instance and know the 'name' to use for this cut-down server.
> Thoughts on how to proceed include:
>  
>  * Better formalize its current digestion of hbase internals; make it so rpcserver is
allowed to be used by others, etc. This would be hard to do given they use basics like Server,
Protobuf serdes for WAL types, and AdminProtos Service. Any change in this wide API breaks
(again) hbase-indexer. We have made a 'channel' for Coprocessor Endpoints so they continue
to work though they use 'internal' types. They can use protos in hbase-protocol. hbase-protocol
protos are in a limbo currently where they are sort-of 'public'; a TODO. Perhaps the hbase-indexer
could do similar relying on the hbase-protocol (pb2.5) content and we could do something to
reveal rpcserver and zk for hbase-indexer safe use.
>  * Start an actual RegionServer only have it register the AdminProtos Service only --
not ClientProtos and the Service that does Master interaction, etc. [I checked, this is not
as easy to do as I at first thought -- St.Ack] Then have the hbase-indexer implement an AdminCoprocessor
to override the replicateWALEntry method (the Admin CP implementation may need work). This
would narrow the hbase-indexer exposure to that of the Admin Coprocessor Interface
>  * Over in HBASE-10504, [~enis] suggested "... if we want to provide isolation for the
replication services in hbase, we can have a simple host as another daemon which hosts the
ReplicationEndpoint implementation. RS's will use a built-in RE to send the edits to this
layer, and the host will delegate it to the RE implementation. The flow would be something
like:  RS --> RE inside RS --> Host daemon for RE --> Actual RE implementation -->
third party system..."
>  
> Other crazy notions occur including the setup of an Admin Interface Coprocessor Endpoint.
A new ReplicationEndpoint would feed the replication stream to the remote cluster via the
CPEP registered channel.
> But time is short. Hopefully we can figure something that will work in 2.0 timeframe
w/o too much code movement.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message