hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15711) Add client side property to allow logging details for batch errors
Date Wed, 27 Apr 2016 02:35:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259424#comment-15259424
] 

Yu Li commented on HBASE-15711:
-------------------------------

Thanks for review [~stack]

bq. Needs some doc though else we'll just forget it exists.
Agree. Where should be place the doc? Here in the release note, or hbase-default.xml, or anywhere
else? Thanks.

bq. PIty we could not dynamically set this config? Can we not? Via the UI as we set log levels
now? We could set a few UI configs... like this one?
It's also possible to dynamically set the client-side config such as in AsyncProcess? I thought
it's a server-side feature only... Could you give me more hints here? Thanks.

> Add client side property to allow logging details for batch errors
> ------------------------------------------------------------------
>
>                 Key: HBASE-15711
>                 URL: https://issues.apache.org/jira/browse/HBASE-15711
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Yu Li
>            Assignee: Yu Li
>         Attachments: HBASE-15711.patch
>
>
> Recently we observed below error message on client side when put process blocked:
> {noformat}
> 2016-04-26 10:27:11,707 ERROR [Sink: Unnamed (1/1)] com.alibaba.search.blink.streaming.connector.hbase.HBaseOutputFormat:
Doing mutation failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action:
IOException: 1 time,
>         at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
>         at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
>         at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1694)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> {noformat}
> And checking RS logs we found nothing noticable. After adding some logging to show the
detailed exception, it turns out something went wrong in one of our coprocessors:
> {noformat}
> 2016-04-26 12:03:13,776 ERROR [Sink: Unnamed (1/1)] org.apache.hadoop.hbase.client.AsyncProcess:
Exception occurred! Exception details: [java.io.IOException: java.io.IOException: notify meta
has not load success.
>         at com.taobao.kart.coprocessor.server.CoprocessorNotifyMeta.checkMetaLoadSuccess(CoprocessorNotifyMeta.java:38)
>         at com.taobao.kart.coprocessor.server.NotifyQualifySetter.updatePut(NotifyQualifySetter.java:47)
>         at com.taobao.kart.coprocessor.server.NotifyQualifySetter.updatePut(NotifyQualifySetter.java:39)
>         at com.taobao.kart.coprocessor.server.KartCoprocessor.prePut(KartCoprocessor.java:176)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:902)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:898)
>         at org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2890)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2865)
> {noformat}
> So in this JIRA we propose to add a property to allow logging detailed exception stacktrace
rather than statistics for batch errors, and I believe this would be useful for debugging
in some cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message