predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Szeto <don...@apache.org>
Subject Re: Hbase issue
Date Thu, 12 Apr 2018 15:43:03 GMT
Hi Bala,

Are you running a single-machine HBase setup? The ZooKeeper embedded in
such a setup is pretty fragile to disk space issue and your ZNode might
have corrupted.

If that’s indeed your setup, please take a look at HBase log files,
specifically on messages from ZooKeeper. In this situation, one way to
recover is to remove ZooKeeper files and let HBase recreate them, assuming
from your log output that you don’t have other services depend on the same
ZK.

Regards,
Donald

On Thu, Apr 12, 2018 at 5:34 AM bala vivek <bala.vivek123@gmail.com> wrote:

> Hi,
>
> I use PIO 0.10.0 version and hbase 1.2.4. The setup was working fine till
> today morning. I saw PIO was down as the mount space issue was present on
> the server and cleared the unwanted files.
>
> After doing a pio-stop-all and pio-start-all the HMaster service is not
> working. I tried multiple times the pio restart.
>
> I can see whenever I do a pio-stop-all and check the service using jps,
> the Hmaster seems running. Similarly I tried to run the ./start-hbase.sh
> script but still pio status is not showing as success.
>
> pio error log :
>
> [INFO] [Console$] Inspecting PredictionIO...
> [INFO] [Console$] PredictionIO 0.10.0-incubating is installed at
> /opt/tools/PredictionIO-0.10.0-incubating
> [INFO] [Console$] Inspecting Apache Spark...
> [INFO] [Console$] Apache Spark is installed at
> /opt/tools/PredictionIO-0.10.0-incubating/vendors/spark-1.6.3-bin-hadoop2.6
> [INFO] [Console$] Apache Spark 1.6.3 detected (meets minimum requirement
> of 1.3.0)
> [INFO] [Console$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [INFO] [Storage$] Verifying Model Data Backend (Source: LOCALFS)...
> [INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)...
> [ERROR] [RecoverableZooKeeper] ZooKeeper exists failed after 1 attempts
> [ERROR] [ZooKeeperWatcher] hconnection-0x7c891ba7, quorum=localhost:2181,
> baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
> [WARN] [ZooKeeperRegistry] Can't retrieve clusterId from Zookeeper
> [ERROR] [StorageClient] Cannot connect to ZooKeeper (ZooKeeper ensemble:
> localhost). Please make sure that the configuration is pointing at the
> correct ZooKeeper ensemble. By default, HBase manages its own ZooKeeper, so
> if you have not configured HBase to use an external ZooKeeper, that means
> your HBase is not started or configured properly.
> [ERROR] [Storage$] Error initializing storage client for source HBASE
> [ERROR] [Console$] Unable to connect to all storage backends successfully.
> The following shows the error message from the storage backend.
> [ERROR] [Console$] Data source HBASE was not properly initialized.
> (org.apache.predictionio.data.storage.StorageClientException)
> [ERROR] [Console$] Dumping configuration of initialized storage backend
> sources. Please make sure they are correct.
> [ERROR] [Console$] Source Name: ELASTICSEARCH; Type: elasticsearch;
> Configuration: TYPE -> elasticsearch, HOME ->
> /opt/tools/PredictionIO-0.10.0-incubating/vendors/elasticsearch-1.7.3
> [ERROR] [Console$] Source Name: LOCALFS; Type: localfs; Configuration:
> PATH -> /root/.pio_store/models, TYPE -> localfs
> [ERROR] [Console$] Source Name: HBASE; Type: (error); Configuration:
> (error)
>
>
> Regards,
> Bala
>

Mime
View raw message