hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gagan Brahmi <gaganbra...@gmail.com>
Subject Re: PXF service could not be reached
Date Wed, 16 Mar 2016 01:58:48 GMT
I made some progress (or so I think) and have a different error now.
The interesting part remains the write into HDFS. The SELECT query
continue to work fine.

Here's the error I am receiving now.

gagan=# INSERT INTO ext_put_foo VALUES (1, 'Gagan Brahmi');
ERROR:  remote component error (0): Failed to connect to
my-hadoop-cluster port 51200: Connection timed out (libchurl.c:878)
(seg0 my-hadoop-cluster:40000 pid=25073) (dispatcher.c:1753)

The select query continue to work and fetch results from HDFS.

gagan=# SELECT * FROM ext_get_foo ;
  i  |      bar
-----+---------------
 100 |  Gagan Brahmi
 101 |  Another Emp
 102 |  Some More
(3 rows)

The telnet for 51200 works fine from the segment & master (single node
machine) so does the connection to hdfs (8020).

I have attached some logs and Stack Trace found in Master and Segment logs.

Is there a simpler way to test if pxf-service with it's classpaths and
jar able to write to HDFS?



Regards,
Gagan Brahmi

On Tue, Mar 15, 2016 at 4:09 PM, Ting(Goden) Yao <tyao@pivotal.io> wrote:
> Gagan - did you see any error in HAWQ logs or Hadoop logs?
> This doesn't look like a PXF issue.
>
> On Fri, Mar 11, 2016 at 8:59 AM Gagan Brahmi <gaganbrahmi@gmail.com> wrote:
>>
>> And this is what I mean when I say that fetch (select) seems to be
>> working. I place a dummy file to run select query and it provides the
>> results.
>>
>> hdfs@my-hadoop-cluster:~> cat /tmp/test_fetch
>> 100 | Random Value
>> 101 | Another Random
>> hdfs@my-hadoop-cluster:~> hadoop fs -put /tmp/test_fetch /tmp/foo_bar/
>> hdfs@suse11-workplace:~> logout
>> my-hadoop-cluster:~ # su - gpadmin
>> gpadmin@my-hadoop-cluster:~> source /usr/local/hawq/greenplum_path.sh
>> gpadmin@my-hadoop-cluster:~> psql -p 10432 gagan
>> psql (8.2.15)
>> Type "help" for help.
>>
>> gagan=# SELECT * FROM ext_get_foo ;
>>   i  |       bar
>> -----+-----------------
>>  100 |  Random Value
>>  101 |  Another Random
>> (2 rows)
>>
>> gagan=#
>>
>>
>> Table DDLs
>>
>> gagan=# CREATE WRITABLE EXTERNAL TABLE ext_put_foo (i int, bar text)
>> LOCATION
>> ('pxf://my-hadoop-cluster:51200/tmp/foo_bar?profile=HdfsTextSimple')
>> FORMAT 'text' (delimiter '|' null 'null');
>> CREATE EXTERNAL TABLE
>> gagan=# CREATE EXTERNAL TABLE ext_get_foo (i int, bar text) LOCATION
>> ('pxf://my-hadoop-cluster:51200/tmp/foo_bar?profile=HdfsTextSimple')
>> FORMAT 'text' (delimiter '|' null 'null');
>> CREATE EXTERNAL TABLE
>> gagan=# INSERT into ext_put_foo VALUES (1, 'Gagan');
>> ERROR:  failed sending to remote component (libchurl.c:574)  (seg0
>> my-hadoop-cluster:40000 pid=824) (dispatcher.c:1753)
>>
>>
>> Regards,
>> Gagan Brahmi
>>
>> On Fri, Mar 11, 2016 at 9:53 AM, Gagan Brahmi <gaganbrahmi@gmail.com>
>> wrote:
>> > Nothing in the pxf-service.log or the catalina.out for pxf service.
>> >
>> > It has the normal startup messages while the webapp starts up. I did
>> > modified the overcommit memory to 1 and restarted all the service
>> > (just in case). But that still didn't seem to have made any
>> > difference.
>> >
>> > I still see the file is closed by DFSClient message in HDFS everytime
>> > I try to run an insert command. The select looks to be working fine.
>> >
>> >
>> > Regards,
>> > Gagan Brahmi
>> >
>> > On Fri, Mar 11, 2016 at 9:21 AM, Daniel Lynch <dlynch@pivotal.io> wrote:
>> >> check the pxf service logs for errors.  I suspect there is an out of
>> >> memory
>> >> event at some point during the connection considering this is a single
>> >> node
>> >> deployment.
>> >>
>> >> Also make sure overcommit is disabled to prevent virtual mem OOM
>> >> errors.
>> >> This of course would not be recommended in production but for single
>> >> node
>> >> deployments you will need this setting.
>> >> echo 1 > /proc/sys/vm/overcommit_memory
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Daniel Lynch
>> >> Mon-Fri 9-5 PST
>> >> Office: 408 780 4498
>> >>
>> >> On Fri, Mar 11, 2016 at 2:01 AM, Gagan Brahmi <gaganbrahmi@gmail.com>
>> >> wrote:
>> >>
>> >>> This a standalone box with no ha for hdfs.
>> >>>
>> >>> I haven't enabled the ha properties in hawq site.
>> >>>
>> >>> Regards,
>> >>> Gagan
>> >>> On Mar 11, 2016 00:56, "Leon Zhang" <leoncamel@gmail.com> wrote:
>> >>>
>> >>> > Hi, Gagang
>> >>> >
>> >>> > It seems you use HA hdfs cluster? I am not sure if HAWQ can work
>> >>> > like
>> >>> > this. Can any HAWQ developer clarify this condition?
>> >>> > If so, you can try a non-HA hdfs cluster with direct IP access.
All
>> >>> > PXF
>> >>> > services are working perfect here.
>> >>> >
>> >>> >
>> >>> > On Fri, Mar 11, 2016 at 10:25 AM, Gagan Brahmi
>> >>> > <gaganbrahmi@gmail.com>
>> >>> > wrote:
>> >>> >
>> >>> >> Thank you Ting!
>> >>> >>
>> >>> >> That was the problem. It seemed to have worked, but now I am
stuck
>> >>> >> with a different error.
>> >>> >>
>> >>> >> gagan=# INSERT into ext_put_foo VALUES (1, 'Gagan');
>> >>> >> ERROR:  failed sending to remote component (libchurl.c:574)
 (seg0
>> >>> >> my-hadoop-cluster:40000 pid=24563) (dispatcher.c:1753)
>> >>> >>
>> >>> >> This certainly mean that the back ground service has stopped
>> >>> >> serving
>> >>> >> connection for some reason.
>> >>> >>
>> >>> >> I check the namenode and find this.
>> >>> >>
>> >>> >> 2016-03-10 19:28:11,759 INFO  hdfs.StateChange
>> >>> >> (FSNamesystem.java:completeFile(3503)) - DIR* completeFile:
>> >>> >> /tmp/foo_bar/1350_0 is closed by
>> >>> >> DFSClient_NONMAPREDUCE_-244490296_23
>> >>> >>
>> >>> >> I have a single node installation with a HDFS replication factor
of
>> >>> >> 1
>> >>> >> (both in hdfs-site and hdfs-client for hawq).
>> >>> >>
>> >>> >> I have also tried to update the connectTimeout value to 60
secs in
>> >>> >> the
>> >>> >> server.xml file for pxf webapp.
>> >>> >>
>> >>> >> A normal write to HDFS works fine. I see file being created
in the
>> >>> >> directory foor_bar but are 0 bytes in size.
>> >>> >>
>> >>> >> -rw-r--r--   1 pxf hdfs          0 2016-03-10 19:08
>> >>> >> /tmp/foo_bar/1336_0
>> >>> >> -rw-r--r--   1 pxf hdfs          0 2016-03-10 19:27
>> >>> >> /tmp/foo_bar/1349_0
>> >>> >> -rw-r--r--   1 pxf hdfs          0 2016-03-10 19:28
>> >>> >> /tmp/foo_bar/1350_0
>> >>> >>
>> >>> >> Not sure if someone has encountered this before. Would appreciate
>> >>> >> any
>> >>> >> inputs.
>> >>> >>
>> >>> >>
>> >>> >> Regards,
>> >>> >> Gagan Brahmi
>> >>> >>
>> >>> >> On Thu, Mar 10, 2016 at 11:45 AM, Ting(Goden) Yao <tyao@pivotal.io>
>> >>> >> wrote:
>> >>> >> > Your table definition:
>> >>> >> >
>> >>> >> > ('pxf://my-hadoop-cluster:*50070*/foo_bar?profile=HdfsTextSimple')
>> >>> >> > if you installed pxf on 51200, you need to use the port
51200
>> >>> >> >
>> >>> >> >
>> >>> >> > On Thu, Mar 10, 2016 at 10:34 AM Gagan Brahmi
>> >>> >> > <gaganbrahmi@gmail.com>
>> >>> >> wrote:
>> >>> >> >
>> >>> >> >> Hi Team,
>> >>> >> >>
>> >>> >> >> I was wondering if someone has encountered this problem
before.
>> >>> >> >>
>> >>> >> >> While trying to work with PXF on hawq 2.0 I am encountering
the
>> >>> >> following
>> >>> >> >> error:
>> >>> >> >>
>> >>> >> >> gagan=# CREATE EXTERNAL TABLE ext_get_foo (i int,
bar text)
>> >>> >> >> LOCATION
>> >>> >> >> ('pxf://my-hadoop-cluster:50070/foo_bar?profile=HdfsTextSimple')
>> >>> >> >> FORMAT 'text' (delimiter '|' null 'null');
>> >>> >> >>
>> >>> >> >> gagan=# SELECT * FROM ext_get_foo ;
>> >>> >> >> ERROR:  remote component error (404): PXF service
could not be
>> >>> >> >> reached. PXF is not running in the tomcat container
>> >>> >> >> (libchurl.c:878)
>> >>> >> >>
>> >>> >> >> The same happens when I try to write to an external
table using
>> >>> >> >> PXF.
>> >>> >> >>
>> >>> >> >> I believe the above error signifies that PXF service
isn't
>> >>> >> >> running or
>> >>> >> >> unavailable. But PXF is running on port 51200.
>> >>> >> >>
>> >>> >> >> Curl response works fine as well:
>> >>> >> >>
>> >>> >> >> # curl -s http://localhost:51200/pxf/v0
>> >>> >> >> Wrong version v0, supported version is v14
>> >>> >> >>
>> >>> >> >> PXF is build using gradlew and installed as RPM files.
I also
>> >>> >> >> have
>> >>> >> >> tomcat 7.0.62 installed with the PXF packages.
>> >>> >> >>
>> >>> >> >> The following is how PXF is running on the instance:
>> >>> >> >>
>> >>> >> >> pxf      21405  0.3  2.8 825224 115164 ?       Sl
  02:07   0:10
>> >>> >> >> /usr/java/latest/bin/java
>> >>> >> >>
>> >>> >> >>
>> >>> >>
>> >>>
>> >>> -Djava.util.logging.config.file=/var/pxf/pxf-service/conf/logging.properties
>> >>> >> >>
>> >>> >> >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> >>> >> >> -Xmx512M -Xss256K
>> >>> >> >> -Djava.endorsed.dirs=/var/pxf/pxf-service/endorsed
>> >>> >> >> -classpath
>> >>> >> >>
>> >>> >>
>> >>>
>> >>> /var/pxf/pxf-service/bin/bootstrap.jar:/var/pxf/pxf-service/bin/tomcat-juli.jar
>> >>> >> >> -Dcatalina.base=/var/pxf/pxf-service
>> >>> >> >> -Dcatalina.home=/var/pxf/pxf-service
>> >>> >> >> -Djava.io.tmpdir=/var/pxf/pxf-service/temp
>> >>> >> >> org.apache.catalina.startup.Bootstrap start
>> >>> >> >>
>> >>> >> >> I do not have apache-tomcat running. Not sure how
are the two
>> >>> >> >> interrelated. But the RPM file created by gradlew
requires
>> >>> >> >> tomcat for
>> >>> >> >> pxf-service.
>> >>> >> >>
>> >>> >> >> I would appreciate any inputs into this problem.
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Regards,
>> >>> >> >> Gagan Brahmi
>> >>> >> >>
>> >>> >>
>> >>> >
>> >>> >
>> >>>

Mime
View raw message