hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankur (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1229) allow pig to write output into a JDBC db
Date Wed, 04 Aug 2010 06:06:23 GMT

     [ https://issues.apache.org/jira/browse/PIG-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ankur updated PIG-1229:
-----------------------

    Attachment: jira-1229-final.test-fix.patch

Here is my understanding of what happens

1. The main thread in the JVM executing the test initializes MiniDFSCluster,  MiniMRCluster
and HSQLDB server all in different threads.
2. The test setUp() method then executed to create table 'ttt' to which data will be written
by DBStorage() in the test.
3. Pig statements are then executed that spawn M/R job as a separate process that tries to
get a connection to the database and create a preparedStatement for table 'ttt'. This fails
sometimes as DB thread does NOT get a chance to fully persist the table information and the
exception is thrown from the map-tasks as noted by Ashutosh.

The fix for this is to add a 5 sec sleep in setUp() method to give DB a chance to persist
table information. This alleviates the problem and test passes for repeated multiple runs.


Note that Ideal fix would have been to do a busy wait for table creation completion but i
don't see a method in HSqlDB to do that. 

> allow pig to write output into a JDBC db
> ----------------------------------------
>
>                 Key: PIG-1229
>                 URL: https://issues.apache.org/jira/browse/PIG-1229
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>            Reporter: Ian Holsman
>            Assignee: Ankur
>            Priority: Minor
>             Fix For: 0.8.0
>
>         Attachments: jira-1229-final.patch, jira-1229-final.test-fix.patch, jira-1229-v2.patch,
jira-1229-v3.patch, pig-1229.2.patch, pig-1229.patch
>
>
> UDF to store data into a DB

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message