sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park" <cheol...@cloudera.com>
Subject Re: Review Request: SQOOP-470 Create tests for blob support for Avro import
Date Wed, 04 Apr 2012 02:15:31 GMT


> On 2012-03-29 22:04:43, Bilung Lee wrote:
> > Look good overall.  It would be better if you could also add some to test out the
area of compression.
> 
> Cheolsoo Park wrote:
>     Thank you for reviewing my patch. I will add test cases for compression as per your
request.

Hi Bilung, in terms of compression, I think that there are two aspects that we can test: 1)
Avro compression and 2) LobFile compression.

As for 1, I added two more test cases.

As for 2, even though LobFile supports compression, using LobFile with compression codec requires
quite a few changes to various places. Currently, the LobFile writer that is created by LargeObjectLoader
is constructed with no codec parameter as follows:

  public static Writer create(Path p, Configuration conf, boolean isCharData)
      throws IOException {
    return org.apache.sqoop.io.LobFile.create(p, conf, isCharData, null); <-- null is for
codec.
  }

To support LobFile compression, we will have to extend LargeObjectLoader first. I think that
it is better to defer adding tests for LobFile compression until we update LargeObjectLoader.

Please let me know what you think.


- Cheolsoo


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4527/#review6548
-----------------------------------------------------------


On 2012-04-04 01:55:06, Cheolsoo Park wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4527/
> -----------------------------------------------------------
> 
> (Updated 2012-04-04 01:55:06)
> 
> 
> Review request for Sqoop, Arvind Prabhakar and Bilung Lee.
> 
> 
> Summary
> -------
> 
> Add Avro blob import tests as part of thrid party test suite since Hsqldb does not support
BLOB/CLOB type.
> 
> The changes include:
> 
> - abstract LobAvroImportTestCase class
>   1) testBlobAvroImportInline: test inline import
>   2) testBlobAvroImportExternal: test external file import
>   3) testBlobAvroImportMultiCols: test multiple columns import
> 
> - OracleLobAvroImportTest class
> - MySQLLobAvroImportTest class
> 
> 
> This addresses bug SQOOP-470.
>     https://issues.apache.org/jira/browse/SQOOP-470
> 
> 
> Diffs
> -----
> 
>   /src/test/com/cloudera/sqoop/manager/MySQLLobAvroImportTest.java PRE-CREATION 
>   /src/test/com/cloudera/sqoop/manager/OracleLobAvroImportTest.java PRE-CREATION 
>   /src/test/com/cloudera/sqoop/testutil/LobAvroImportTestCase.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/4527/diff
> 
> 
> Testing
> -------
> 
> Tested with Oracle XE 10g and MySQL 5.1
> 
> Ran ant test
> Ran ant test -Dthirdparty=true
> Ran ant checkstyle
> 
> 
> Thanks,
> 
> Cheolsoo
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message