oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Mattmann" <mattm...@apache.org>
Subject Re: Review Request: NumberFormatException when using RSS service to view transfers for large files
Date Tue, 07 Aug 2012 05:25:57 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6409/#review9946
-----------------------------------------------------------

Ship it!


Ship It!

- Chris Mattmann


On Aug. 7, 2012, 12:38 a.m., Ross Laidlaw wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6409/
> -----------------------------------------------------------
> 
> (Updated Aug. 7, 2012, 12:38 a.m.)
> 
> 
> Review request for oodt, Chris Mattmann and Paul Ramirez.
> 
> 
> Description
> -------
> 
> A NumberFormatException occurs when using the cas-product RSS service to view file transfers
in progress, as detailed by OODT-483.  I traced this exception to two methods in the XmlRpcStructFactory
class, getXmlRpcFileTransferStatus and getFileTransferStatusFromXmlRpc:
> 
> public static Hashtable<String, Object> getXmlRpcFileTransferStatus(FileTransferStatus
status)
> {
>   Hashtable<String, Object> statusHash = new Hashtable<String, Object>();
>   statusHash.put("bytesTransferred", Long.toBinaryString(status.getBytesTransferred()));
>   statusHash.put("parentProduct", getXmlRpcProduct(status.getParentProduct()));
>   statusHash.put("fileRef", getXmlRpcReference(status.getFileRef()));
>   return statusHash;
> }
> 
> public static FileTransferStatus getFileTransferStatusFromXmlRpc(Hashtable<String,
Object> statusHash) 
> {
>   FileTransferStatus status = new FileTransferStatus();
>   status.setBytesTransferred(Long.parseLong(statusHash.get("bytesTransferred").toString()));
>   status.setParentProduct(getProductFromXmlRpc((Hashtable<String, Object>) statusHash.get("parentProduct")));
>   status.setFileRef(getReferenceFromXmlRpc((Hashtable<String, Object>) statusHash.get("fileRef")));
>   return status;
> }
> 
> The following line in getXmlRpcFileTransferStatus converts the value returned by status.getBytesTransferred()
to a binary representation in a string and stores the result in the statusHash Hashtable:
> 
>   statusHash.put("bytesTransferred", Long.toBinaryString(status.getBytesTransferred()));
> 
> 
> The following line in getFileTransferStatusFromXmlRpc retrieves the value for key "bytesTransferred"
from the supplied statusHash Hashtable, but it does not assume that the number is in binary
format and attempts to convert it directly to a long to pass to the setBytesTransferred()
method for the status (type FileTransferStatus) object:
> 
>   status.setBytesTransferred(Long.parseLong(statusHash.get("bytesTransferred").toString()));
> 
> 
> This causes a NumberFormatException for larger files because the parseLong can't fit
the binary representation inside a long.
> 
> 
> Here are two alternative simple fixes:
> 
> 1) in getXmlRpcFileTransferStatus use 'toString' instead of 'toBinaryString' to store
a decimal representation of the number in the string (in statusHash):
> 
>   statusHash.put("bytesTransferred", Long.toString(status.getBytesTransferred()));
> 
> 
> 2) in getFileTransferStatusFromXmlRpc use a radix argument for parseLong to specify that
the "bytesTransferred" string (from statusHash) represents a binary number:
> 
>   status.setBytesTransferred(Long.parseLong(statusHash.get("bytesTransferred").toString(),
2));
> 
> 
> I have uploaded two patches, one for each of the above suggestions.  
> 
> How important is the binary representation?  Is it used to prevent loss of precision?
 Looking back at the commit history for XmlRpcStructFactory, it looks like the 'toBinaryString'
conversion was introduced as a result of OODT-54.  This was part of a bug fix to prevent loss
of precision that happened when casting from longs to integers and then back again.
> 
> The changes can be seen here (see changes for lines 75 and 86): http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java?r1=1096655&r2=1070942&diff_format=h
> 
> 
> This addresses bugs OODT-483 and OODT-54.
>     https://issues.apache.org/jira/browse/OODT-483
>     https://issues.apache.org/jira/browse/OODT-54
> 
> 
> Diffs
> -----
> 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
1365193 
> 
> Diff: https://reviews.apache.org/r/6409/diff/
> 
> 
> Testing
> -------
> 
> I have tested both of the patches by using File Manager to ingest a large file (over
100MB) and then running the following from the command line to access the RSS output from
the RSSProductTransferServlet in the 'cas-product' (fmprod) web application:
> 
> curl "http://localhost:8080/cas-product-0.5-SNAPSHOT/viewTransfers"
> 
> An example output is shown below:
> 
> <?xml version="1.0" encoding="UTF-8" standalone="no"?>
> <rss xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas" version="2.0">
> <channel>
> <title>File Manager Transfers</title>
> <link>http://localhost:8080/cas-product-0.5-SNAPSHOT/viewTransfers</link>
> <description>Current Files Being Transferred to the File Manager</description>
> <language>en-us</language>
> <copyright>Copyright 2010: Apache Software Foundation</copyright>
> <pubDate>Mon, 06 Aug 2012 12:07:41 BST</pubDate>
> <category>data transfer</category>
> <generator>CAS File Manager</generator>
> <lastBuildDate>Mon, 06 Aug 2012 12:07:41 BST</lastBuildDate>
> <item>
> <title>t1.m4v</title>
> <description>GenericFile</description>
> <link>http://localhost:8080/cas-product-0.5-SNAPSHOT/viewTransfer?ref=file:///usr/local/oodt/cas-filemgr-0.5/test/t1.m4v&amp;size=124699632</link>
> <pubDate>Mon, 06 Aug 2012 13:07:32 BST</pubDate>
> <source>file manager transfers</source>
> <cas:source>file manager transfers</cas:source>
> <cas:bytesTransferred>101998592</cas:bytesTransferred>
> <cas:totalBytes>124699632</cas:totalBytes>
> <cas:percentComplete>0.8179542342193921</cas:percentComplete>
> </item>
> </channel>
> </rss>
> 
> 
> Thanks,
> 
> Ross Laidlaw
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message