oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ross Laidlaw" <rlaidlaw.o...@gmail.com>
Subject Re: Review Request: NumberFormatException when using RSS service to view transfers for large files
Date Tue, 07 Aug 2012 00:38:20 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6409/
-----------------------------------------------------------

(Updated Aug. 7, 2012, 12:38 a.m.)


Review request for oodt, Chris Mattmann and Paul Ramirez.


Description (updated)
-------

A NumberFormatException occurs when using the cas-product RSS service to view file transfers
in progress, as detailed by OODT-483.  I traced this exception to two methods in the XmlRpcStructFactory
class, getXmlRpcFileTransferStatus and getFileTransferStatusFromXmlRpc:

public static Hashtable<String, Object> getXmlRpcFileTransferStatus(FileTransferStatus
status)
{
  Hashtable<String, Object> statusHash = new Hashtable<String, Object>();
  statusHash.put("bytesTransferred", Long.toBinaryString(status.getBytesTransferred()));
  statusHash.put("parentProduct", getXmlRpcProduct(status.getParentProduct()));
  statusHash.put("fileRef", getXmlRpcReference(status.getFileRef()));
  return statusHash;
}

public static FileTransferStatus getFileTransferStatusFromXmlRpc(Hashtable<String, Object>
statusHash) 
{
  FileTransferStatus status = new FileTransferStatus();
  status.setBytesTransferred(Long.parseLong(statusHash.get("bytesTransferred").toString()));
  status.setParentProduct(getProductFromXmlRpc((Hashtable<String, Object>) statusHash.get("parentProduct")));
  status.setFileRef(getReferenceFromXmlRpc((Hashtable<String, Object>) statusHash.get("fileRef")));
  return status;
}

The following line in getXmlRpcFileTransferStatus converts the value returned by status.getBytesTransferred()
to a binary representation in a string and stores the result in the statusHash Hashtable:

  statusHash.put("bytesTransferred", Long.toBinaryString(status.getBytesTransferred()));


The following line in getFileTransferStatusFromXmlRpc retrieves the value for key "bytesTransferred"
from the supplied statusHash Hashtable, but it does not assume that the number is in binary
format and attempts to convert it directly to a long to pass to the setBytesTransferred()
method for the status (type FileTransferStatus) object:

  status.setBytesTransferred(Long.parseLong(statusHash.get("bytesTransferred").toString()));


This causes a NumberFormatException for larger files because the parseLong can't fit the binary
representation inside a long.


Here are two alternative simple fixes:

1) in getXmlRpcFileTransferStatus use 'toString' instead of 'toBinaryString' to store a decimal
representation of the number in the string (in statusHash):

  statusHash.put("bytesTransferred", Long.toString(status.getBytesTransferred()));


2) in getFileTransferStatusFromXmlRpc use a radix argument for parseLong to specify that the
"bytesTransferred" string (from statusHash) represents a binary number:

  status.setBytesTransferred(Long.parseLong(statusHash.get("bytesTransferred").toString(),
2));


I have uploaded two patches, one for each of the above suggestions.  

How important is the binary representation?  Is it used to prevent loss of precision?  Looking
back at the commit history for XmlRpcStructFactory, it looks like the 'toBinaryString' conversion
was introduced as a result of OODT-54.  This was part of a bug fix to prevent loss of precision
that happened when casting from longs to integers and then back again.

The changes can be seen here (see changes for lines 75 and 86): http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java?r1=1096655&r2=1070942&diff_format=h


This addresses bugs OODT-483 and OODT-54.
    https://issues.apache.org/jira/browse/OODT-483
    https://issues.apache.org/jira/browse/OODT-54


Diffs
-----

  trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java 1365193


Diff: https://reviews.apache.org/r/6409/diff/


Testing
-------

I have tested both of the patches by using File Manager to ingest a large file (over 100MB)
and then running the following from the command line to access the RSS output from the RSSProductTransferServlet
in the 'cas-product' (fmprod) web application:

curl "http://localhost:8080/cas-product-0.5-SNAPSHOT/viewTransfers"

An example output is shown below:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<rss xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas" version="2.0">
<channel>
<title>File Manager Transfers</title>
<link>http://localhost:8080/cas-product-0.5-SNAPSHOT/viewTransfers</link>
<description>Current Files Being Transferred to the File Manager</description>
<language>en-us</language>
<copyright>Copyright 2010: Apache Software Foundation</copyright>
<pubDate>Mon, 06 Aug 2012 12:07:41 BST</pubDate>
<category>data transfer</category>
<generator>CAS File Manager</generator>
<lastBuildDate>Mon, 06 Aug 2012 12:07:41 BST</lastBuildDate>
<item>
<title>t1.m4v</title>
<description>GenericFile</description>
<link>http://localhost:8080/cas-product-0.5-SNAPSHOT/viewTransfer?ref=file:///usr/local/oodt/cas-filemgr-0.5/test/t1.m4v&amp;size=124699632</link>
<pubDate>Mon, 06 Aug 2012 13:07:32 BST</pubDate>
<source>file manager transfers</source>
<cas:source>file manager transfers</cas:source>
<cas:bytesTransferred>101998592</cas:bytesTransferred>
<cas:totalBytes>124699632</cas:totalBytes>
<cas:percentComplete>0.8179542342193921</cas:percentComplete>
</item>
</channel>
</rss>


Thanks,

Ross Laidlaw


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message