oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Bennett <lmzxq....@gmail.com>
Subject Re: How can I update a filemanager's catalogue?
Date Tue, 17 Jul 2012 12:57:36 GMT
Hi everyone,

Thanks for the responses. I always appreciate the fact that post a
question/problem on the forum and the next morning there are stack of
replies.

+1 to everyone!

As it turns out the problem was entirely mine :-)

However, I've discovered how the file manager works out the percentage
file transferred and why it thinks the file Percentage Complete was 0%.

The magic all happens with the following piece of code:

    private long getBytesTransferred(Reference r) {
        File destFile = null;
        try {
            destFile = new File(new URI(r.getDataStoreReference()));
           * long bytesTransferred = destFile.length();*
            return bytesTransferred;
        } catch (URISyntaxException e) {
            LOG.log(Level.WARNING,
                    "URISyntaxException when checking size of destFile: ["
                            + r.getDataStoreReference() + "]: Message: "
                            + e.getMessage());
            return -1L;
        }
    }

So the file transfer size is worked out on the fly - its not stored in the
catalog.

The origin of the weirdness relates to my Versioner.

My versioner stores files to
file:/var/kat/archive/data/comm/[StartDate]/1234567890.h5

My original code (pre-OODT) stored the data in the same way. However, for
OODT I store my dates in UTC, and thus there is a 2 hour difference between
my preOODT and OODT implementations, so for my product 1287353769.h5 the
references are:

origReference "file:/var/kat/archive/data/comm/2010/10/*18*/1287353769.h5"
(id=115)
dataStoreReference "file:/var/kat/archive/data/comm/2010/10/*17*/1287353769.h5"
(id=112)

Since I used the InPlaceDataTransfer the data reference was updated but the
file was not moved.

So files captured between midnight and 2am will now have a new reference to
the time shifted directory (e.g. moved from the 18'th to the 17'th)

When the %complete is calculated it uses the dataStoreReference.

Is it possible for the file manager to know that the InPlaceDataTransfer
has been used and that the reference should not change? I guess this would
not be possible.

Cheers,
Tom

On 17 July 2012 05:29, Sheryl John <sheryljj@gmail.com> wrote:

> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lmzxq.tom@gmail.com> wrote:
>> > Hi,
>> >
>> > I've recently created a Lucene file manager catalog. I cataloged about
>> 2000 files "in place", but have encountered a problem.
>> >
>> > From the fmbrowser some of the files are listed as being:
>> >
>> > Product               Transfer Status     Percent Complete    Received
>> Time   References          Metadata
>> > 1287587805.h5 RECEIVED            100%                UNKNOWN
>> View References     View Metadata
>> > 1287353769.h5 RECEIVED            0%                  UNKNOWN
>> View References     View Metadata
>> > 1287355996.h5 RECEIVED            0%                  UNKNOWN
>> View References     View Metadata
>> > 1287425397.h5 RECEIVED            100%                UNKNOWN
>> View References     View Metadata
>> >
>> > The problem lies with the two products that show % complete being 0%.
>> >
>> > When I use fmprod to download the product:
>> >
>> > wget http://
>> ..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>> >
>> > I get:
>> > HTTP request sent, awaiting response... 500 Internal Server Error
>> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>> >
>> > So I'd like to "edit" the lucene catalog to update this piece of
>> metadata to reflect that the % complete is 100. Are there any tools
>> available to do this?
>> >
>> > Also at some point I must figure out why its happening :)
>> >
>> > Cheers,
>> > Tom
>>
>

Mime
View raw message