accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keys Botzum <kbot...@maprtech.com>
Subject Re: Accumulo on MapR Continued
Date Sun, 08 Apr 2012 14:01:10 GMT
Billie,

I apologize for taking so long to reply to your message. As soon as I read
it I thought to myself "this is a very good argument." I did some more
research myself and talked with folks at MapR and we agreed that the right
thing to do here is to change MapR to be tolerant of a seek to position
just after the last byte.

I've delayed posting this because I wanted to report the results of the
change and retest. This was delayed for reasons that have nothing to do
with Accumulo or really MapR, I just was too busy.

In any case, we've tried the change in MapR (being tracked as change 6539)
and this does indeed address the issue below. The test case in question now
passes. Thank you VERY much for your help. It is much appreciated.

I am continuing testing and have encountered additional issues but I
haven't yet had time to analyze them. I will post something when I have a
more coherent question. I would just like to confirm that all of the auto
tests do run successfully on stock Apache Hadoop? I ask because at least
one of the tests appears to have a bug in the script that runs it. I just
want to make sure I'm testing the right stuff.

Incidentally, I have upgrade to the official Accumulo 1.4.0 release.


Keys


On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <
billie.j.rinaldi@ugov.gov> wrote:

> It is my understanding that the file length is a valid file position.  If
> you are at position 0 and read one byte, you get the first byte of the
> file.  If you are at position length-1 and read one byte, you get the last
> byte of the file.  At that point you are at position equal to the file
> length, and attempting to read an additional byte will give an EOF
> indicator (for an InputStream read(), this would mean returning -1).
>  Seeking to the file length should just cause a read to give an EOF.
>
> Billie
>
>
> ----- Original Message -----
> > From: "Keys Botzum" <kbotzum@maprtech.com>
> > To: user@accumulo.apache.org
> > Sent: Friday, March 30, 2012 10:12:00 PM
> > Subject: Re: Accumulo on MapR Continued
> > Billie,
> >
> >
> > Thanks for the quick reply. I did some checking and the Hadoop docs
> > indicate that seeking past the end of the file is illegal.
> >
> >
> >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
> > seek
> > public void seek (long desired)
> > throws IOException Description copied from interface: Seekable Seek to
> > the given offset from the start of the file. The next read() will be
> > from that location. Can't seek past the end of the file.
> >
> > Specified by: seek in interface Seekable Throws: IOException
> >
> >
> > The code below that is seeking to the file length is actually seeking
> > one byte past the end of the file since the first byte in a file is
> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
> >
> >
> > Thanks,
> > Keys
> > ________________________________
> > Keys Botzum
> > Senior Principal Technologist
> > WW Systems Engineering
> > kbotzum@maprtech.com
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
> >
> >
> >
> > I ran the following JUnit test and verified that the value printed out
> > was the file length.
> >
> > @Test
> > public void testSeekToEOF() throws IOException {
> > FileSystem fs = FileSystem.get(new Configuration());
> > FSDataInputStream fdi = fs.open(new Path("README"));
> > System.out.println("seeking to end: " + fdi.available());
> > fdi.seek(fdi.available());
> > fdi.close();
> > }
> >
> >
> > ----- Original Message -----
> >
> >
> > From: "Keys Botzum" < kbotzum@maprtech.com >
> >
> >
> > To: user@accumulo.apache.org
> >
> >
> > Sent: Friday, March 30, 2012 4:56:09 PM
> >
> >
> > Subject: Re: Accumulo on MapR Continued
> >
> >
> > Can you share the program? I can try running it and answer your
> >
> >
> > question. I can also of course check with MapR engineering to see if
> >
> >
> > that's the issue.
> >
> >
> >
> >
> >
> >
> >
> >
> > Much appreciated,
> >
> >
> > Keys
> >
> >
> >
> >
> >
> > ________________________________
> >
> >
> > Keys Botzum
> >
> >
> > Senior Principal Technologist
> >
> >
> > WW Systems Engineering
> >
> >
> > kbotzum@maprtech.com
> >
> >
> > 443-718-0098
> >
> >
> > MapR Technologies
> >
> >
> > http://www.mapr.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
> >
> >
> > kbotzum@maprtech.com > wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> >
> >
> >
> >
> >
> >
> >
> >
> > Seeking beyond EOF, filelen 733, wantpos 733
> >
> >
> >
> >
> >
> > It looks like Accumulo is trying to seek to the last position in a
> >
> >
> > file. I just wrote a little program that opens a file in HDFS and
> >
> >
> > seeks to its file length, and it didn't throw an error. Does MapR not
> >
> >
> > allow this? If that is the desired MapR behavior, we could try to
> >
> >
> > determine whether Accumulo could avoid performing that seek.
> >
> >
> >
> >
> >
> > Billie
>



-- 
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com <http://mapr.com>

Mime
View raw message