accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keys Botzum <kbot...@maprtech.com>
Subject Re: Accumulo on MapR Continued
Date Mon, 09 Apr 2012 00:43:36 GMT
Eric,

Thanks for that hint. Do you have a sense (or is this written somewhere) as to what tests
may not work? I am running the tests as a user with very high permissions so I think they
can write anywhere in HDFS, but I can double check that easily enough.

If there are tests that are known not to work, I can avoid those tests or at least not post
so much to the mailing list about such tests.

If you don't know, I'll just carry on and post what I learn. Obvious script bugs I can easily
fix. Other more subtle issues require more investigation.


Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 8, 2012, at 10:23 AM, Eric Newton wrote:

> The auto tests are a bit flakey, especially when they rely on local conditions, like
the speed at which my workstation will run something.  If you have found a bug in one of the
scripts, please make a Jira ticket.
> 
> Right now several of the tests assume they have superuser rights to scribble all over
the distributed file system, see ACCUMULO-355.
> 
> -Eric
> 
> On Sun, Apr 8, 2012 at 10:01 AM, Keys Botzum <kbotzum@maprtech.com> wrote:
> Billie,
> 
> I apologize for taking so long to reply to your message. As soon as I read it I thought
to myself "this is a very good argument." I did some more research myself and talked with
folks at MapR and we agreed that the right thing to do here is to change MapR to be tolerant
of a seek to position just after the last byte. 
> 
> I've delayed posting this because I wanted to report the results of the change and retest.
This was delayed for reasons that have nothing to do with Accumulo or really MapR, I just
was too busy.
> 
> In any case, we've tried the change in MapR (being tracked as change 6539) and this does
indeed address the issue below. The test case in question now passes. Thank you VERY much
for your help. It is much appreciated.
> 
> I am continuing testing and have encountered additional issues but I haven't yet had
time to analyze them. I will post something when I have a more coherent question. I would
just like to confirm that all of the auto tests do run successfully on stock Apache Hadoop?
I ask because at least one of the tests appears to have a bug in the script that runs it.
I just want to make sure I'm testing the right stuff.
> 
> Incidentally, I have upgrade to the official Accumulo 1.4.0 release.
> 
> 
> Keys
> 
> 
> On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <billie.j.rinaldi@ugov.gov>
wrote:
> It is my understanding that the file length is a valid file position.  If you are at
position 0 and read one byte, you get the first byte of the file.  If you are at position
length-1 and read one byte, you get the last byte of the file.  At that point you are at position
equal to the file length, and attempting to read an additional byte will give an EOF indicator
(for an InputStream read(), this would mean returning -1).  Seeking to the file length should
just cause a read to give an EOF.
> 
> Billie
> 
> 
> ----- Original Message -----
> > From: "Keys Botzum" <kbotzum@maprtech.com>
> > To: user@accumulo.apache.org
> > Sent: Friday, March 30, 2012 10:12:00 PM
> > Subject: Re: Accumulo on MapR Continued
> > Billie,
> >
> >
> > Thanks for the quick reply. I did some checking and the Hadoop docs
> > indicate that seeking past the end of the file is illegal.
> >
> >
> >
> > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
> > seek
> > public void seek (long desired)
> > throws IOException Description copied from interface: Seekable Seek to
> > the given offset from the start of the file. The next read() will be
> > from that location. Can't seek past the end of the file.
> >
> > Specified by: seek in interface Seekable Throws: IOException
> >
> >
> > The code below that is seeking to the file length is actually seeking
> > one byte past the end of the file since the first byte in a file is
> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
> >
> >
> > Thanks,
> > Keys
> > ________________________________
> > Keys Botzum
> > Senior Principal Technologist
> > WW Systems Engineering
> > kbotzum@maprtech.com
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
> >
> >
> >
> > I ran the following JUnit test and verified that the value printed out
> > was the file length.
> >
> > @Test
> > public void testSeekToEOF() throws IOException {
> > FileSystem fs = FileSystem.get(new Configuration());
> > FSDataInputStream fdi = fs.open(new Path("README"));
> > System.out.println("seeking to end: " + fdi.available());
> > fdi.seek(fdi.available());
> > fdi.close();
> > }
> >
> >
> > ----- Original Message -----
> >
> >
> > From: "Keys Botzum" < kbotzum@maprtech.com >
> >
> >
> > To: user@accumulo.apache.org
> >
> >
> > Sent: Friday, March 30, 2012 4:56:09 PM
> >
> >
> > Subject: Re: Accumulo on MapR Continued
> >
> >
> > Can you share the program? I can try running it and answer your
> >
> >
> > question. I can also of course check with MapR engineering to see if
> >
> >
> > that's the issue.
> >
> >
> >
> >
> >
> >
> >
> >
> > Much appreciated,
> >
> >
> > Keys
> >
> >
> >
> >
> >
> > ________________________________
> >
> >
> > Keys Botzum
> >
> >
> > Senior Principal Technologist
> >
> >
> > WW Systems Engineering
> >
> >
> > kbotzum@maprtech.com
> >
> >
> > 443-718-0098
> >
> >
> > MapR Technologies
> >
> >
> > http://www.mapr.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
> >
> >
> > kbotzum@maprtech.com > wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> >
> >
> >
> >
> >
> >
> >
> >
> > Seeking beyond EOF, filelen 733, wantpos 733
> >
> >
> >
> >
> >
> > It looks like Accumulo is trying to seek to the last position in a
> >
> >
> > file. I just wrote a little program that opens a file in HDFS and
> >
> >
> > seeks to its file length, and it didn't throw an error. Does MapR not
> >
> >
> > allow this? If that is the desired MapR behavior, we could try to
> >
> >
> > determine whether Accumulo could avoid performing that seek.
> >
> >
> >
> >
> >
> > Billie
> 
> 
> 
> -- 
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 


Mime
View raw message