accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Newton <eric.new...@gmail.com>
Subject Re: Accumulo on MapR Continued
Date Sun, 08 Apr 2012 14:23:18 GMT
The auto tests are a bit flakey, especially when they rely on local
conditions, like the speed at which my workstation will run something.  If
you have found a bug in one of the scripts, please make a Jira ticket.

Right now several of the tests assume they have superuser rights to
scribble all over the distributed file system, see ACCUMULO-355.

-Eric

On Sun, Apr 8, 2012 at 10:01 AM, Keys Botzum <kbotzum@maprtech.com> wrote:

> Billie,
>
> I apologize for taking so long to reply to your message. As soon as I read
> it I thought to myself "this is a very good argument." I did some more
> research myself and talked with folks at MapR and we agreed that the right
> thing to do here is to change MapR to be tolerant of a seek to position
> just after the last byte.
>
> I've delayed posting this because I wanted to report the results of the
> change and retest. This was delayed for reasons that have nothing to do
> with Accumulo or really MapR, I just was too busy.
>
> In any case, we've tried the change in MapR (being tracked as change 6539)
> and this does indeed address the issue below. The test case in question now
> passes. Thank you VERY much for your help. It is much appreciated.
>
> I am continuing testing and have encountered additional issues but I
> haven't yet had time to analyze them. I will post something when I have a
> more coherent question. I would just like to confirm that all of the auto
> tests do run successfully on stock Apache Hadoop? I ask because at least
> one of the tests appears to have a bug in the script that runs it. I just
> want to make sure I'm testing the right stuff.
>
> Incidentally, I have upgrade to the official Accumulo 1.4.0 release.
>
>
> Keys
>
>
> On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <
> billie.j.rinaldi@ugov.gov> wrote:
>
>> It is my understanding that the file length is a valid file position.  If
>> you are at position 0 and read one byte, you get the first byte of the
>> file.  If you are at position length-1 and read one byte, you get the last
>> byte of the file.  At that point you are at position equal to the file
>> length, and attempting to read an additional byte will give an EOF
>> indicator (for an InputStream read(), this would mean returning -1).
>>  Seeking to the file length should just cause a read to give an EOF.
>>
>> Billie
>>
>>
>> ----- Original Message -----
>> > From: "Keys Botzum" <kbotzum@maprtech.com>
>> > To: user@accumulo.apache.org
>> > Sent: Friday, March 30, 2012 10:12:00 PM
>> > Subject: Re: Accumulo on MapR Continued
>> > Billie,
>> >
>> >
>> > Thanks for the quick reply. I did some checking and the Hadoop docs
>> > indicate that seeking past the end of the file is illegal.
>> >
>> >
>> >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
>> > seek
>> > public void seek (long desired)
>> > throws IOException Description copied from interface: Seekable Seek to
>> > the given offset from the start of the file. The next read() will be
>> > from that location. Can't seek past the end of the file.
>> >
>> > Specified by: seek in interface Seekable Throws: IOException
>> >
>> >
>> > The code below that is seeking to the file length is actually seeking
>> > one byte past the end of the file since the first byte in a file is
>> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
>> >
>> >
>> > Thanks,
>> > Keys
>> > ________________________________
>> > Keys Botzum
>> > Senior Principal Technologist
>> > WW Systems Engineering
>> > kbotzum@maprtech.com
>> > 443-718-0098
>> > MapR Technologies
>> > http://www.mapr.com
>> >
>> >
>> >
>> >
>> >
>> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
>> >
>> >
>> >
>> > I ran the following JUnit test and verified that the value printed out
>> > was the file length.
>> >
>> > @Test
>> > public void testSeekToEOF() throws IOException {
>> > FileSystem fs = FileSystem.get(new Configuration());
>> > FSDataInputStream fdi = fs.open(new Path("README"));
>> > System.out.println("seeking to end: " + fdi.available());
>> > fdi.seek(fdi.available());
>> > fdi.close();
>> > }
>> >
>> >
>> > ----- Original Message -----
>> >
>> >
>> > From: "Keys Botzum" < kbotzum@maprtech.com >
>> >
>> >
>> > To: user@accumulo.apache.org
>> >
>> >
>> > Sent: Friday, March 30, 2012 4:56:09 PM
>> >
>> >
>> > Subject: Re: Accumulo on MapR Continued
>> >
>> >
>> > Can you share the program? I can try running it and answer your
>> >
>> >
>> > question. I can also of course check with MapR engineering to see if
>> >
>> >
>> > that's the issue.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Much appreciated,
>> >
>> >
>> > Keys
>> >
>> >
>> >
>> >
>> >
>> > ________________________________
>> >
>> >
>> > Keys Botzum
>> >
>> >
>> > Senior Principal Technologist
>> >
>> >
>> > WW Systems Engineering
>> >
>> >
>> > kbotzum@maprtech.com
>> >
>> >
>> > 443-718-0098
>> >
>> >
>> > MapR Technologies
>> >
>> >
>> > http://www.mapr.com
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
>> >
>> >
>> > kbotzum@maprtech.com > wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Seeking beyond EOF, filelen 733, wantpos 733
>> >
>> >
>> >
>> >
>> >
>> > It looks like Accumulo is trying to seek to the last position in a
>> >
>> >
>> > file. I just wrote a little program that opens a file in HDFS and
>> >
>> >
>> > seeks to its file length, and it didn't throw an error. Does MapR not
>> >
>> >
>> > allow this? If that is the desired MapR behavior, we could try to
>> >
>> >
>> > determine whether Accumulo could avoid performing that seek.
>> >
>> >
>> >
>> >
>> >
>> > Billie
>>
>
>
>
> --
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com <http://mapr.com>
>
>

Mime
View raw message