incubator-accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse McConnell <jesse.mcconn...@gmail.com>
Subject Re: TableOperations import directory
Date Tue, 18 Oct 2011 20:14:24 GMT
I have an idea...it could be that fail that not being on the correct
FS is then making subsequent calls in that BulkImportHelper method not
execute against the correct file system...that would explain why its
not finding the rf files..

validating now

cheers,
jesse

--
jesse mcconnell
jesse.mcconnell@gmail.com



On Tue, Oct 18, 2011 at 15:06, Jesse McConnell
<jesse.mcconnell@gmail.com> wrote:
> Hold the horses...now when I importDirectory from the shell I am
> seeing information in the scan...
>
> so that means the rfile itself is fine...
>
> --
> jesse mcconnell
> jesse.mcconnell@gmail.com
>
>
>
> On Tue, Oct 18, 2011 at 15:00, Jesse McConnell
> <jesse.mcconnell@gmail.com> wrote:
>>>  1) Move files from src dir into an /accumulo/tables/<tid>/bulk_XXX
>>> dir.
>>
>>> So after the bulk import files should be missing from source dir, is this the
case?
>>
>> Yep
>>
>>> Do you see bulk dirs under the /accumulo/tables/<tid> dir, if so do they
contain anything?
>>
>> Yep
>>
>>> The move is done by a random tablet server.  In some tablet server debug log,
>>> should see messages like "Moved <src> to <dest>" coming from the
class
>>> ClientServiceHandler.
>>
>> I have seen this in the debugging as well
>>
>>>  2) Inspect the files to determine first and last key.  I did not see
>>> any client side related debugging for this.
>>
>> I see the first and last key using the rfile.PrintInfo and it looks
>> reasonable, though it does seem to mention only one of the column
>> families in play for the file.
>>
>>>  3) Contact tablet servers to assign files to tablet.  There should
>>> be client and server side debugging related to this. If you set the
>>> log4j level to DEBUG on the client side for org.apache.accumulo.core,
>>> I am thinking you should see the following message.
>>>
>>>           log.debug("Assigning " + uniqMapFiles.size() + " map files to
"
>>>                    + assignmentsPerTablet.size() + " tablets at " +
location);
>>
>> Not seeing this either, again, while debugging here I don't ever see
>> any files being returned in the BulkImportHelper, it always returns an
>> empty list of mapFilesInfo which lead me down the rathole to getGlobs
>> pattern matching [0-9]* files.  Which is what lead us to start looking
>> at your example to see what was fundamentally different. :/
>>
>>> On the server side, you should see the following on a tablet server
>>>
>>>          log.log(TLevel.TABLET_HIST, extent+" import "+path+"
>>> "+paths.get(tpath));
>>>
>>> Could grep for TABLET_HIST and then grep for import to find this.
>>
>> I don't see any 'import' text in the tserver_host.debug.log like this...
>>
>>>  4) Anything that fails, is copied to the failure dir you supplied.
>>> Is the failure dir empty after the import?
>>
>> Empty, but I should double check, I get errors if this failure
>> directory is on the same hdfs as the input to the importDirectory, it
>> errors and says its expecting file:///  But when I just pass in a new
>> Path("FAIL-" + uuid) relative path it runs.
>>
>> thanks for the help
>>
>> cheers,
>> jesse
>>
>

Mime
View raw message