asterixdb-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <dtab...@gmail.com>
Subject Re: Error loading data
Date Thu, 14 Apr 2016 23:38:33 GMT
We should definitely have a "load more" feature so that users can add to 
existing datasets - under the hood it can simply be a big old batched 
insert - but it would be much nicer than having to offer workarounds.  
I've added a JIRA improvement issue to capture this.  (I thought there 
was one, but I didn't find it - might have missed it....)

On 4/14/16 9:34 AM, Ildar Absalyamov wrote:
> Magnus,
>
> You could still add data to non-empty dataset via inserts.
> The easiest way to do that, granted you have the data you want to 
> insert in a files, is to bulkhead data to new temp dataset and insert 
> it to the desired dataset:
>
> create dataset posdata_temp(table) primary key uid auto generated;
> load dataset posdata_temp using localfs 
> "path"="localhost:///data/path/to/file/file.adm,localhost:///data/path/to/file/file2.adm,localhost:///data/path/to/file/file3.adm"),("format"="adm"));
> insert into dataset posdata(
>   for $x in dataset posdata_temp return $x
> )
>
>> On Apr 14, 2016, at 07:41, Magnus Kongshem <kongshem@stud.ntnu.no 
>> <mailto:kongshem@stud.ntnu.no>> wrote:
>>
>> Does this mean that adding additional data to an instance and 
>> dataverse is not supported?
>>
>> Magnus
>>
>> On Wed, Mar 30, 2016 at 8:11 PM, Ian Maxon <imaxon@uci.edu 
>> <mailto:imaxon@uci.edu>> wrote:
>>
>>     It should just be a quoted string with commas inside separating
>>     the URL-ish paths, so like:
>>
>>     load dataset foo using localfs
>>     "path"="localhost:///data/path/to/file/file.adm,localhost:///data/path/to/file/file2.adm,localhost:///data/path/to/file/file3.adm"),("format"="adm"));
>>
>>     On Wed, Mar 30, 2016 at 6:24 AM, Magnus Kongshem
>>     <kongshem@stud.ntnu.no <mailto:kongshem@stud.ntnu.no>> wrote:
>>
>>         Yes I am.
>>
>>         So, combinding each file will and doing the command once will
>>         solve it, or do I have to input the AQL for each file like below?
>>
>>         use dataverse bigd;
>>         load dataset posdata using localfs
>>            
>>         (("path"="localhost:///data/path/to/file/file.adm"),("format"="adm"));
>>         (("path"="localhost:///data/path/to/file/file2.adm"),("format"="adm"));
>>         (("path"="localhost:///data/path/to/file/file3.adm"),("format"="adm"));
>>
>>
>>         BG
>>         Magnus
>>
>>
>>         On Wed, Mar 30, 2016 at 3:21 PM, Wail Alkowaileet
>>         <wael.y.k@gmail.com <mailto:wael.y.k@gmail.com>> wrote:
>>
>>             Are you trying to load each file separately?
>>             That AFAK is not supported.
>>
>>             On Mar 30, 2016 16:13, "Magnus Kongshem"
>>             <kongshem@stud.ntnu.no <mailto:kongshem@stud.ntnu.no>> wrote:
>>
>>                 I will be loading 12 files.
>>
>>                 AQL below:
>>
>>                 use dataverse bigd;
>>                 load dataset posdata using localfs
>>                  
>>                 (("path"="localhost:///data/path/to/file/file.adm"),("format"="adm"));
>>
>>                 Will it be solved if I concatinate the files and do
>>                 the dataset loading only once?
>>
>>                 Magnus
>>
>>                 On Wed, Mar 30, 2016 at 3:06 PM, Wail Alkowaileet
>>                 <wael.y.k@gmail.com <mailto:wael.y.k@gmail.com>> wrote:
>>
>>                     How many files you're loading?
>>                     Can you send the loading AQL?
>>
>>                     On Mar 30, 2016 16:01, "Magnus Kongshem"
>>                     <kongshem@stud.ntnu.no
>>                     <mailto:kongshem@stud.ntnu.no>> wrote:
>>
>>                         Using asterixdb v0.8.8.
>>
>>                         I am loading data into my asterixDB instance.
>>
>>                         Loading the first file is successful. But
>>                         when I try to load another file, I get a
>>                         "Internal error. Please check instance logs
>>                         for further details. [NullPointerException]"
>>
>>                         The files are of the type adm and as good as
>>                         equal in size (3 gb).
>>
>>                         My instance was initialized with these commands:
>>
>>                         drop dataverse bigd if exists;
>>                           create dataverse bigd;
>>                           use dataverse bigd;
>>
>>                           create type table as open {
>>                         uid: uuid,
>>                               campus: string,
>>                         building: string,
>>                         floor: string,
>>                         timestamp: int32,
>>                         dayOfWeek: int32,
>>                         hourOfDay: int32,
>>                         latitude: double,
>>                         salt_timestamp: int32,
>>                         longitude: double,
>>                               id: string,
>>                         accuracy: double
>>                           }
>>                         create dataset posdata(table)
>>                           primary key uid autogenerated;
>>                         create index stamp on posdata(timestamp);
>>                         create index hour on posdata(hourOfDay);
>>                         create index day on posdata(dayOfWeek);
>>
>>                         My log file is attached.
>>
>>                         Any help?
>>
>>                         -- 
>>
>>                         Mvh
>>
>>                         Magnus Kongshem
>>
>>                         NTNU
>>                         +47 415 65 906 <tel:%2B47%20415%2065%20906>
>>
>>
>>
>>
>>                 -- 
>>
>>                 Mvh
>>
>>                 Magnus Alderslyst Kongshem
>>                 Leder av seniorkomiteen
>>                 Online, linjeforeningen for informatikk
>>                 +47 415 65 906 <tel:%2B47%20415%2065%20906>
>>
>>
>>
>>
>>         -- 
>>
>>         Mvh
>>
>>         Magnus Alderslyst Kongshem
>>         Leder av seniorkomiteen
>>         Online, linjeforeningen for informatikk
>>         +47 415 65 906 <tel:%2B47%20415%2065%20906>
>>
>>
>>
>>
>>
>> -- 
>>
>> Mvh
>>
>> Magnus Alderslyst Kongshem
>> Leder av seniorkomiteen
>> Online, linjeforeningen for informatikk
>> +47 415 65 906
>>
>
> Best regards,
> Ildar
>


Mime
View raw message