Return-Path: X-Original-To: apmail-asterixdb-users-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2EFAA19536 for ; Tue, 19 Apr 2016 17:11:41 +0000 (UTC) Received: (qmail 15867 invoked by uid 500); 19 Apr 2016 17:11:40 -0000 Delivered-To: apmail-asterixdb-users-archive@asterixdb.apache.org Received: (qmail 15838 invoked by uid 500); 19 Apr 2016 17:11:40 -0000 Mailing-List: contact users-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@asterixdb.incubator.apache.org Delivered-To: mailing list users@asterixdb.incubator.apache.org Received: (qmail 15828 invoked by uid 99); 19 Apr 2016 17:11:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2016 17:11:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B638E180591 for ; Tue, 19 Apr 2016 17:11:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.198 X-Spam-Level: ** X-Spam-Status: No, score=2.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id M4-vCAPg0PjL for ; Tue, 19 Apr 2016 17:11:37 +0000 (UTC) Received: from mail-pf0-f176.google.com (mail-pf0-f176.google.com [209.85.192.176]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 95D415FAF2 for ; Tue, 19 Apr 2016 17:11:36 +0000 (UTC) Received: by mail-pf0-f176.google.com with SMTP id 184so8761427pff.0 for ; Tue, 19 Apr 2016 10:11:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:message-id:mime-version:subject:date:references:to :in-reply-to; bh=LT469pba5Enl3CAEuzgoayt7RkNwL1COWX7sCphTs/A=; b=HQ5ePg07My0hdCuv9dyJi+2PgOi5Zr9MTgs6ooyWKeBakYw3001vGuXxEIKgplj16V 4XDMjJrkiqW4QudKVLlv56ECSDHpk+7hPbBZk7V5CG2iWuJVQ2FZW7xE5nXSHSi1APDI KPIzZbVNXm1yx1Vt/Dh+nSNbdEpJaEsYdHKZAIFx5ezBlO6HkS4RvHXKlpDmfmrzCdky 2NJc/zvcVpLFZpfrFbQLD7wsR66eKxfvlUzZX3SXpEDwn/1ayWZerTpUIniQhO2ORIpq ItoC96wiZbXHVYO4nSlXXN87yixLwnw3I16eicr9xQrnsyErYGAE6hY5UR4S1KB3y5Qj lq4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:message-id:mime-version:subject:date :references:to:in-reply-to; bh=LT469pba5Enl3CAEuzgoayt7RkNwL1COWX7sCphTs/A=; b=OdvQTdamHKNrkRjB82Q200EwKUQMf+QKCCWhPJf1ZrFPjhyO5ZUCoBEeHZRxMGjn3u fPi+r5hH9RSLCWqraRKufJmSGBZYs6dSbDVtLm0b+jJ4GUZKOrv0BYZKATy8dOmsljx9 oBLQcQatCLjwBFC2m0wAmBm8gRDc3vpKH9Xn0M5TaOYk1eftIM5xD2N4v+XWK1Psph4D zzY9jQgEcXojy89RcOUKjJEC0ktwg9+lYfubtqPtryEJrZ4KWUUPmHNYvqrApmbbHhll uCecsWRhThmTI2B7NgT4+eP9Zm1Q2/AvILnHFJThh7j+DNZJNdeuP+BYXWd0k/QEa0g3 PnTA== X-Gm-Message-State: AOPr4FX3uurNoYEj4MdpiCGtaPflnjGXGgNmzOCACsDdoAviXz2C5RI0xfu+fQm12XUtUg== X-Received: by 10.98.92.71 with SMTP id q68mr5664774pfb.70.1461085889078; Tue, 19 Apr 2016 10:11:29 -0700 (PDT) Received: from [192.168.1.149] (66-215-226-0.dhcp.rvsd.ca.charter.com. [66.215.226.0]) by smtp.gmail.com with ESMTPSA id t85sm7076812pfi.55.2016.04.19.10.11.27 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 19 Apr 2016 10:11:28 -0700 (PDT) Sender: Ildar Absalyamov From: Ildar Absalyamov Content-Type: multipart/alternative; boundary="Apple-Mail=_E9429199-A90A-4C3F-BC07-47A497B614A5" Message-Id: <32458841-9CB6-42DD-A467-D760CBC3BB69@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Error loading data Date: Tue, 19 Apr 2016 10:11:26 -0700 References: <93CD8F2B-BCE2-48D8-A237-46BE15FF3288@gmail.com> To: users@asterixdb.incubator.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3112) --Apple-Mail=_E9429199-A90A-4C3F-BC07-47A497B614A5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Magnus, Since you are using autogenerated key the inserted record should not = have that field. To do that you need to change the record format in the = return clause: insert into dataset posdata( for $x in dataset posdata_temp return { "campus": $x.campus, "building": $x.building, "floor": $x.floor, "timestamp": $x.timestamp, "dayOfWeek": $x.day, "hourOfDay": $x.hour, "latitude": $x.latitude, "salt_timestamp": $x.salt, "longitude": $x.longitude, "id": $x.id, "accuracy": $x.accuracy } ) > On Apr 19, 2016, at 04:54, Magnus Kongshem = wrote: >=20 > Your suggestion does not work because you get duplicate fields.=20 >=20 > Exception: Duplicate field "uuid" encountered = [AlgebricksException] >=20 > Any other suggestions? This is a major issue in my view, and as Mike = Carey said: It should be easy and seamless to add more data to the = dataset. >=20 > BG, > Magnus >=20 > On Thu, Apr 14, 2016 at 6:34 PM, Ildar Absalyamov = > wrote: > Magnus, >=20 > You could still add data to non-empty dataset via inserts. > The easiest way to do that, granted you have the data you want to = insert in a files, is to bulkhead data to new temp dataset and insert it = to the desired dataset: >=20 > create dataset posdata_temp(table) primary key uid auto generated; > load dataset posdata_temp using localfs = "path"=3D"localhost:///data/path/to/file/file.adm,localhost:///data/path/t= o/file/file2.adm,localhost:///data/path/to/file/file3.adm = <>"),("format"=3D"adm")); > insert into dataset posdata( > for $x in dataset posdata_temp return $x > ) >=20 >> On Apr 14, 2016, at 07:41, Magnus Kongshem > wrote: >>=20 >> Does this mean that adding additional data to an instance and = dataverse is not supported? >>=20 >> Magnus >>=20 >> On Wed, Mar 30, 2016 at 8:11 PM, Ian Maxon > wrote: >> It should just be a quoted string with commas inside separating the = URL-ish paths, so like: >>=20 >> load dataset foo using localfs = "path"=3D"localhost:///data/path/to/file/file.adm,localhost:///data/path/t= o/file/file2.adm,localhost:///data/path/to/file/file3.adm = <>"),("format"=3D"adm")); >>=20 >> On Wed, Mar 30, 2016 at 6:24 AM, Magnus Kongshem = > wrote: >> Yes I am. >>=20 >> So, combinding each file will and doing the command once will solve = it, or do I have to input the AQL for each file like below? >>=20 >> use dataverse bigd; >> load dataset posdata using localfs >> (("path"=3D"localhost:///data/path/to/file/file.adm = <>"),("format"=3D"adm")); >> (("path"=3D"localhost:///data/path/to/file/file2.adm = <>"),("format"=3D"adm")); >> (("path"=3D"localhost:///data/path/to/file/file3.adm = <>"),("format"=3D"adm")); >>=20 >>=20 >> BG >> Magnus >>=20 >>=20 >> On Wed, Mar 30, 2016 at 3:21 PM, Wail Alkowaileet > wrote: >> Are you trying to load each file separately? >> That AFAK is not supported. >> On Mar 30, 2016 16:13, "Magnus Kongshem" > wrote: >> I will be loading 12 files. >>=20 >> AQL below: >>=20 >> use dataverse bigd; >> load dataset posdata using localfs >> (("path"=3D"localhost:///data/path/to/file/file.adm = <>"),("format"=3D"adm")); >>=20 >> Will it be solved if I concatinate the files and do the dataset = loading only once? >>=20 >> Magnus >>=20 >> On Wed, Mar 30, 2016 at 3:06 PM, Wail Alkowaileet > wrote: >> How many files you're loading? >> Can you send the loading AQL? >>=20 >> On Mar 30, 2016 16:01, "Magnus Kongshem" > wrote: >> Using asterixdb v0.8.8. >>=20 >> I am loading data into my asterixDB instance. >>=20 >> Loading the first file is successful. But when I try to load another = file, I get a "Internal error. Please check instance logs for further = details. [NullPointerException]" >>=20 >> The files are of the type adm and as good as equal in size (3 gb). >>=20 >> My instance was initialized with these commands: >>=20 >> drop dataverse bigd if exists; >> create dataverse bigd; >> use dataverse bigd; >>=20 >> create type table as open { >> uid: uuid, >> campus: string, >> building: string, >> floor: string, >> timestamp: int32, >> dayOfWeek: int32, >> hourOfDay: int32, >> latitude: double, >> salt_timestamp: int32, >> longitude: double, >> id: string, >> accuracy: double >> } >> create dataset posdata(table) >> primary key uid autogenerated; >> create index stamp on posdata(timestamp); >> create index hour on posdata(hourOfDay); >> create index day on posdata(dayOfWeek); >>=20 >> My log file is attached. >>=20 >> Any help? >>=20 >> --=20 >> Mvh >>=20 >> Magnus Kongshem >>=20 >> NTNU >> +47 415 65 906 >>=20 >>=20 >> --=20 >> Mvh >>=20 >> Magnus Alderslyst Kongshem >> Leder av seniorkomiteen >> Online, linjeforeningen for informatikk >> +47 415 65 906 >>=20 >>=20 >> --=20 >> Mvh >>=20 >> Magnus Alderslyst Kongshem >> Leder av seniorkomiteen >> Online, linjeforeningen for informatikk >> +47 415 65 906 >>=20 >>=20 >>=20 >> --=20 >> Mvh >>=20 >> Magnus Alderslyst Kongshem >> Leder av seniorkomiteen >> Online, linjeforeningen for informatikk >> +47 415 65 906 > Best regards, > Ildar >=20 >=20 >=20 >=20 > --=20 > Mvh >=20 > Magnus Alderslyst Kongshem > Leder av seniorkomiteen > Online, linjeforeningen for informatikk > +47 415 65 906 Best regards, Ildar --Apple-Mail=_E9429199-A90A-4C3F-BC07-47A497B614A5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Magnus,

Since you are using autogenerated key = the inserted record should not have that field. To do that you need to = change the record format in the return clause:

insert into dataset = posdata(
  for $x in dataset = posdata_temp return {
"campus": = $x.campus,
"building": $x.building,
"floor": $x.floor,
"timestamp": = $x.timestamp,
"dayOfWeek": $x.day,
"hourOfDay": $x.hour,
"latitude": = $x.latitude,
"salt_timestamp": $x.salt,
"longitude": $x.longitude,
"id": = $x.id,
"accuracy": $x.accuracy
}
)

On = Apr 19, 2016, at 04:54, Magnus Kongshem <kongshem@stud.ntnu.no> wrote:

Your suggestion does not work because you get duplicate = fields. 

Exception: Duplicate field &quot;uuid&quot; encountered = [AlgebricksException]

Any other suggestions? This is a major = issue in my view, and as Mike Carey said: It should be easy and seamless = to add more data to the dataset.

BG,
Magnus

On Thu, Apr 14, 2016 at 6:34 PM, Ildar Absalyamov = <ildar.absalyamov@gmail.com> wrote:
Magnus,

You could still add data to non-empty = dataset via inserts.
The easiest way to do that, = granted you have the data you want to insert in a files, is to bulkhead = data to new temp dataset and insert it to the desired dataset:

create dataset = posdata_temp(table) primary key uid auto generated;
load dataset posdata_temp using = localfs "path"=3D"localhost:///data/path/to/file/file.adm,localhost:///data/path/= to/file/file2.adm,localhost:///data/path/to/file/file3.adm"),("format"= =3D"adm"));
insert into dataset posdata(
  for $x in dataset posdata_temp return = $x
)

On Apr 14, 2016, at 07:41, = Magnus Kongshem <kongshem@stud.ntnu.no> = wrote:

Does this mean that adding additional data to an instance and = dataverse is not supported?

Magnus

On Wed, Mar 30, 2016 at 8:11 PM, = Ian Maxon <imaxon@uci.edu> wrote:
It should = just be a quoted string with commas inside separating the URL-ish paths, = so like:

load = dataset foo using localfs "path"=3D"localhost:///data/path/to/file/file.adm,localhost:///data/path/= to/file/file2.adm,localhost:///data/path/to/file/file3.adm"),("format"= =3D"adm"));

On Wed, = Mar 30, 2016 at 6:24 AM, Magnus Kongshem <kongshem@stud.ntnu.no> wrote:
Yes I am.

So, combinding each file = will and doing the command once will solve it, or do I have to input the = AQL for each file like below?

use dataverse bigd;
load dataset posdata using localfs
    (("path"=3D"localhost:///data/path/to/file/file.adm"),("format"=3D"adm"= ));
(("path"=3D"localhost:///data/path/to/file/file2.adm"),("format"=3D"adm= "));
(("path"=3D"localhost:///data/path/to/file/file3.adm"),("format"=3D"adm= "));


BG
Magnus


On Wed, = Mar 30, 2016 at 3:21 PM, Wail Alkowaileet <wael.y.k@gmail.com> wrote:

Are you trying = to load each file separately?
That AFAK is not supported.

On Mar 30, 2016 16:13, "Magnus Kongshem" = <kongshem@stud.ntnu.no> wrote:
I will be = loading 12 files.

AQL = below:

use dataverse bigd;
load dataset posdata = using localfs
    (("path"=3D"localhost:///data/path/to/file/file.adm"),("format"=3D"adm"= ));

Will it be = solved if I concatinate the files and do the dataset loading only = once?

Magnus

On Wed, Mar 30, 2016 at 3:06 PM, Wail Alkowaileet = <wael.y.k@gmail.com> wrote:

How many files = you're loading?
Can you send the loading AQL?

On Mar 30, 2016 16:01, "Magnus Kongshem" = <kongshem@stud.ntnu.no> wrote:
Using = asterixdb v0.8.8.

I = am loading data into my asterixDB instance.

Loading the first file is successful. = But when I try to load another file, I get a "Internal error. Please check instance logs for = further details. [NullPointerException]"

The files are of the type adm and as = good as equal in size (3 gb).

My instance was initialized with these = commands:

drop dataverse bigd if exists;
  =   create dataverse bigd;
    use = dataverse bigd;

    create type table as open {
= uid: uuid,
        = campus: string,
building: string,
floor: = string,
        timestamp: = int32,
dayOfWeek: int32,
hourOfDay: = int32,
        latitude: = double,
        salt_timestamp: = int32,
        longitude: = double,
        id: = string,
accuracy: double
  =   }
create dataset posdata(table)
    primary key uid autogenerated;
create index stamp on posdata(timestamp);
create index hour on posdata(hourOfDay);
create index day on posdata(dayOfWeek);

My log = file is attached.

Any help?

--

Mvh

Magnus = Kongshem

NTNU
+47 415 65 = 906




--

Mvh

Magnus = Alderslyst Kongshem
Leder av = seniorkomiteen
Online, linjeforeningen for = informatikk
+47 415 65 = 906




--

Mvh

Magnus = Alderslyst Kongshem
Leder av = seniorkomiteen
Online, linjeforeningen for = informatikk
+47 415 65 = 906





--

Mvh

Magnus = Alderslyst Kongshem
Leder av = seniorkomiteen
Online, linjeforeningen for = informatikk
+47 415 65 = 906


Best regards,
Ildar




--

Mvh

Magnus = Alderslyst Kongshem
Leder av = seniorkomiteen
Online, linjeforeningen for = informatikk
+47 415 65 = 906


Best = regards,
Ildar

= --Apple-Mail=_E9429199-A90A-4C3F-BC07-47A497B614A5--