Mailing-List: contact dev-help@sqoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@sqoop.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+ShD0f0FhrqNbcAMhSdWnhcXHdvUiKdYWv9xOCcbYEx8Y1gSw@mail.gmail.com>
References: 
 <CA+ShD0fDZ5Vv+LtC4+wpHPcYf6WapoK3+96wNmNdiO+tyA7hAw@mail.gmail.com>
	<7F91673573F5D241AFCE8EDD6A313D24604168@SHSMSX103.ccr.corp.intel.com>
	<CA+ShD0f3W1ppFzD3TrX7agSNWr-EnYz6amRwEWMjDiB2yCKVww@mail.gmail.com>
	<7F91673573F5D241AFCE8EDD6A313D24604265@SHSMSX103.ccr.corp.intel.com>
	<CA+ShD0f0FhrqNbcAMhSdWnhcXHdvUiKdYWv9xOCcbYEx8Y1gSw@mail.gmail.com>
Date: Tue, 2 Jun 2015 12:11:47 -0700
Message-ID: 
 <CA+ShD0cTm81x4YS=xRWU=2Fi3hk2CVM+XmffsyTh_UzdnKzkgw@mail.gmail.com>
Subject: Re: Sqoop2 job hung for long time
From: SHANKAR REDDY <sankara.telukutla@gmail.com>
To: "dev@sqoop.apache.org" <dev@sqoop.apache.org>
Content-Type: multipart/alternative; boundary=001a1135f1a6c2b2fa05178db683

--001a1135f1a6c2b2fa05178db683
Content-Type: text/plain; charset=UTF-8

This seems like the clear problem with Yarn which is unable start even
sample word count example as well.


Kind Regards,
Sankara Telukutla


On Tue, Jun 2, 2015 at 11:13 AM, SHANKAR REDDY <sankara.telukutla@gmail.com>
wrote:

> Hi Richard,
> Attached the logs from Yarn and Sqoop2.
>
> After recycle the cluster I see that new files are created. The logs
> attached are fresh logs after server restart.
>
> Most of the places I see the below error from the logs.
>
> Caused by: java.lang.UnsupportedOperationException: Usage threshold is not
> supported
>
> Please suggest.
>
>
> Kind Regards,
> Sankara Telukutla
>
>
> On Tue, Jun 2, 2015 at 12:27 AM, Zhou, Richard <richard.zhou@intel.com>
> wrote:
>
>> Hey, it seems that resource manager cannot create container.
>> For the log file in
>>
>> /var/log/hadoop-yarn
>> /var/log/sqoop2
>>
>> You cannot delete the log file manually. CDH will append log into the log
>> file, but if there is no log file existing in the folder, CDH will not
>> create a new log file.
>> You will find error message in CM (cluster->yarn->resource manager->log
>> file link in summary tab).
>> You need to touch a new log file and change owner to yarn and change mode
>> to 644.
>>
>> [root@server-654 richard]# ll
>> /var/log/hadoop-yarn/hadoop-cmf-yarn2-RESOURCEMANAGER-server-654.novalocal.log.out
>> -rw-r--r-- 1 yarn yarn 54243 Jun  2 14:42
>> /var/log/hadoop-yarn/hadoop-cmf-yarn2-RESOURCEMANAGER-server-654.novalocal.log.out
>>
>>
>> Regards
>> Richard
>>
>>
>> -----Original Message-----
>> From: SHANKAR REDDY [mailto:sankara.telukutla@gmail.com]
>> Sent: Tuesday, June 02, 2015 12:51 PM
>> To: dev@sqoop.apache.org
>> Subject: Re: Sqoop2 job hung for long time
>>
>> Hi Richard,
>> Please see below.
>>
>> sqoop:000> set option -name verbose -value true Verbose option was
>> changed to true sqoop:000> sqoop:000> start job -jid 2 -s Submission
>> details Job ID: 2 Server URL: http://localhost:12000/sqoop/ Created by:
>> ubuntu Creation date: 2015-06-02 04:41:56 UTC Lastly updated by: ubuntu
>> External ID: job_1433186285881_0007
>>
>>
>> http://ip-172-31-1-201.us-west-1.compute.internal:8088/proxy/application_1433186285881_0007/
>> Source Connector schema: Schema{name=clp_sandbox.HADOOP_TEST,columns=[
>>
>>
>> FixedPoint{name=SERIAL_NO,nullable=true,type=FIXED_POINT,byteSize=4,signed=true},
>>
>>
>> FixedPoint{name=EMPLOYEE_ID,nullable=true,type=FIXED_POINT,byteSize=4,signed=true},
>>         Text{name=NAME,nullable=true,type=TEXT,charSize=null}]}
>> 2015-06-02 04:41:56 UTC: BOOTING  - Progress is not available
>> 2015-06-02 04:42:07 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:42:17 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:42:27 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:42:37 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:42:47 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:42:57 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:43:07 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:43:17 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:43:27 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:43:37 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:43:47 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:43:57 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:44:07 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:44:17 UTC: BOOTING  - 0.00 %
>> 2015-06-02 04:44:27 UTC: BOOTING  - 0.00 %
>>
>>
>> ..
>> I cleaned up the logs and before doing the above steps and found that no
>> new logs are created at below location.
>>
>> /var/log/hadoop-yarn
>> /var/log/sqoop2
>>
>>
>> Is there anything I can verify?
>>
>> Kind Regards,
>> Sankara Telukutla
>>
>>
>> On Mon, Jun 1, 2015 at 8:10 PM, Zhou, Richard <richard.zhou@intel.com>
>> wrote:
>>
>> > Hey, would you send out the log, including the Sqoop log and Yarn log.
>> > It should be in /var/log/ if Cloudera 5.4.1 is installed.
>> > And run command "set option -name verbose -value true" to set verbose,
>> > then re-run the job with "start job -jid 2 -s" to show more information.
>> >
>> >
>> > Regards
>> > Richard
>> >
>> > -----Original Message-----
>> > From: SHANKAR REDDY [mailto:sankara.telukutla@gmail.com]
>> > Sent: Tuesday, June 02, 2015 9:16 AM
>> > To: dev@sqoop.apache.org
>> > Subject: Sqoop2 job hung for long time
>> >
>> > Team,
>> >
>> > I have a job which is from MySql to HDFS transformation using sqoop2.
>> > The job I have started not responding for long time and seems like
>> > there is a problem with YARN which couldn't able to pick up the same.
>> > Could you please help me to rectify this problem?
>> >
>> > Version details:
>> > SQOOP2  : 1.99.5
>> > Cloudera :  5.4.1
>> >
>> > application_1433186285881_0004
>> > <
>> > http://ec2-52-8-94-128.us-west-1.compute.amazonaws.com:8088/cluster/ap
>> > p/application_1433186285881_0004
>> > >
>> > sqoop2Sqoop: Test Job-copyMAPREDUCEroot.sqoop2Mon Jun 1 17:18:39 -0700
>> > 2015
>> >
>> > N/AACCEPTEDUNDEFINED
>> > UNASSIGNED
>> > <http://ec2-52-8-94-128.us-west-1.compute.amazonaws.com:8088/cluster/#
>> > >
>> >
>> >
>> > And the job information :
>> > sqoop:000> show job -jid 2
>> > 1 job(s) to show:
>> > Job with id 2 and name Test Job-copy (Enabled: true, Created by null
>> > at
>> > 5/21/15 8:45 AM, Updated by null at 6/2/15 12:18 AM) Using link id 1
>> > and Connector id 4
>> >   From database configuration
>> >     Schema name: clp_sandbox
>> >     Table name: HADOOP_TEST
>> >     Table SQL statement:
>> >     Table column names:
>> >     Partition column name: EMPLOYEE_ID
>> >     Null value allowed for the partition column: true
>> >     Boundary query:
>> >   Throttling resources
>> >     Extractors: 2
>> >     Loaders: 1
>> >   ToJob configuration
>> >     Override null value: false
>> >     Null value:
>> >     Output format: SEQUENCE_FILE
>> >     Compression format: NONE
>> >     Custom compression format:
>> >     Output directory: /hadooptest
>> >
>> >
>> >
>> > Let me know if you need any information I can provide.
>> >
>> >
>> >
>> > -Shankar
>> >
>>
>
>

--001a1135f1a6c2b2fa05178db683--