hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Question - Remote Logging to Hadoop
Date Mon, 18 May 2009 07:55:27 GMT
We have deployed a pilot for making scribe log directly into a hdfs 0.19
cluster. We will be pushing these changes to scribe to the open-source
location this week. Also, there are a few changes to HDFS that were required
to make this work smoothly, e.g. HADOOP-2757. These changes are being
debated to figure out what will be pushed into hdfs trunk.

 Also, here are some of the configuration settings:

        <property>
                <name>ipc.client.idlethreshold</name>
                <value>10000</value>
                <description>Defines the threshold number of connections
after which
               connections will be inspected for idleness.
               </description>
        </property>
        <property>
          <name>ipc.client.connection.maxidletime</name>
          <value>10000</value>
          <description>The maximum time in msec after which a client will
bring down the
                       connection to the server.
          </description>
        </property>
        <property>
          <name>ipc.client.connect.max.retries</name>
          <value>2</value>
          <description>Indicates the number of retries a client will make to
establish
                       a server connection.
          </description>
        </property>
        <property>
          <name>ipc.server.listen.queue.size</name>
          <value>128</value>
          <description>Indicates the length of the listen queue for servers
accepting
                       client connections.
          </description>
        </property>
      <property>
          <name>ipc.server.tcpnodelay</name>
          <value>true</value>
          <description>Turn on/off Nagle's algorithm for the TCP socket
connection on
          the server. Setting to true disables the algorithm and may
decrease latency
          with a cost of more/smaller packets.
          </description>
        </property>
        <property>
          <name>ipc.client.tcpnodelay</name>
          <value>true</value>
          <description>Turn on/off Nagle's algorithm for the TCP socket
connection on
          the client. Setting to true disables the algorithm and may
decrease latency
          with a cost of more/smaller packets.
          </description>
        </property>
        <property>
          <name>ipc.ping.interval</name>
          <value>5000</value>
          <description>The Client sends a ping message to server every
period. This is helpful
          to detect socket connections that were idle and have been
terminated by a failed server.
          </description>
        </property>
        <property>
          <name>ipc.client.connect.maxwaittime</name>
          <value>5000</value>
          <description>The Client waits for this much time for a socket
connect call to be establised
          with the server.
          </description>
        </property>
        <property>
          <name>dfs.datanode.socket.write.timeout</name>
          <value>20000</value>
          <description>The dfs Client waits for this much time for a socket
write call to the datanode.
          </description>
        </property>
        <property>
          <name>dfs.leaserenewal.timeout</name>
          <value>10000</value>
          <description>The dfs writer waits for this much time after the
last successful lease renewal
          before aborting the write to the file.
          </description>
        </property>


On Mon, May 18, 2009 at 12:22 AM, Amr Awadallah <aaa@cloudera.com> wrote:

> Scribe from Facebook:
>
> http://developers.facebook.com/scribe/
>
> -- amr
>
>> All,
>>
>> Does anyone have a working method to capture system and application
>> logging for remote servers directly into HDFS in real-time?
>>
>> We have tested a few methods but all seem reliant on a batch process run
>> every 5 to 10 minutes.
>>
>> Any ideas would be appreciated?
>>
>>
>>
>> Warm Regards,
>>
>>
>> Scott
>>
>> ________________________________
>> Disclaimer: This email and any attachments thereto are confidential and it
>> may contain information protected by intellectual Property Laws or otherwise
>> legally privileged. It is intended solely for the use of the addressee(s).
>> If you are not the intended recipient, please notify the sender immediately
>> and delete this email and any attachments from your system and destroy any
>> printout thereof. Any unauthorized use or disclosure of this email and its
>> attachments is strictly prohibited and may be unlawful. MEEZA QSTP-LLC
>> (MEEZA) does not certify that this email or any attachments are free of
>> viruses or defects and recommends that you undertake your own virus checking
>> procedures before opening this e-mail or any attachments. MEEZA does not
>> accept any liability for any loss or damage resulting from the use of this
>> e-mail or any attachments. The contents of this e-mail and any attachments
>> do not necessarily represent the views or polices of MEEZA. As the integrity
>> of e-mails across the internet cannot be guaranteed, MEEZA does not accept
>> any liability for any e-mails and attachments which may violate the relevant
>> applicable laws.
>>
>>
>>
>>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message