Introduction

System Logging Protocol (Syslog) is a data protocol used by network devices to collect logs to a logging server. It can record log messages of multiple event types and is supported by almost all network devices, such as routers, switches, and printers. Even Unix-like servers can generate syslog messages to record user logins, firewall events, and Apache or Nginx access.

Syslog defines related format specifications based on RFC5424 and RFC3164. RFC3164 was released in 2001, and RFC5424 was an upgraded version released in 2009. The new version is compatible with the old version and solves many problems of the old version. Therefore, RFC 5424 is recommended. For more information, see RFC5424 and RFC3164.

This section describes how to use the syslog protocol to upload logs to LTS. You need to create an ECS as a syslog aggregation server. Rsyslog comes preinstalled by default on Linux servers. However, ECSs do not receive remote syslog writes by default. You need to manually enable this function.

Solution

• You can create a Linux ECS and configure it as a syslog aggregation server to receive log data from other devices. If the size of a log received by a syslog server exceeds 1024 bytes, the log will be truncated.
• The log processing rate of a single syslog server is 10 MB/s. To process a large number of logs or ensure high reliability, you can create multiple ECSs as syslog servers and configure load balancers for distributing traffic.
• You need to install ICAgent on syslog servers and configure log collection rules to collect logs to LTS.

Prerequisites

Request two ECSs. One serves as a syslog aggregation server, and the other serves as a service ECS to simulate user systems or devices to send logs.

Procedure

1. Log in to the ECS console and request an ECS as a syslog aggregation server.

   You are advised to use CentOS 6.5 64bit or later images. The recommended specifications are 2 vCPUs and 4 GB of memory.

2. Log in to the syslog server as user root to install ICAgent.
   1. Click the server name to go to the details page. On the Security Groups tab page, click the ID name to set security group rules. For details, see "Adding a Security Group Rule" in the "Security Group" section of Virtual Private Cloud User Guide. Allow TCP ports 30200, 30201, 8149, 8923, and 8102 in the outbound rules of the syslog server, and then allow UDP port 514 in the inbound rules as the default listening port.

   2. Log in to the LTS console. In the navigation pane, choose Host Management. Click Install ICAgent. Change the AK/SK value in the command to the obtained AK/SK.

   3. Log in to the syslog server as user root and run the ICAgent installation command. If the message ICAgent install success is displayed, ICAgent is successfully installed. You can then view the ICAgent status by choosing Host Management in the navigation pane of the LTS console and then clicking Hosts.

3. Enable the rsyslog listening and receiving functions.
   By default, rsyslog of ECSs does not receive remote syslog writes. You need to manually enable this function.

   1. Log in to the ECS.
   2. Modify the rsyslog configuration file.

      vi /etc/rsyslog.conf

   3. Add the following content to the configuration file to enable TCP and UDP remote receiving:

      Provides UDP syslog reception 
      $ModLoad imudp 
      $UDPServerRun 514
      
      Provides TCP syslog reception 
      $ModLoad imtcp 
      $InputTCPServerRun 514

   4. Save the settings. Click More > Restart in the Operation column to restart the ECS.

   5. Run any of the following commands. If the command output is normal, the service is working.

      Run service rsyslog status to check whether the rsyslog running status is running.

      Run systemctl status rsyslog to check whether the rsyslog running status is running.

      Run netstat -anp | grep 514 to check whether the listening function is enabled.

4. Configure syslog collection.
   1. Choose Host Management in the LTS navigation pane and click Create Host Group to create a host group. On the displayed page, enter a host group name and select hosts.

   2. Choose Log Ingestion in the LTS navigation pane. On the displayed page, click ECS (Elastic Cloud Server).
   3. Select a log stream.

   4. Select a host group.
   5. Set the collection path to /var/log/messages.

5. Log in to the service ECS for verification.

   After your service system or device generates logs, you can view the logs on the LTS console. Log in to the service ECS and run the logger -n x.x.x.x -P 514 testremotelog command to send syslog messages to the aggregation server. x.x.x.x indicates the IP address (public or private) of the syslog server. testremotelog indicates the log content, which can be customized.

   After the command is executed, you can view the log in the configured log group and log stream.

   Alternatively, log in to the syslog aggregation server and check whether the testremotelog log exists in /var/log/messages.

   tail -f /var/log/messages

6. Use multiple syslog servers and load balancers to implement load balancing.

   The log processing rate of a single syslog server is 10 MB/s. To process a large number of logs, you can use multiple syslog servers and load balancers.

   1. Create syslog aggregation servers and install ICAgent.
   2. Create a load balancer. For details, see "Getting Started" > "Applying for a Load Balancer" in Elastic Load Balance User Guide.
   3. Add listeners for TCP/UDP ports and port 514. For details, see "Getting Started" > "Adding a Listener" in Elastic Load Balance User Guide.
   4. Add backend servers to the backend server group. For details, see "Getting Started" > "Adding a Backend Server" in Elastic Load Balance User Guide.