Deprecation Note

We published the last version of Graylog Documentation before the release of Graylog 4.2. Now, all documentation and help content for Graylog products are available at https://docs.graylog.org/.

There will be no further updates to these pages as of October 2021.

Do you have questions about our documentation? You may place comments or start discussions about documentation here: https://community.graylog.org/c/documentation-campfire/30

AWS Kinesis/CloudWatch Input

Note

This input is available since Graylog version 3.1.1. Installation of an additional graylog-integrations-plugins package is required. See the Integrations Setup page for more info.

Attention

An understanding of how AWS CloudWatch and Kinesis is required.

Attention

Versions of Graylog and graylog-integrations-plugins must be the same.

This input allows Graylog to read log messages from CloudWatch via Kinesis. When reading logs from CloudWatch, Kinesis is required in order to stream messages to Graylog.

The following message types are supported:

CloudWatch Logs

Raw text strings within in Cloudwatch.

CloudWatch Flow Logs

Flow Logs within a Cloud Watch log group.

Kinesis Raw Logs

Raw text strings written to Kinesis.

Manual Setup Flow

For this setup to function as expected, the Least Privilege Policy shown below must be allowed for the authorized user. (See Permission Policies below)

1. AWS Kinesis Authorize

   Type in input name, AWS Access Key, AWS Secret Key and select AWS Region in order to authorize Graylog and click the Authorize & Choose Stream button to continue. (See image below.)

2. AWS Kinesis Setup

   Select the Kinesis stream to pull logs from and click the Verify Stream & Format button to continue.

3. AWS CloudWatch Health Check

   Graylog will read a message from the Kinesis stream and check it’s format. We’ll automatically parse the message if it’s a Flow Log.

4. AWS Kinesis Review

   Final step to review and finalize the details for the input.

Automatic Setup Flow

When adding the AWS Kinesis/CloudWatch input to Graylog, you will be guided throughout the setup process. For this setup to function as expected, the Recommended Policy shown below must be allowed for the authorized user. (See Permission Policies below)

1. AWS Kinesis Authorize

   Type in input name, AWS Access Key, AWS Secret Key and select AWS Region in order to authorize Graylog and click the Authorize & Choose Stream button to continue. (See image above)

2. AWS Kinesis Setup

   In the blue dialog box (seen in the image above), click the Setup Kinesis Automatically button. Type in a name for the Kinesis stream name, and select a Cloudwatch log Group from the dropdown list and click the Begin Automated Setup button.

   You will be prompted with Kinesis Auto Setup Agreement and will need to acknowledge that you are aware of the resources that will be created and click the I Agree! Create these AWS resources now. button. (See images below).

   Once agreed and acknowledge, the auto-setup will detail and reference the resources that were created and you can click the Continue Setup button. (see Executing Auto-Setup image below)

3. AWS CloudWatch Health Check

   Graylog will read a message from the Kinesis stream and check its format. Graylog will attempt to automatically parse the message if it is of a known type.

4. AWS Kinesis Review

   Final step to review and finalize the details for the input.

Permission Policies

Manual Setup Flow Permissions

Automatic Setup Flow Permissions