Sinefa probes can ingest and report on AWS VPC Flow Logs. This guide explains how to configure Sinefa probes to retrieve AWS VPC Flow Logs from an AWS S3 bucket.
Sinefa Probes (deployed either inside or outside of AWS) are configured to constantly monitor an AWS S3 bucket for new Flow Log files. Similar to Netflow, these Flow Log records contain summary information about the network flows in/out of each VM with Flow Logs enabled.
For details on how to configure AWS VPC Flow Logs, refer to the AWS documentation. The only requirement is that AWS VPC Flow Logs must be saved to S3 and use the default AWS VPC Flow Log format. Sinefa currently does not support reading AWS VPC Flow Logs from CloudWatch.
Once AWS VPC Flow Logs are saved to S3, a Sinefa Probe needs to be configured to read these files as they become available. This requires 2 steps, configuring a download and configuring a schedule.
AWS VPC Flog Log Download
To configure a Sinefa Probe to download AWS VPC Flow Logs from S3, you must access the probe's CLI and run the following commands.
download create <name>
download.<name> url set s3://<bucket>/AWSLogs/<account_id>/vpcflowlogs/<region>/
download.<name> handler set aws-vpc2-flowlogs
download.<name> recursive enable
download.<name> max_age set 2d
download.<name> aws_access_key_id set <id>
download.<name> aws_secret_access_key set <secret>
The download name must not contain spaces. A Sinefa source is created for each AWS VPC Flow Log download. For example, you can export AWS VPC Flow Logs from multiple VPCs to a single S3 bucket and configure a single download. This will result in all the traffic from these VPCs reported as a single Sinefa source. Alternatively, you can send AWS VPC Flow Logs from different VPCs to different buckets and configure a download for each one. This will result in each VPC reported seperately within its own Sinefa source.
The following example download configuration will connect to the specified S3 bucket and recursively look for AWS VPC Flow Logs. The downloader will track which AWS VPC Flow Logs have been downloaded and will only process new AWS VPC Flow Logs up to 48 hours old.
download create us-west-1-prod-vpc
download.us-west-1-prod-vpc url set s3://my-vpc-fflow-lgos/AWSLogs/123456789/vpcflowlogs/us-west-1/
download.us-west-1-prod-vpc handler set aws-vpc2-flowlogs
download.us-west-1-prod-vpc recursive enable
download.us-west-1-prod-vpc max_age set 2d
download.us-west-1-prod-vpc aws_access_key_id set my-aws-id
download.us-west-1-prod-vpc aws_secret_access_key set my-aws-secret
To test this config, run:
Once your VPC Flow Logs are downloading as expected, proceed to scheduling this process.
AWS VPC Flow Log Schedule
A schedule simply runs an action periodically. Since an AWS VPC Flow Log download has already been configured, a schedule simply runs this download as required. Usually this is every 10 minutes. The following CLI commands can be used to create the schedule.
schedule create <name>
schedule.<name> minute set */10
schedule.<name> commands set "download.<download_name> run"
The above AWS VPC Flow Log example can be scheduled as follows.
schedule create download-my-flowlogs
schedule.download-my-flowlogs minute set */10
schedule.<download-my-flowlogs commands set "download.us-west-1-prod-vpc run"
If you have multiple AWS VPC Flow Log downloads configured, these can be run from the same schedule by specifying multiple commands.
This will execute each AWS VPC Flow Log download every 10 minutes, new Flow Logs will be downloaded and processed and will update their respective Sources in the Sinefa reports.
You should inspect the resulting reports in the Sinefa UI and ensure and local or LAN subnets are defined so that traffic direction is reported correctly.