About

The ALB Observability setup by Instrumentify, makes its native access logs queryable and filterable live through AWS Cloudwatch.

With ALB Observability we can create custom metrics, alerts and dashboards, so we can see live who is accessing our APIs, what are they accessing, what sort of response times we have, what are our error rates, and even what sort of parameters our clients send.

The solution takes advantage of existing AWS functionality to setup the listeners for new access logs, and only requires a custom piece of code (the lambda that parses the original access log lines into structured JSON events). The rest is pure AWS configuration, even the custom metrics.

ALB Observability can be deployed in a few clicks as single Cloudformation template, Terraform module, Pulumi package or IaC of your choice. Once we have the basics running, with structured and queryable access logs, we will spend time with your team to define dashboards, metrics and alerts that adapt to your business needs.

Architecture Diagram

graph TD
  alb[ALB <em>$my-alb-name</em>]
  s3[S3 bucket <em>alb-access-logs-$my-alb-name</em>]
  lambda[Lambda <em>alb2cloudwatch-$my-alb-name</em>]
  cw_log_group[Cloudwatch LogGroup <em>/aws/lambda/alb2cloudwatch-$my-alb-name</em>]
  cw_metrics[Cloudwatch Metrics]
  cw_insights[Cloudwatch Log Insights]
  cw_dashboards[Cloudwatch Dashboards]
  cw_alerts[Cloudwatch Alerts]

  alb--->|set attribute <em>access_logs.s3.enabled=true</em>|s3
  s3--->|S3 event notification on <em>*.gz object create</em> to trigger|lambda
  lambda--->|parse each access <a href='https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-examples'>log line</a> into a structured <a href='/docs/alb-ingress/alb-events'>JSON event</a> |cw_log_group
  cw_log_group-->|create <a href='https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html'>metric filters</a> for custom metrics|cw_metrics
  cw_log_group-->|create queries to analyze and filter incoming traffic|cw_insights

  cw_metrics-->cw_dashboards
  cw_insights-->cw_dashboards

  cw_metrics-->|define API SLA|cw_alerts

Limitations

Due to how AWS implements access logs for their Application Load Balancers, there is usually a 5 minute lag between a request being made and the same request being pushed as a gzipped access log to S3. This effectively means that the structured JSON logs queryable from Cloudwatch will always have a 5-6 minute lag.

Additionally, since the lambda processing the access logs runs in batch over the entire gzipped file, metric resolutions smaller than 5 minutes won't be reliable.