Ingress Events

ALB

Instrumentify sets up your AWS application load balancers to forward its access logs to S3. Additionally it sets up a lambda that automatically gets triggered whenever new logs are added onto the S3 bucket. The access logs are stored in a gzipped file, in batches, in approximately 5min intervals, so there's always a 5 to 6 minute lag between requests being replied, and when they will appear in CloudWatch, as structured JSON objects.

Instrumentify's lambda structures the original access log line format into a JSON object, which can be filtered with Cloudwatch or can be used to create custom metrics with CloudWatch Metric filters.

Location

All ALB access log events are pushed as JSON lines to the cloudwatch log group named after the lambda that processes the original logs stored in S3. The log group name should be in the format:

/aws/lambda/${ALBToCloudwatchLambdaName}

Event Format

{
    "alb": {...},
    "req": {...},
    "res": {...}
}

Attributes

The attributes under the alb.* key are parsed straight out of the original access log source, and cast to number, when appropriate.

Attributes under req.* and res.* may include transformations over the original alb.* values, which facilitate advanced filtering and custom metrics.

Name	Type	Description
alb.type	`String`	Type of request. The possible values are: `http`, `https`, `h2`, `grpcs`, `ws` and `wss`.
alb.time	`String`	Time when the load balancer generated a response to the client, in ISO 8601 format.
alb.elb	`String`	Resource ID of the load balancer.
alb.client	`String`	IP address and port of the requesting client, in the format `ip:port`.
alb.target	`String`	IP address and port of the target that processed the event, in the format `ip:port`. If the event was not forwarded, or the target is a lambda function, the value is set to `-`.
alb.request_processing_time	`Number`	Total time elapsed (seconds, with millisecond precision) from the time that the load balancer received the request until it is forwarded to a target. `-1` if the request is not forwarded, or the target does not respond before the idle timeout.
alb.target_processing_time	`Number`	Total time elapsed (seconds, with millisecond precision) from the time the load balancer sent the request to a target, until the target started to send the response headers. `-1` if the load balancer didn't forward the request, or no target responds before the idle timeout.
alb.response_processing_time	`Number`	Total time elapsed (seconds, with millisecond precision) from the time the load balancer recieved the response header from the target, until it started to send the response to the client. This includes queueing time in the load balancer and connection acquisition to the client. Value set to `-1` if no response is received from the target.
alb.elb_status_code	`Number`	HTTP status code of the response from the load balancer. Should match `alb.target_status_code`, if exists.
alb.target_status_code	`Optional[Number]`	HTTP status code of the response from the target. `null` if no valid reply from target (due to timeout, or the request not being forwarded).
alb.received_bytes	`Number`	Size of the request, in bytes, received from the client.
alb.sent_bytes	`Number`	Size of the response, in bytes, sent to the client.
alb.request_method	`String`	HTTP method of the request.
alb.request_url	`String`	URL of the request sent by the client, in the format `protocol://host:port/uri`.
alb.user_agent	`String`	User-Agent string that identifies the client that originated the request.
alb.ssl_cipher	`String`	[HTTPS listener] The SSL cipher. `-` if the listener is not HTTPS.
alb.ssl_protocol	`String`	[HTTPS listener] The SSL protocol. `-`if the listener is not HTTPS.
alb.target_group_arn	`String`	ARN of the target group.
alb.trace_id	`String`	Contents of the X-Amazn-Trace-Id header
alb.domain_name	`String`	[HTTPS listener] SNI domain providede by the client during the TLS handshake. `-` if the client does not support SNI or the domain doesn't match the certificate presented to the client.
alb.chosen_cert_arn	`String`	[HTTPS listener] ARN of the certificate presented to the client. `session-reused` if the session is reused, `-`if the listener is not an HTTPS listener.
alb.matched_rule_priority	`Number`	Priority value of the rule that matched the request. Values from 1 to 50000. If the default action was taken, this value is set to 0``. If an error ocurrs during rule evaluation, value set to-1`.
alb.request_creation_time	`String`	Time when the load balancer received the request from the client, in ISO 8601 format.
alb.actions_executed	`String`	Actions taken when processing the request in the form of comma-separated values. Valid strings: `authenticate`, `fixed-response`, `forward`, `redirect`, `waf`, `waf-failed`.
alb.redirect_url	`String`	URL of the redirect target, if actions_taken is `redirect`. If no redirects, the value is set to `-`.
alb.error_reason	`String`	Error reason code, if the request failed at the load balancer level. If the actions taken do not include an authenticate action, or the target is not a lambda, this value is set to `-`.
alb.target_port_list	`List[String]`	A list of `ip:port` strings for the targets that processed this request. Currently this list should only contain 1 item, if the request is forwarded. Otherwise the list will be empty.
alb.target_status_code_list	`List[Number]`	A list of status codes from the target responses. Currently this list should only contain 1 item, if the request is forwarded. Otherwise the list will be empty.
alb.classification	`String`	The classification for desync migitaction. If the request does not comply with RFC 7230, the possible values are `Acceptable`, `Ambiguous`, and `Severe`. If the request complies, the value is set to `-`.
alb.classification_reason	`String`	Classification reason code, if the request does not comply with RFC 7230. If the request complies, this value is set to `-`.
req.client	`{"addr": String, "port": Number}`	Value from `alb.client` parsed into an object with independent `addr` and `port` keys. Contains the IP and port of the client originating the request.
req.target	`Optional[{"addr": String, "port": Number}]`	Value from `alb.target` parsed into an object with independent `addr` and `port` keys. Contains the IP and port of the target originating the response. If the request was not forwarded, or the target didn't respond before the idle connection timeout, the field will be set to `null`.
req.summary	`String`	Value composed of the following fields: `{req.method} {req.pathname}`. Useful to setup as a dimension in a custom chart or metric, when you want to compare usage and statistics between different REST endpoints.
req.method	`String`	HTTP method of the request. Value extracted from `alb.request_method`.
req.href	`String`	URL of the request sent by the client, in the format `protocol://host:port/uri`. Value extracted from `alb.request_url`.
req.origin	`String`	URL of the request sent by the client, in the format `protocol://host:port/uri`.
req.protocol	`String`	String containing the protocol scheme of the URL, including the final `:`. From javascript's URL.protocol property.
req.username	`String`	String containing the username specified before the domain name, if any. From javascript's URL.username property.
req.password	`String`	String containing the password specified before the domain name, if any. From javascript's URL.password property.
req.host	`String`	String containing the hostname and the port, if any. From javascript's URL.host property.
req.hostname	`String`	String containing the hostname. From javascript's URL.hostname property.
req.domain	`String`	Top level domain name from the request url. For example, from `http://www.google.com`, `req.domain = google.com`. If the URL contains an IP address, it's value will be set to `IP`.
req.subdomain	`Optional[String]`	Subdomain from the request url. For example, from `http://www.google.com`, `req.subdomain = www`. If the URL only contains a top level domain, or an IP, it's value will be `null`.
req.port	`Number`	Number containing the port of the URL, if any. Otherwise `null`. From javascript's URL.port property.
req.pathname	`String`	String containing all the paths segments of the URL. For example, from `https://example.org/users?sort=modified`, `req.pathname = /users`. From javascript's URL.pathname property.
req.search	`String`	String containing the querystring from the URL, if any. For example, from `https://example.org/users?sort=modified`, `req.search = ?sort=modified`. From javascript's URL.search property.
req.search_params	`Optional[Object]`	Object parsing `req.search` parameters into a Key-Value object. `null` if `req.search` is empty. For example, from `https://example.org/users?sort=modified`, `req.searchParams = {"sort": "modified"}`.
res.processing_time	`Number`	Response time from the target, corresponding to `alb.target_processing_time`. The value is returned as an integer, in milliseconds. `-1` if the request was not forwarded to a target, or it timed out.
res.status_code	`Number`	HTTP status code of the response. Corresponds to `alb.elb_status_code`.
res.status_class	`String`	HTTP status class. Derived from `res.status_code`. Its valid values are `2xx`, `3xx`, `4xx` and `5xx`. Groups status codes from `res.status_code`into those 4 classes. This is a useful field to generate custom metrics upon which to build alerts to keep track of API health.

Examples

HTTPS request forwarded to target

{
    "alb": {
        "type": "https",
        "time": "2023-09-15T15:29:59.137592Z",
        "elb": "app/k8s-default-apichart-be0d348986/244108bde4c2e4ab",
        "client": "46.6.33.173:23461",
        "target": "10.1.16.21:8000",
        "request_processing_time": 0,
        "target_processing_time": 0.003,
        "response_processing_time": 0,
        "elb_status_code": 200,
        "target_status_code": 200,
        "received_bytes": 150,
        "sent_bytes": 174,
        "request_method": "GET",
        "request_url": "https://apichart.instrumentify.io:443/?param1=value1&param2=value2",
        "user_agent": "hey/0.0.1",
        "ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256",
        "ssl_protocol": "TLSv1.2",
        "target_group_arn": "arn:aws:elasticloadbalancing:eu-west-3:022747666878:targetgroup/k8s-default-apichart-5f3a7e13bd/6a3e1160ef8a0468",
        "trace_id": "Root=1-65047877-57a0847a4cee0d6375de4659",
        "domain_name": "apichart.instrumentify.io",
        "chosen_cert_arn": "arn:aws:acm:eu-west-3:022747666878:certificate/64daee1d-efc5-4caf-9dfe-b1660f96e0c4",
        "matched_rule_priority": 1,
        "request_creation_time": "2023-09-15T15:29:59.134000Z",
        "actions_executed": "forward",
        "redirect_url": "-",
        "error_reason": "-",
        "target_port_list": [
            "10.1.16.21:8000"
        ],
        "target_status_code_list": [
            200
        ],
        "classification": "-",
        "classification_reason": "-"
    },
    "req": {
        "client": {
            "addr": "46.6.33.173",
            "port": 23461
        },
        "target": {
            "addr": "10.1.16.21",
            "port": 8000
        },
        "summary": "GET /",
        "method": "GET",
        "href": "https://apichart.instrumentify.io/?param1=value1&param2=value2",
        "origin": "https://apichart.instrumentify.io",
        "protocol": "https:",
        "username": "",
        "password": "",
        "host": "apichart.instrumentify.io",
        "hostname": "apichart.instrumentify.io",
        "domain": "instrumentify.io",
        "subdomain": "apichart",
        "port": null,
        "pathname": "/",
        "search": "?param1=value1&param2=value2",
        "search_params": {
            "param1": "value1",
            "param2": "value2"
        },
        "hash": ""
    },
    "res": {
        "processing_time": 3,
        "status_code": 200,
        "status_class": "2xx"
    }
}

HTTPS request with fixed-response, no forward

{
    "alb": {
        "type": "https",
        "time": "2023-09-15T12:54:19.702592Z",
        "elb": "app/k8s-default-apichart-be0d348986/244108bde4c2e4ab",
        "client": "52.200.22.20:60336",
        "target": "-",
        "request_processing_time": -1,
        "target_processing_time": -1,
        "response_processing_time": -1,
        "elb_status_code": 404,
        "target_status_code": null,
        "received_bytes": 230,
        "sent_bytes": 167,
        "request_method": "GET",
        "request_url": "https://13.36.23.220:443/test.php",
        "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256",
        "ssl_protocol": "TLSv1.2",
        "target_group_arn": "-",
        "trace_id": "Root=1-650453fb-1878874d0b9cb6af7d6f804d",
        "domain_name": "-",
        "chosen_cert_arn": "arn:aws:acm:eu-west-3:022747666878:certificate/64daee1d-efc5-4caf-9dfe-b1660f96e0c4",
        "matched_rule_priority": 0,
        "request_creation_time": "2023-09-15T12:54:19.702000Z",
        "actions_executed": "fixed-response",
        "redirect_url": "-",
        "error_reason": "-",
        "target_port_list": [],
        "target_status_code_list": [],
        "classification": "-",
        "classification_reason": "-"
    },
    "req": {
        "client": {
            "addr": "52.200.22.20",
            "port": 60336
        },
        "target": {
            "addr": "-",
            "port": null
        },
        "summary": "GET /test.php",
        "method": "GET",
        "href": "https://13.36.23.220/test.php",
        "origin": "https://13.36.23.220",
        "protocol": "https:",
        "username": "",
        "password": "",
        "host": "13.36.23.220",
        "hostname": "13.36.23.220",
        "domain": "IP",
        "subdomain": null,
        "port": null,
        "pathname": "/test.php",
        "search": "",
        "search_params": {},
        "hash": ""
    },
    "res": {
        "processing_time": -1,
        "status_code": 404,
        "status_class": "4xx"
    }
}