Static Website Monitoring

·11 min read
awswebsitearchitecturecloudcloudformation
Static Website Monitoring

In the previous post I covered the AWS resources behind my static website - including the S3 buckets collecting access logs for both the website itself and the CloudFront distribution in front of it.

Once those logs started accumulating, I wanted a way to interrogate them: which pages are people actually requesting? Where do requests come from? How much is the traffic costing me in data transfer? Copying .log files down and grepping them wasn’t going to scale.

The obvious answer was Athena, AWS’s serverless query engine - it reads straight out of S3 with no infrastructure to manage. All Athena needs to run a query is a table definition telling it where the data lives and how it’s formatted. That’s where Glue comes in: an AWS Glue catalog database holding tables that describe the log file schemas in the buckets.

This post is part two of the series, covering:

  • The monitoring resources - a Glue database plus one table for the website S3 bucket access logs and one for the CloudFront access logs
  • Deployment - a GitHub Actions workflow that deploys the monitoring stack after the website stack
  • IAM policy - the least-privilege policy for the monitoring resources
  • Final template - the complete, copy-pasteable CloudFormation
  • References - references used when building the template

The monitoring resources

Only three resources are needed to unlock Athena:

Glue monitoring database

First up, an AWS::Glue::Database to hold the metadata tables. The database only stores the metadata - not the data itself - so the DeletionPolicy can safely be set to Delete, and a DatabaseInput is the only property that needs specifying:

  MonitoringDatabase:
    Type: AWS::Glue::Database
    DeletionPolicy: Delete
    Properties:
      CatalogId: !Ref AWS::AccountId
      DatabaseInput:
        Name: !Sub ${WebsiteName}-requests

Glue table for the S3 access logs

Next, the table for the S3 bucket access logs. Like the database, the table stores no data - it’s pure schema - so DeletionPolicy: Delete again. S3 server access logs are space-delimited with quoted fields and bracketed timestamps, so the safest parser is a regular expression rather than a simple delimiter:

  BucketLogsTable:
    Type: AWS::Glue::Table
    DeletionPolicy: Delete
    Properties:
      CatalogId: !Ref "AWS::AccountId"
      DatabaseName: !Ref MonitoringDatabase
      TableInput:
        Name: bucket_request_logs
        TableType: EXTERNAL_TABLE
        Parameters:
          has_encrypted_data: "false"
          classification: "aws s3 access log"
        StorageDescriptor:
          Location: !Sub s3://${WebsiteDomain}-bucket-requests/
          InputFormat: org.apache.hadoop.mapred.TextInputFormat
          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
          SerdeInfo:
            SerializationLibrary: org.apache.hadoop.hive.serde2.RegexSerDe
            Parameters:
              field.delim: "\t"
              serialization.format: "\t"
              input.regex: '([^ ]*) ([^ ]*) \[(.*?)\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ("[^"]*"|-) (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ("[^"]*"|-) ([^ ]*)(?: ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*))?.*$'
          Columns:
            - Name: bucket_owner
              Type: string
            - Name: bucket_name
              Type: string
            - Name: request_datetime
              Type: string
            - Name: remote_ip
              Type: string
            - Name: requester
              Type: string
            - Name: request_id
              Type: string
            - Name: operation
              Type: string
            - Name: key
              Type: string
            - Name: request_uri
              Type: string
            - Name: http_status
              Type: int
            - Name: error_code
              Type: string
            - Name: bytes_sent
              Type: bigint
            - Name: object_size
              Type: bigint
            - Name: total_time
              Type: int
            - Name: turn_around_time
              Type: int
            - Name: referrer
              Type: string
            - Name: user_agent
              Type: string
            - Name: version_id
              Type: string
            - Name: host_id
              Type: string
            - Name: signature_version
              Type: string
            - Name: cipher_suite
              Type: string
            - Name: authentication_type
              Type: string
            - Name: host_header
              Type: string
            - Name: tls_version
              Type: string
            - Name: access_point_arn
              Type: string
            - Name: acl_required
              Type: string
            - Name: source_region
              Type: string

Three things worth explaining:

RegexSerDe, not simple delimiters - S3 access logs aren’t cleanly delimited. Quoted user agents, bracketed timestamps and - sentinel values for absent fields mean positional parsing breaks as soon as a request arrives with an unusual referrer. The input.regex pattern captures each field in order, with the quoted-or-dash alternation (\"[^\"]*\"|-) handling fields that may or may not be quoted.

Column count must equal capture group count - the twenty-six columns map positionally to the twenty-six capture groups. This is the single most fragile part of the whole setup: S3 has added fields to the log format over time, so a regex written against an older format silently stops matching when AWS appends a new field. If Athena returns zero rows or a wall of NULLs, this alignment is the first thing to check.

The trailing anchor - note the regex ends )?$ rather than )?.*$. The .* variant is widely copied around the internet and can mask mismatches rather than surfacing them. Anchoring at $ means a line whose field count doesn’t match the pattern fails visibly, which is far easier to debug than silently shifted columns.

Glue table for the CloudFront request logs

The CloudFront table is where things get interesting, because CloudFront has two log formats and they need completely different tables.

Legacy versus v2 - CloudFront’s original access logs were gzipped, tab-delimited text. The newer standard logging (v2) format - which is what the delivery configuration in part one sets up, with OutputFormat: json - delivers newline-delimited JSON instead. A tab-delimited table pointed at JSON files parses successfully and returns complete nonsense, so the format matters.

Since part one configured JSON output, the table uses a JSON SerDe:

  CloudfrontLogsTable:
    Type: AWS::Glue::Table
    DeletionPolicy: Delete
    Properties:
      CatalogId: !Ref "AWS::AccountId"
      DatabaseName: !Ref MonitoringDatabase
      TableInput:
        Name: cloudfront_request_logs
        TableType: EXTERNAL_TABLE
        Parameters:
          classification: json
          'use.null.for.invalid.data': 'true'
          'ignore.malformed.json': 'true'
        StorageDescriptor:
          Location: !Sub s3://${WebsiteDomain}-cloudfront-requests/
          StoredAsSubDirectories: true
          InputFormat: org.apache.hadoop.mapred.TextInputFormat
          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
          SerdeInfo:
            SerializationLibrary: org.openx.data.jsonserde.JsonSerDe
            Parameters:
              'mapping.client_ip': 'client-ip'
              'mapping.cs_method': 'cs-method'
              'mapping.cs_host': 'cs-host'
              'mapping.cs_uri_stem': 'cs-uri-stem'
              'mapping.cs_referrer': 'cs-referrer'
              'mapping.cs_user_agent': 'cs-user-agent'
              'mapping.cs_uri_query': 'cs-uri-query'
              'mapping.cs_cookie': 'cs-cookie'
              'mapping.cs_protocol': 'cs-protocol'
              'mapping.cs_protocol_version': 'cs-protocol-version'
              'mapping.sc_bytes': 'sc-bytes'
              'mapping.sc_status': 'sc-status'
              'mapping.sc_content_type': 'sc-content-type'
              'mapping.sc_content_len': 'sc-content-len'
              'mapping.sc_range_start': 'sc-range-start'
              'mapping.sc_range_end': 'sc-range-end'
              'mapping.x_edge_location': 'x-edge-location'
              'mapping.x_edge_result_type': 'x-edge-result-type'
              'mapping.x_edge_request_id': 'x-edge-request-id'
              'mapping.x_edge_response_result_type': 'x-edge-response-result-type'
              'mapping.x_edge_detailed_result_type': 'x-edge-detailed-result-type'
              'mapping.x_host_header': 'x-host-header'
              'mapping.x_forwarded_for': 'x-forwarded-for'
              'mapping.time_taken': 'time-taken'
              'mapping.time_to_first_byte': 'time-to-first-byte'
              'mapping.ssl_protocol': 'ssl-protocol'
              'mapping.ssl_cipher': 'ssl-cipher'
              'mapping.fle_status': 'fle-status'
              'mapping.fle_encrypted_fields': 'fle-encrypted-fields'
              'mapping.c_port': 'c-port'
              'mapping.cs_bytes': 'cs-bytes'
              'mapping.timestamp_ms': 'timestamp(ms)'
              'mapping.origin_fbl': 'origin-fbl'
          Columns:
            - Name: date
              Type: date
            - Name: time
              Type: string
            - Name: x_edge_location
              Type: string
            - Name: sc_bytes
              Type: bigint
            - Name: client_ip
              Type: string
            - Name: cs_method
              Type: string
            - Name: cs_host
              Type: string
            - Name: cs_uri_stem
              Type: string
            - Name: sc_status
              Type: int
            - Name: cs_referrer
              Type: string
            - Name: cs_user_agent
              Type: string
            - Name: cs_uri_query
              Type: string
            - Name: cs_cookie
              Type: string
            - Name: x_edge_result_type
              Type: string
            - Name: x_edge_request_id
              Type: string
            - Name: x_host_header
              Type: string
            - Name: cs_protocol
              Type: string
            - Name: cs_bytes
              Type: bigint
            - Name: time_taken
              Type: float
            - Name: x_forwarded_for
              Type: string
            - Name: ssl_protocol
              Type: string
            - Name: ssl_cipher
              Type: string
            - Name: x_edge_response_result_type
              Type: string
            - Name: cs_protocol_version
              Type: string
            - Name: fle_status
              Type: string
            - Name: fle_encrypted_fields
              Type: int
            - Name: c_port
              Type: int
            - Name: time_to_first_byte
              Type: float
            - Name: x_edge_detailed_result_type
              Type: string
            - Name: sc_content_type
              Type: string
            - Name: sc_content_len
              Type: bigint
            - Name: sc_range_start
              Type: bigint
            - Name: sc_range_end
              Type: bigint
            - Name: asn
              Type: string
            - Name: timestamp_ms
              Type: bigint
            - Name: origin_fbl
              Type: float

Four things worth explaining:

Hyphens versus underscores - this is the fiddliest part of the JSON variant. CloudFront’s v2 JSON keys are hyphenated (client-ip, cs-method, x-edge-location), and Hive column names can’t contain hyphens without backtick-quoting them in every single query. The mapping.* SerDe properties translate between the two: the column stays client_ip, the JSON key stays client-ip, and queries look normal.

client-ip, not c-ip - note the rename. Legacy logs call this field c-ip; v2 calls it client-ip. Anyone migrating a table by find-and-replacing underscores for hyphens gets a column of silent NULLs.

InputFormat and OutputFormat are still required - this caught me out. It’s tempting to assume the JSON SerDe handles everything and omit them, but the SerDe only handles parsing; the input format tells Hadoop how to read the raw bytes off disk. Leave them out and Athena throws HIVE_UNSUPPORTED_FORMAT: Unable to create input format.

use.null.for.invalid.data - CloudFront writes "-" as a sentinel for values that don’t apply to a given request. Numeric columns like fle_encrypted_fields and c_port then fail to parse, and the query dies with BAD_DATA. This table property converts unparseable values to NULL instead.

Deployment

The monitoring stack deploys through the same GitHub Actions pipeline as the website stack, using the OIDC trust relationship set up in part one. Three steps:

  1. Clone repository - actions/checkout
  2. Configure AWS credentials - assume the same role via aws-actions/configure-aws-credentials
  3. Deploy CloudFormation stack - aws-actions/aws-cloudformation-github-deploy
  DeployMonitoring:
    runs-on: ubuntu-latest
    needs: DeployWebsite
    steps:
      - name: Clone repository
        uses: actions/checkout@v7

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ secrets.MONITORING_ROLE_ARN }}
          role-session-name: MaceageMonitoringDeployment
          aws-region: us-east-1

      - name: Deploy CloudFormation Stack
        uses: aws-actions/aws-cloudformation-github-deploy@v2
        with:
          name: MaceageWebsiteMonitoring
          template: deploy/infra-monitoring.yml
          parameter-overrides: "WebsiteDomain=${{ env.WEBSITE_DOMAIN }}"

IAM policy

As with the website stack, the policy was assembled through trial and error - mostly error:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ManageCloudFormationStacks",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DescribeEvents",
                "cloudformation:CreateChangeSet",
                "cloudformation:DescribeChangeSet",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:DeleteChangeSet",
                "cloudformation:DescribeStacks"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ManageDatabase",
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:UpdateDatabase",
                "glue:DeleteDatabase",
                "glue:CreateTable",
                "glue:CreateDatabase",
                "glue:UpdateTable",
                "glue:DeleteTable"
            ],
            "Resource": "*"
        }
    ]
}

Since the tables store only metadata and not the underlying data, this policy is far smaller than the website stack’s monster policy. Note that it contains no S3 permissions at all - the Glue catalog tables merely point at the buckets, and the role deploying them never touches the logs themselves. Running Athena queries happens under my own credentials in the console, outside CI.

Final template

The complete template for the static website monitoring resources:

Parameters:
  WebsiteDomain:
    Description: Domain name of the website (e.g. example.com)
    Type: String

Resources:
  MonitoringDatabase:
    Type: AWS::Glue::Database
    DeletionPolicy: Delete
    Properties:
      CatalogId: !Ref AWS::AccountId
      DatabaseInput:
        Name: !Sub ${WebsiteDomain}-requests

  BucketLogsTable:
    Type: AWS::Glue::Table
    DeletionPolicy: Delete
    Properties:
      CatalogId: !Ref "AWS::AccountId"
      DatabaseName: !Ref MonitoringDatabase
      TableInput:
        Name: bucket_request_logs
        TableType: EXTERNAL_TABLE
        Parameters:
          has_encrypted_data: "false"
          classification: "aws s3 access log"
        StorageDescriptor:
          Location: !Sub s3://${WebsiteDomain}-bucket-requests/
          InputFormat: org.apache.hadoop.mapred.TextInputFormat
          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
          SerdeInfo:
            SerializationLibrary: org.apache.hadoop.hive.serde2.RegexSerDe
            Parameters:
              field.delim: "\t"
              serialization.format: "\t"
              input.regex: '([^ ]*) ([^ ]*) \[(.*?)\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ("[^"]*"|-) (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ("[^"]*"|-) ([^ ]*)(?: ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*))?.*$'
          Columns:
            - Name: bucket_owner
              Type: string
            - Name: bucket_name
              Type: string
            - Name: request_datetime
              Type: string
            - Name: remote_ip
              Type: string
            - Name: requester
              Type: string
            - Name: request_id
              Type: string
            - Name: operation
              Type: string
            - Name: key
              Type: string
            - Name: request_uri
              Type: string
            - Name: http_status
              Type: int
            - Name: error_code
              Type: string
            - Name: bytes_sent
              Type: bigint
            - Name: object_size
              Type: bigint
            - Name: total_time
              Type: int
            - Name: turn_around_time
              Type: int
            - Name: referrer
              Type: string
            - Name: user_agent
              Type: string
            - Name: version_id
              Type: string
            - Name: host_id
              Type: string
            - Name: signature_version
              Type: string
            - Name: cipher_suite
              Type: string
            - Name: authentication_type
              Type: string
            - Name: host_header
              Type: string
            - Name: tls_version
              Type: string
            - Name: access_point_arn
              Type: string
            - Name: acl_required
              Type: string
            - Name: source_region
              Type: string

  CloudfrontLogsTable:
    Type: AWS::Glue::Table
    DeletionPolicy: Delete
    Properties:
      CatalogId: !Ref "AWS::AccountId"
      DatabaseName: !Ref MonitoringDatabase
      TableInput:
        Name: cloudfront_request_logs
        TableType: EXTERNAL_TABLE
        Parameters:
          classification: json
          'use.null.for.invalid.data': 'true'
          'ignore.malformed.json': 'true'
        StorageDescriptor:
          Location: !Sub s3://${WebsiteDomain}-cloudfront-requests/
          StoredAsSubDirectories: true
          InputFormat: org.apache.hadoop.mapred.TextInputFormat
          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
          SerdeInfo:
            SerializationLibrary: org.openx.data.jsonserde.JsonSerDe
            Parameters:
              'mapping.client_ip': 'client-ip'
              'mapping.cs_method': 'cs-method'
              'mapping.cs_host': 'cs-host'
              'mapping.cs_uri_stem': 'cs-uri-stem'
              'mapping.cs_referrer': 'cs-referrer'
              'mapping.cs_user_agent': 'cs-user-agent'
              'mapping.cs_uri_query': 'cs-uri-query'
              'mapping.cs_cookie': 'cs-cookie'
              'mapping.cs_protocol': 'cs-protocol'
              'mapping.cs_protocol_version': 'cs-protocol-version'
              'mapping.sc_bytes': 'sc-bytes'
              'mapping.sc_status': 'sc-status'
              'mapping.sc_content_type': 'sc-content-type'
              'mapping.sc_content_len': 'sc-content-len'
              'mapping.sc_range_start': 'sc-range-start'
              'mapping.sc_range_end': 'sc-range-end'
              'mapping.x_edge_location': 'x-edge-location'
              'mapping.x_edge_result_type': 'x-edge-result-type'
              'mapping.x_edge_request_id': 'x-edge-request-id'
              'mapping.x_edge_response_result_type': 'x-edge-response-result-type'
              'mapping.x_edge_detailed_result_type': 'x-edge-detailed-result-type'
              'mapping.x_host_header': 'x-host-header'
              'mapping.x_forwarded_for': 'x-forwarded-for'
              'mapping.time_taken': 'time-taken'
              'mapping.time_to_first_byte': 'time-to-first-byte'
              'mapping.ssl_protocol': 'ssl-protocol'
              'mapping.ssl_cipher': 'ssl-cipher'
              'mapping.fle_status': 'fle-status'
              'mapping.fle_encrypted_fields': 'fle-encrypted-fields'
              'mapping.c_port': 'c-port'
              'mapping.cs_bytes': 'cs-bytes'
              'mapping.timestamp_ms': 'timestamp(ms)'
              'mapping.origin_fbl': 'origin-fbl'
          Columns:
            - Name: date
              Type: date
            - Name: time
              Type: string
            - Name: x_edge_location
              Type: string
            - Name: sc_bytes
              Type: bigint
            - Name: client_ip
              Type: string
            - Name: cs_method
              Type: string
            - Name: cs_host
              Type: string
            - Name: cs_uri_stem
              Type: string
            - Name: sc_status
              Type: int
            - Name: cs_referrer
              Type: string
            - Name: cs_user_agent
              Type: string
            - Name: cs_uri_query
              Type: string
            - Name: cs_cookie
              Type: string
            - Name: x_edge_result_type
              Type: string
            - Name: x_edge_request_id
              Type: string
            - Name: x_host_header
              Type: string
            - Name: cs_protocol
              Type: string
            - Name: cs_bytes
              Type: bigint
            - Name: time_taken
              Type: float
            - Name: x_forwarded_for
              Type: string
            - Name: ssl_protocol
              Type: string
            - Name: ssl_cipher
              Type: string
            - Name: x_edge_response_result_type
              Type: string
            - Name: cs_protocol_version
              Type: string
            - Name: fle_status
              Type: string
            - Name: fle_encrypted_fields
              Type: int
            - Name: c_port
              Type: int
            - Name: time_to_first_byte
              Type: float
            - Name: x_edge_detailed_result_type
              Type: string
            - Name: sc_content_type
              Type: string
            - Name: sc_content_len
              Type: bigint
            - Name: sc_range_start
              Type: bigint
            - Name: sc_range_end
              Type: bigint
            - Name: asn
              Type: string
            - Name: timestamp_ms
              Type: bigint
            - Name: origin_fbl
              Type: float

References

Wrapping up

Two Glue tables and a database turn two buckets of raw log files into something queryable from the Athena console, no servers required - but the path there is littered with quiet failure modes:

  • Glue databases and tables hold metadata only, so they’re cheap and safe to DeletionPolicy: Delete
  • S3 access logs need a RegexSerDe table whose column count matches the regex capture groups exactly - and S3 has appended fields to that format over time
  • CloudFront v2 logs need a JsonSerDe table with mapping.* properties for the hyphenated keys, plus explicit InputFormat/OutputFormat or you get HIVE_UNSUPPORTED_FORMAT
  • CloudFront’s - sentinel breaks numeric columns unless you set use.null.for.invalid.data
  • Glacier Instant Retrieval logs stay queryable, so the part-one lifecycle rules don’t sacrifice Athena access - only Flexible Retrieval and Deep Archive need a restore plus read_restored_glacier_objects

Share

X Facebook LinkedIn Bluesky

Related Posts

Static Website Architecture

2026-01-07

AWS Solutions Architect - Professional

2024-07-03

System Design Learnings

2024-08-30