Skip to main content

Observability service

Log processing​

  • A lambda is triggered lambda/log-group/index.ts every time we receive a cloudtrail event for a new log creation.
  • This lambda then sets the retention policy automatically.
  • The lambda also sets a subscription filter on the log group to a central lambda which pipes all the logs over to Elastic Cloud.

Env vars​

  • IS_LOG_GROUP_AUTOSUBSCRIBE_ENABLED - This automatic subscription flow is only enabled if this env var is set to true. If disabled, new log groups won't automatically be subscribed and have the retention policy set.
  • LOG_GROUP_DAYS_OF_RETENTION - The number of days of log retention to set on new log groups. If not supplied, we default to a value of 30.
  • LOGS_LAMBDA_CONSUMER_ARN - The ARN of the lambda to forward our logs to, in this case it will be set to the Elastic function beat

Backfilling our log groups to set retention policy​

  • Local script (scripts/log-groups/set-log-retention.ts) to be invoked locally which fetches all our lambda log groups and sets the retention policy to the provided value.

Example usage:​

cd scripts
npm run backfill:log-retention -- --region eu-west-2 --retention 7

Backfilling our log groups to set subscription filters​

  • Local script (scripts/log-groups/set-subscription-filters.ts) to be invoked locally which fetches all our lambda log groups and sets the subscription filter.

Example usage:​

cd scripts
npm run backfill:log-subscription-filters -- --region eu-west-2 --lambdaDestinationArn arn:aws:lambda:eu-west-2:527132486578:function:elasticCloudwatchShipLogs

Backfilling our log groups to delete subscription filters​

  • Local script (scripts/log-groups/delete-subscription-filters.ts) to be invoked locally which deletes all subscription filters on our log groups.

Example usage:​

cd scripts
npm run delete:log-subscription-filters -- --region eu-west-2