AWS Lambda Proxy Scraping: Secrets, S3 and Schedules

Fetch an approved page through an authenticated proxy from Lambda, store credentials in Secrets Manager, save output to S3, then add a schedule.

Instead of manually launching scraping scripts, use AWS Lambda proxy scraping to fetch an authorized page through an authenticated proxy and save the response to Amazon S3. This workflow builds a bounded Python worker, keeps credentials in AWS Secrets Manager, and adds a schedule only after the first invocation passes.

TL;DR

  • AWS Lambda proxy scraping fits bounded HTTP jobs; use Secrets Manager for credentials and S3 for results.
  • Validate the saved output against an expected page marker before trusting it, then add a schedule.
  • Start with one URL per invocation, explicit timeouts, and restricted target hosts.
  • Test proxy authentication before adding a recurring schedule.

The authenticated proxy requests guide covers the connection check separately. A working proxy connection comes before Lambda configuration.

What Lambda does and does not supply

Lambda runs your code, but it does not supply a proxy exit, maintain browser sessions, or decide whether a response contains usable data. Those responsibilities remain in your application.

For a serverless scraping deployment, separate execution, network identity, and storage. Lambda handles execution. Your proxy handles outbound forwarding. S3 holds the captured response after the function exits.

This guide uses plain HTTP requests, not browser rendering. It assumes you have permission to collect the target content and that the page returns useful HTML without executing JavaScript. Respect access restrictions and published request limits; changing an exit address does not grant permission.

Before you start

  • Prepare an AWS account with permission to configure Lambda, IAM, Secrets Manager, S3, and CloudWatch Logs. Prepare an authenticated HTTP proxy endpoint separately, such as a Node4 datacenter proxy.
  • Choose an approved HTTPS target, its exact hostname, and an S3 bucket in the function's Region. Install Python 3.12, pip, and zip locally for packaging.
  • Check the network path first. This walkthrough leaves Lambda outside your VPC. Attaching a function to a public subnet does not give it a public IP address or internet access; a private-network deployment needs an appropriate outbound route.

The examples deliberately reject redirects and unapproved hosts. Do not expose this worker as a public endpoint accepting arbitrary URLs. A scraping function with unrestricted URL input creates a server-side request forgery risk.

Proxy credentials and storage

Create the secret and output destination

  1. In S3, choose Create bucket and create a bucket for scraping output. Keep public access blocked. Record the bucket name.
  2. In Secrets Manager, choose Store a new secret, then Other type of secret. Add a key named proxy_url whose value is your complete authenticated proxy URL.
  3. Use the endpoint, port, and authentication format supplied for your proxy. For this Requests example, use an HTTP proxy URL with credentials, not a SOCKS endpoint.
  4. Record the secret ARN. Do not place the credential in the Lambda test event, source archive, or logs.

The secret value uses this structure:

{
  "proxy_url": "http://ENCODED_USERNAME:ENCODED_PASSWORD@PROXY_HOST:PROXY_PORT"
}

Replace every uppercase token before saving. Percent-encode reserved characters in the username and password individually; do not encode the whole URL. An unencoded @ in a password can change how a client parses the proxy address.

With an HTTP proxy, an HTTPS destination normally uses a CONNECT tunnel. The destination connection still uses TLS, but the proxy connection is not itself TLS-protected merely because the target is HTTPS. Use a trusted network path and keep certificate verification enabled.

Expected result: You have a private output bucket and a secret containing a usable proxy URL. Neither resource depends on a Lambda invocation succeeding.

Lambda execution configuration

Create a bounded worker

  1. In Lambda, choose Create function, then Author from scratch. Select Python 3.12 and create an execution role for this function.
  2. Give the role CloudWatch logging permissions. Add secretsmanager:GetSecretValue for the specific secret ARN and s3:PutObject for the bucket's scrapes/* prefix. Customer-managed encryption keys require the corresponding key permissions too.
  3. Under Configuration, open Environment variables and choose Edit. Add the values below.
  4. Under General configuration, choose Edit and set the timeout to 60 seconds. Under Concurrency, choose Edit and reserve 2 concurrent executions as an initial guardrail.
PROXY_SECRET_ID=<secret ARN>
OUTPUT_BUCKET=<bucket name>
ALLOWED_HOSTS=<approved hostname>

Separate multiple approved hostnames with commas. Enter hostnames only, without schemes or paths. Include each required subdomain explicitly rather than allowing every subdomain automatically.

These are conservative starting settings for this walkthrough, not performance benchmarks. The code uses a 5-second connection timeout, a 20-second read timeout, and a 2 MiB response ceiling. Adjust those limits only after measuring your authorized workload.

A Requests read timeout limits waiting between reads, not the total download duration. The Lambda timeout remains the outer execution boundary. Reserved concurrency limits simultaneous function executions; it does not enforce a target site's requests-per-second limit.

Expected result: The function can write logs, retrieve one secret, and write output under one prefix. It cannot read arbitrary secrets merely because it is a scraping worker.

Package the handler

Create lambda_function.py with the following code. The example fetches one page, rejects redirects, and writes only successful responses. It intentionally implements no automatic retry loop.

import hashlib
import json
import os
from urllib.parse import urlsplit

import boto3
import requests

secrets = boto3.client("secretsmanager")
s3 = boto3.client("s3")
MAX_BYTES = 2 * 1024 * 1024


def lambda_handler(event, context):
    target = event["target_url"]
    task_id = event["task_id"]
    if not isinstance(task_id, str) or not task_id:
        raise ValueError("task_id must be a nonempty string")

    parsed = urlsplit(target)
    allowed = {
        host.strip().lower()
        for host in os.environ["ALLOWED_HOSTS"].split(",")
        if host.strip()
    }
    if (
        parsed.scheme != "https"
        or parsed.hostname not in allowed
        or parsed.port not in (None, 443)
        or parsed.username is not None
        or parsed.password is not None
        or parsed.fragment
    ):
        raise ValueError("Target is not permitted")

    secret = secrets.get_secret_value(
        SecretId=os.environ["PROXY_SECRET_ID"]
    )
    proxy = json.loads(secret["SecretString"])["proxy_url"]
    if urlsplit(proxy).scheme != "http":
        raise ValueError("This example requires an HTTP proxy URL")

    with requests.Session() as session:
        session.trust_env = False
        session.proxies = {"http": proxy, "https": proxy}
        try:
            with session.get(
                target,
                timeout=(5, 20),
                allow_redirects=False,
                stream=True,
            ) as response:
                print(json.dumps({"http_status": response.status_code}))
                if not 200 <= response.status_code < 300:
                    raise RuntimeError("Target returned a non-success status")
                body = bytearray()
                for chunk in response.iter_content(chunk_size=65536):
                    body.extend(chunk)
                    if len(body) > MAX_BYTES:
                        raise ValueError("Response exceeds configured limit")
                content_type = response.headers.get(
                    "Content-Type", "application/octet-stream"
                )
        except requests.RequestException:
            raise RuntimeError("Proxy fetch failed; inspect network settings") from None

    digest = hashlib.sha256(
        (task_id + "\n" + target).encode("utf-8")
    ).hexdigest()
    key = f"scrapes/{digest}.body"
    s3.put_object(
        Bucket=os.environ["OUTPUT_BUCKET"],
        Key=key,
        Body=bytes(body),
        ContentType=content_type,
    )
    return {"key": key, "bytes": len(body)}

Package the dependencies alongside the handler, not in a parent directory:

mkdir -p package
python3.12 -m pip install requests boto3 --target package
cp lambda_function.py package/
(cd package && zip -qr ../function.zip .)

In the function's Code tab, choose Upload from, then .zip file, and upload function.zip. Set Handler under Runtime settings to lambda_function.lambda_handler.

For repeatable deployments, retain a tested dependency lockfile rather than resolving new versions on every build. Build any future native dependencies for the function's operating system and architecture.

Expected result: The deployed archive contains lambda_function.py, Requests, and its dependencies at the archive root. Importing the handler does not require a layer.

Invocation test and output checks

Run one controlled request

  1. Open Test and create a test event containing target_url and a nonempty task_id. Use your approved HTTPS target and a unique identifier for this capture.
  2. Invoke the function once. Read the returned S3 key and byte count, then open the corresponding object.
  3. Inspect the saved content. A successful HTTP status does not prove that the response is the intended page rather than a login screen or challenge.
  4. Invoke the same event again. Confirm that the same task identifier and target produce the same object key.

Expected result: The function returns an object key only after S3 accepts the write. A failed fetch produces no successful return value.

The deterministic key prevents duplicate object names for repeated delivery of the same logical task. It does not stop another network request from firing, it does not establish exactly-once execution, and it does not preserve earlier object versions. Reusing a task identifier overwrites the same key unless your bucket configuration preserves versions.

Before scheduling, add a content check appropriate to your target. Require an expected page marker or parse a required field before accepting the response. For structured extraction, save the raw response separately from the parsed record so parser failures remain diagnosable.

Keep logs narrow. This handler records HTTP status without logging the target URL, credentials, or page body. Query strings and collected pages can contain sensitive information.

Second workflow: scheduled collection

For recurring collection, use Amazon EventBridge Scheduler to invoke the same worker. Keep the fetching logic unchanged; change the task identity supplied with each scheduled run.

  1. Create a Scheduler execution role trusted by scheduler.amazonaws.com. Allow lambda:InvokeFunction only on this function's ARN.
  2. In EventBridge Scheduler, choose Create schedule and configure a recurring schedule. For an initial controlled test, choose a rate of 1 hour.
  3. Select the Lambda function as the target. Supply the same approved target URL and use <aws.scheduler.scheduled-time> as the task_id value in the target input.
  4. Configure retry behavior deliberately. Start without automatic retries while validating the worker, then introduce bounded retries and a dead-letter queue if the workload requires them.

Scheduler substitutes its scheduled-time context value in the target input. A scheduled timestamp distinguishes successive captures while keeping the logical identity tied to the scheduled event.

Check both delivery layers. Scheduler delivery failures and Lambda asynchronous execution failures are different paths. Configure Lambda's asynchronous retry and failure-destination settings too; disabling Scheduler retries alone does not disable Lambda's execution retries.

For a deployment, keep a named owner for failed runs and retained output. A schedule without failure handling is unattended execution, not a reliable collection pipeline.

Troubleshooting

Proxy authentication fails

Check the endpoint, port, encoded credentials, and authentication method outside Lambda first. HTTP 407 indicates proxy authentication is required or rejected. For HTTPS tunnels, the client can surface that failure as a proxy exception instead of a normal response.

Requests time out only inside Lambda

Inspect VPC attachment, outbound routes, security groups, and network ACLs. Confirm that any source-IP restriction at the proxy matches the function's actual egress path. A function without fixed egress cannot satisfy a fixed source-IP restriction reliably.

The target returns 403 or 429

Do not add immediate retries or assume another exit grants access. Inspect the target's rules and response headers. Reduce dispatch rate, honor applicable Retry-After guidance, and stop collection when authorization or access is denied.

The function cannot read or write AWS resources

Check the execution role against the exact secret ARN and object prefix. Also inspect resource policies and encryption-key permissions. A bucket policy can deny a write even when the role grants s3:PutObject.

The saved response is empty or incomplete

Inspect content type, encoding, and the configured size limit. A JavaScript application shell needs a different collection approach; increasing the timeout does not turn Requests into a browser. Keep browser dependencies out of this worker unless rendering is actually required.

Customize your workflow

Add rate control before adding parallelism. For larger batches, place approved tasks in a queue and configure the consumer's batch size and concurrency explicitly. Account for retries when calculating target request volume.

A warm Lambda environment is not a durable session store. If your workflow requires cookies or a stable proxy session, manage that state explicitly and keep sensitive session material encrypted. An unchanged function instance does not prove an unchanged proxy exit.

The main trade-off is operational simplicity versus execution boundaries. This worker avoids managing a persistent server, but it is not a fit for indefinite crawls, unrestricted downloads, or persistent browser sessions. Split bounded jobs rather than extending a single invocation indefinitely.

FAQ

Can AWS Lambda scrape a website through a proxy?

Yes. A Lambda function can send HTTP requests through a configured proxy when its network path can reach that proxy. The target must permit your collection, and proxy access does not replace authorization.

Does AWS Lambda provide a rotating proxy automatically?

No. Lambda supplies execution infrastructure, not a configured rotating proxy service. Configure the proxy endpoint and authentication separately.

Where should I store proxy credentials for Lambda?

Store proxy credentials in Secrets Manager and grant the function access to the specific secret. Keep credentials out of source archives, test events, and logs.

Why does my Lambda scraper time out in a public subnet?

Attaching Lambda to a public subnet does not give the function a public IP address. A VPC-connected function needs a suitable outbound network path to reach an external proxy.

Can I scrape JavaScript-rendered pages with this Python handler?

This handler fetches HTTP response content without running JavaScript. Use an authorized data endpoint or a separately packaged browser workflow when rendering is required.

Will reserved concurrency prevent website rate limits?

No. Reserved concurrency limits simultaneous Lambda executions, not requests per second to a destination. Apply target-specific dispatch limits and account for retry traffic.

Does the deterministic S3 key prevent duplicate scraping requests?

No. The deterministic key makes repeated delivery of the same task write to the same object name. Preventing repeat network requests requires additional task-state coordination.

Test the rejection path before the acceptance path

Before enabling the production schedule, deliberately submit an unapproved hostname. The function should reject it before making a request. Then test an oversized response on infrastructure you control.

A useful acceptance test proves that the worker refuses bad work, not just that it completes good work. Keep those negative tests alongside the successful capture test.