GitHub Actions Proxy Scraping: Secrets and Scheduling
GitHub Actions proxy scraping works for scheduled jobs. Set up secrets, Python requests, cron triggers, validated output, and safe failure handling.
Instead of manually launching a scraper every day, configure GitHub Actions proxy scraping with repository secrets, a scheduled workflow, and a saved result artifact. This guide gives you a runnable Python setup that fails on unsuccessful responses instead of silently publishing bad data.
TL;DR
- GitHub Actions proxy scraping works well for periodic jobs that tolerate scheduling delays.
- Keep proxy credentials in GitHub Actions secrets, not source files or workflow YAML.
- Validate response content before saving results; an HTTP success code does not prove collection succeeded.
What a Schedule Does and Does Not Guarantee
A schedule does not make a scraper reliable. You need separate controls for authentication, request failures, overlapping runs, and output retention. Otherwise, a green workflow can hide an empty dataset.
Use GitHub Actions for periodic collection, not exact-time execution. Scheduled workflows can run late, and GitHub documents that scheduled events can be dropped during high load. For a data pipeline with strict delivery deadlines, choose infrastructure with scheduling commitments that match those deadlines.
This example collects one HTML page through an authenticated proxy and extracts its title. It does not render JavaScript, solve challenges, or bypass access controls. Check the destination's terms and your authorization before scheduling collection.
If proxy authentication is new to you, start with the curl authenticated proxy workflow. Establish that the endpoint works before adding a scheduler.
Before you start
- Prepare a GitHub repository where you can edit workflows and manage Actions secrets. Confirm that repository or organization policies permit the actions used below.
- Have an authenticated proxy URL and an HTTPS page you are authorized to fetch. A Node4 datacenter proxy endpoint works well for this. The proxy must accept connections from your runner; a restricted source-IP policy needs separate planning.
- Put the workflow on the repository's default branch before testing its schedule. GitHub runs scheduled workflows from that branch, not from an arbitrary working branch.
The example uses username-and-password proxy authentication. Percent-encode reserved characters inside the username and password before assembling the URL. A literal @ inside a password otherwise changes how a URL parser interprets the address.
Use a private repository if your collection target or output is sensitive. A secret protects the stored credential, but it does not make downloaded artifacts or repository contents private.
Repository credentials
- Open the repository's Settings tab.
- Select Secrets and variables, then Actions.
- On Secrets, select New repository secret. Set Name to
PROXY_URLand enter the complete proxy connection URL under Secret. Select Add secret. - On Variables, select New repository variable. Set Name to
SCRAPE_URLand enter the authorized HTTPS page under Value. Select Add variable.
Copy the endpoint, port, and authentication details from your provider. Do not infer them from a product name. The code accepts an HTTP proxy URL or a SOCKS proxy URL supported by Requests and its installed SOCKS dependency.
For SOCKS with proxy-side destination DNS resolution, use the socks5h scheme. Plain socks5 resolves the destination locally in Requests. That distinction matters when you are checking DNS behavior, not just the HTTP exit address.
Expected result: GitHub lists PROXY_URL as a repository secret and SCRAPE_URL as a repository variable. Neither credential nor password appears in a committed file.
Do not use a repository variable for the proxy password. Variables are configuration values, not a substitute for secrets. If the target URL contains a private token, store that URL as a secret too and change the workflow reference accordingly.
Python request and output validation
- Create
scrape.pyin the repository root. - Paste the script below.
- Commit the file without adding local credential files.
import json
import os
import sys
from datetime import datetime, timezone
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import urlsplit
import requests
class TitleParser(HTMLParser):
def __init__(self):
super().__init__()
self.inside_title = False
self.parts = []
def handle_starttag(self, tag, attrs):
if tag == "title":
self.inside_title = True
def handle_endtag(self, tag):
if tag == "title":
self.inside_title = False
def handle_data(self, data):
if self.inside_title:
self.parts.append(data)
def main():
target = os.environ.get("SCRAPE_URL", "").strip()
proxy = os.environ.get("PROXY_URL", "").strip()
parsed = urlsplit(target)
if parsed.scheme != "https" or not parsed.hostname:
raise ValueError("SCRAPE_URL must be an HTTPS URL")
if parsed.username or parsed.password:
raise ValueError("Do not put target credentials in SCRAPE_URL")
if not proxy:
raise ValueError("PROXY_URL is required")
with requests.Session() as session:
session.trust_env = False
session.proxies = {"http": proxy, "https": proxy}
response = session.get(
target,
timeout=(10, 30),
allow_redirects=False,
headers={"User-Agent": "ScheduledCollector/1.0"},
)
if response.status_code != 200:
raise ValueError(f"Unexpected HTTP status: {response.status_code}")
content_type = response.headers.get("Content-Type", "").lower()
if "text/html" not in content_type:
raise ValueError("Expected an HTML response")
parser = TitleParser()
parser.feed(response.text)
title = " ".join("".join(parser.parts).split())
if not title:
raise ValueError("Expected a nonempty page title")
record = {
"collected_at": datetime.now(timezone.utc).isoformat(),
"host": parsed.hostname,
"title": title,
}
Path("output").mkdir(exist_ok=True)
Path("output/result.json").write_text(
json.dumps(record, ensure_ascii=False, indent=2),
encoding="utf-8",
)
print("Saved one validated record")
if __name__ == "__main__":
try:
main()
except (requests.RequestException, ValueError):
print("Collection failed; check configuration and response validation", file=sys.stderr)
sys.exit(1)The request has a 10-second connection timeout and a 30-second read timeout. These are selected configuration values, not measured proxy performance. The read timeout limits waiting between received data, not the total download duration.
Setting trust_env to False prevents ambient proxy environment settings from changing this session's routing. Both HTTP and HTTPS destinations use the explicitly configured proxy. The script refuses to proceed without it.
Redirects are disabled deliberately. A login redirect should fail this baseline rather than become a successful collection of the wrong page. If the authorized destination redirects normally, configure its final HTTPS URL after checking the redirect yourself.
Expected result: a successful request creates output/result.json containing a UTC collection timestamp, the target hostname, and a nonempty title. A missing title, unexpected content type, or unsuccessful status stops the process.
A challenge page can still contain a title and return status 200. Replace the title check with a target-specific field or known page marker before treating this example as a production extractor. Never equate transport success with useful data.
Scheduled workflow configuration
- Create
.github/workflows/scrape.yml. - Paste the workflow below.
- Commit it to the default branch.
name: Scheduled proxy scrape
on:
workflow_dispatch:
schedule:
- cron: '17 */6 * * *'
permissions:
contents: read
concurrency:
group: scheduled-proxy-scrape
cancel-in-progress: false
jobs:
scrape:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: python -m pip install 'requests[socks]'
- name: Collect page
env:
PROXY_URL: ${{ secrets.PROXY_URL }}
SCRAPE_URL: ${{ vars.SCRAPE_URL }}
run: python scrape.py
- name: Save validated output
uses: actions/upload-artifact@v4
with:
name: scrape-result-${{ github.run_id }}-${{ github.run_attempt }}
path: output/result.json
if-no-files-found: error
retention-days: 7The cron expression requests execution every 6 hours, at minute 17, using UTC. That means four scheduled opportunities each day, not four guaranteed completed collections. Treat cron timing as a target, not a delivery guarantee.
The job has a 10-minute execution limit. Artifact retention is set to 7 days, subject to the repository's permitted retention settings. Change both values to match your collection and retention requirements.
The concurrency group prevents this workflow's jobs from executing together. With cancellation disabled, an active run continues rather than being interrupted by a new one. GitHub concurrency is not a durable queue for every pending collection, so this pattern is unsuitable when every interval must be preserved.
This runnable baseline installs dependencies without a lockfile. For a reproducible deployment, commit tested dependency pins and pin actions to reviewed commit SHAs. Floating package resolution and major-version action tags are convenient, but they are not immutable build inputs.
Expected result: the workflow appears under Actions, supports manual execution, and has a scheduled trigger. It saves an artifact only after the collection step succeeds.
Manual verification and the update-trigger variant
- Open Actions and select Scheduled proxy scrape.
- Select Run workflow, choose the default branch, and select Run workflow again.
- Open the run and inspect Collect page. Confirm that it reports a saved record.
- Download the artifact and inspect its timestamp, hostname, and extracted title.
A passing run proves that this script received and validated a response. It does not independently prove the public exit address. For that check, use an endpoint you control that records the connecting address, then compare its logs with your intended proxy route.
Expected result: the artifact contains the intended page's title, not a login prompt or challenge message. Verify this manually before trusting unattended execution.
Run when scraper configuration changes
Best for: validating approved changes immediately instead of waiting for cron. Add a push trigger under the existing on mapping:
push:
branches: [main]
paths:
- scrape.py
- .github/workflows/scrape.ymlReplace main if your default branch has another name. This variant reuses the same job, secrets, validations, and artifact handling. Its tradeoff is an additional collection whenever a matching change reaches that branch.
Keep credential-bearing collection off untrusted pull-request code. In particular, do not combine pull_request_target, privileged secrets, and checkout of an untrusted contribution. Review collection changes before they reach a trusted execution branch.
Troubleshooting
The schedule never starts
Check that the workflow exists on the default branch and that Actions is enabled. Inspect repository and organization policies, plus whether GitHub has disabled scheduled execution after repository inactivity. A successful manual run does not prove the schedule is active.
Proxy authentication fails
An HTTP 407 indicates that the proxy requires valid authentication. Check the stored endpoint, credentials, and percent-encoding. Do not paste the complete connection URL into logs or an issue; rotate the secret if it has been exposed.
The target returns 403 or 429
A 403 is a refusal, not proof that the proxy is broken. A 429 signals rate limiting. Check authorization and documented limits, respect retry instructions, and reduce collection frequency instead of automatically cycling identities to defeat a restriction.
The artifact is missing
The upload step runs only after a successful collection. Inspect the first failed step rather than debugging artifact storage first. An absent output file should remain a failure, not become an empty dataset uploaded with a green result.
The runner cannot reach the proxy
Check the hostname, port, account restrictions, and any source-IP restriction. GitHub-hosted runner egress addresses are not a single permanent address. If your authentication requires fixed egress, use a runner configuration with controlled networking or another supported authentication method.
Customize your workflow
Best for: small, independent collections that finish within a bounded job. The advantage is a versioned setup with visible execution history; the limitation is that each run starts without your previous runtime state.
For a production workflow, decide where durable state belongs before increasing request volume. Artifacts are useful for inspection, but they are not a transactional checkpoint store. Put deduplication keys and collection cursors in storage designed for those writes.
Keep credentials out of artifacts as well as logs. Store only the fields your downstream process requires, and exclude authorization headers, cookies, and private URL query strings. Artifact retention does not replace a data-handling policy.
When a single-page job becomes a batch, add explicit per-host limits and target-specific validation before adding concurrency. The aiohttp high-concurrency workflow covers that next implementation step. More simultaneous requests change connection demand; they do not fix an incorrect parser.
FAQ
How do I set up GitHub Actions proxy scraping?
Store the proxy URL as an Actions secret, pass it into a collection step, and configure the HTTP client to use it explicitly. Add a scheduled trigger and validate the output before uploading an artifact.
Can GitHub Actions run my scraper at an exact time?
GitHub Actions scheduled workflows do not guarantee exact-time execution. Schedule delays and dropped events under high load make this setup unsuitable for strict collection deadlines.
Where should I put proxy credentials in GitHub Actions?
Put proxy credentials in an Actions secret, not a repository variable or committed file. Expose the secret only to the step that requires it, and avoid printing connection URLs or raw exception details.
Can I use a SOCKS5 proxy with this Python workflow?
Yes, the workflow installs Requests with its SOCKS dependency. Use socks5h when destination DNS resolution should happen through the proxy, and use the endpoint and credentials supplied for your account.
Why does my scheduled scraper get a 403 response?
A 403 means the destination refused the request. Check access permissions, authentication, and destination policies; changing the schedule or proxy does not establish permission to collect the page.
Will this workflow scrape a JavaScript-rendered page?
This Requests-based workflow does not execute JavaScript. Use an authorized data endpoint or a browser-based collector when the required content is absent from the initial HTML response.
Does a successful workflow prove the proxy exit address?
No, a successful workflow proves only that its configured checks passed. Verify the exit separately with a controlled endpoint that records the connecting address.
Test a Known-Bad Page Before Trusting the Schedule
A green run is only as meaningful as its strictest data check. During your workflow review, temporarily point the scraper at an authorized page that lacks the expected field. Confirm that collection fails and no result artifact appears. That negative test catches a dangerous mistake: a pipeline that reports success when the data is wrong.