Jobs & Webhooks - Scrapebento Docs

Sync vs async

Every metered search and scrape endpoint goes through the same job system.

wait: true is the default
a fast terminal result returns 200
a slower request returns 202 with jobId, status, pollUrl, and creditsCharged

Example accepted response:

{
  "data": {
    "jobId": "job_123",
    "status": "queued",
    "pollUrl": "/v1/public/jobs/job_123",
    "creditsCharged": 1
  }
}

Polling

Poll GET /v1/public/jobs/{jobId} with the same bearer token.

202 means the job is still queued or running
200 with status: "succeeded" includes the terminal result
200 with status: "failed" includes a terminal error

Successful terminal payloads may also include:

completedAt
resultTruncated
cached

Webhooks

Send webhookUrl with any scrape and the worker POSTs the terminal job payload to that public http(s) URL once the result is persisted. Available to every organization.

invalid, private, loopback, and link-local callback URLs are rejected up front
delivery is best-effort with bounded retries
a delivery failure never removes the stored result; the job stays pollable
every attempt sequence is recorded in the delivery log, and can be re-sent

Endpoints and secrets

Each destination is a webhook endpoint with its own signing secret, so the secret you need in order to verify your deliveries cannot forge anybody else’s. Naming a webhookUrl on a job registers that URL if it is not registered already — the secret exists before the first delivery does. To have it in hand before you run anything, register up front:

curl -X POST https://api.scrapebento.com/v1/public/webhooks/endpoints \
  -H "Authorization: Bearer satk_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hooks.example.com/scrapebento"}'

{
  "data": {
    "endpoint": {
      "id": "whep_9f2c...",
      "url": "https://hooks.example.com/scrapebento",
      "status": "active",
      "signingSecret": "whsec_..."
    }
  }
}

Registering a URL that is already registered returns the existing endpoint and its current secret rather than issuing a new one — replacing it would break a receiver that is already verifying deliveries.

Operation	Call
List endpoints	`GET /v1/public/webhooks/endpoints`
Read the secret	`GET /v1/public/webhooks/endpoints/{id}/secret`
Rotate the secret	`POST /v1/public/webhooks/endpoints/{id}/rotate`
Pause / resume	`POST /v1/public/webhooks/endpoints/{id}/status`
Stop and remove	`DELETE /v1/public/webhooks/endpoints/{id}`

Rotation takes effect immediately. There is no grace period during which the old secret still verifies — a rotation is usually the response to a leak, and a window where the leaked secret keeps working is the opposite of what was asked for. Deploy the new secret to your receiver first, or accept a brief window of rejected deliveries and replay them from the log.

A disabled or deleted endpoint receives nothing. Those attempts are recorded as skipped rather than dropped, so a gap in your notifications has a visible cause.

Verifying signatures

Every delivery carries:

Header	Meaning
`X-Scrapebento-Event`	Event type, currently always `scrape.completed`. Route on it before parsing.
`X-Scrapebento-Delivery`	This delivery’s id, the same one in the delivery log.
`X-Scrapebento-Timestamp`	Unix seconds at delivery time.
`X-Scrapebento-Signature`	`sha256=<hex>` — HMAC-SHA256 of `"<timestamp>.<raw body>"` under the endpoint’s secret.

To verify: recompute the HMAC over the received timestamp, a ., and the raw request body; compare with a constant-time compare; and reject deliveries whose timestamp falls outside a freshness window (±5 minutes recommended). The timestamp is inside the signed material, which is what stops a captured delivery being replayed at you later.

import hashlib, hmac, time

def verify(secret: str, headers, raw_body: bytes) -> bool:
    timestamp = headers["X-Scrapebento-Timestamp"]
    if abs(time.time() - int(timestamp)) > 300:
        return False
    expected = "sha256=" + hmac.new(
        secret.encode(), f"{timestamp}.".encode() + raw_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, headers["X-Scrapebento-Signature"])

Sign over the bytes you received, not over a re-serialized parse of them: any difference in key order or spacing changes the digest. The older X-Scrape-Signature and X-Scrape-Signature-V2 headers are deprecated and are no longer sent to customer endpoints.

Body shape

{
  "type": "scrape.completed",
  "jobId": "job_123",
  "kind": "url",
  "status": "succeeded",
  "input": { "url": "https://example.com" },
  "result": {},
  "creditsCharged": 1
}

Failures are delivered too, with status: "failed" and an errorCode — a caller waiting on a callback should learn about a failure as promptly as a success, not by timing out. Deduplicate on jobId. A delivery can arrive more than once (a retry, or a replay), and the status is terminal, so a repeat carries the same outcome.

The delivery log

GET /v1/public/webhooks/deliveries returns every attempt sequence, newest first: where it went, what your endpoint answered, and how many attempts it took. Filter by jobId or status, and page with the returned nextBefore. This is what makes a missed notification diagnosable rather than a support thread. Match the X-Scrapebento-Delivery header your receiver logged against id here. To re-send one:

curl -X POST https://api.scrapebento.com/v1/public/webhooks/deliveries/whd_.../replay \
  -H "Authorization: Bearer satk_..."

A replay makes one attempt, synchronously, and returns what your endpoint said. The retry ladder automatic delivery uses is not applied — calling replay again is the retry.

The body is rebuilt from the job rather than replayed from a stored copy, so a replay is exactly what polling the job would return right now. Compare payloadSha256 in the log to tell whether it differs from the original. If the job’s result has aged past its retention window, replay returns 410 result_unavailable rather than delivering an empty body and calling it a success.

Idempotency

Use idempotencyKey when a caller might retry the same submission.

repeated requests with the same key return the same job record
the worker is not duplicated for a winning request
billing is intended to follow the same de-duplicated job outcome

​Sync vs async

​Polling

​Webhooks

​Endpoints and secrets

​Verifying signatures

​Body shape

​The delivery log

​Idempotency

Sync vs async

Polling

Webhooks

Endpoints and secrets

Verifying signatures

Body shape

The delivery log

Idempotency