Cofacts URL Resolver

A gRPC service that scrapes the specified URL and returns scraped result and summary extracted by [Readability.js]

Usage

First, create an .env file from .env.sample.

Run url resolver from built docker images

$ docker pull cofacts/url-resolver
$ docker run --rm --env-file .env -p 4000:4000 cofacts/url-resolver

You can use gRPC clients like BloomRPC to access the service for testing purpose.

To access the gRPC service, you can see docker-test.js for an example that uses @grpc/proto-loader.

CofactsBot

When a client submits a URL for resolution, this service follows redirects and fetches the target page to extract its title, summary, and image URL. Facebook photo links may also require fetching Facebook's Embedded Post endpoint to recover the post URL. These HTTP requests identify themselves as CofactsBot/1.0 (+https://github.com/cofacts/url-resolver); they are triggered by resolution requests, rather than a scheduled crawl.

Set URL_RESOLVER_USER_AGENT to override this shared HTTP User-Agent. If metadata is incomplete or fetching fails, the service may open the page in Puppeteer, which uses a separate browser User-Agent configurable with SCRAPE_USER_AGENT.

Development

First, create an .env file from .env.sample. Fill in your env.

Install development dependencies

# After git clone
$ cd url-resolver
$ npm install

# Build js binary from proto files
$ npm run compile

Start dev server

$ npm start

After editing proto files, run npm run compile to generate corresponding Javascript binary.

Running with Cloudflare Browser Rendering

Instead of launching a local Chromium, url-resolver can talk to Cloudflare Browser Rendering over its WebSocket CDP endpoint, offloading the ~500 MB chromium process to Cloudflare's edge.

Set in .env:

BROWSER_BACKEND=cloudflare
CLOUDFLARE_ACCOUNT_ID=<your account ID>
CLOUDFLARE_API_TOKEN=<API token with Browser Rendering: Edit permission>

The token needs the Browser Rendering: Edit scope at the account level. Create it under My Profile → API Tokens → Custom token.

Plan requirement

Workers Free only allows 10 minutes of browser time per day, which is insufficient for production URL resolution. The Workers Paid plan ($5/month) is required, with $0.09 per browser-hour beyond the 10 hours included monthly.

Indicative cost at 5 seconds per resolution:

Volume	Browser-hours/month	Estimated cost
10,000 URLs/day	~417	~$37
100,000 URLs/day	~4,170	~$374

See https://developers.cloudflare.com/browser-run/pricing/ for current rates.

Local Chromium remains the default and works as a fallback — the Docker image still bundles it, so unsetting BROWSER_BACKEND rolls back instantly without redeploy.

Build

Directly use docker to build image.

$ docker build -t cofacts/url-resolver:latest .

Resource limits

Two env vars bound puppeteer memory at scrape time. Both are optional.

Variable	Default	Description
SCRAPE_MAX_CONCURRENCY	3	Maximum concurrent scrape() operations across all gRPC calls. Each in-flight scrape holds one puppeteer page (image, JS heap, network sockets). Lower this if the server OOMs; raise it if tail latency dominates and there is RAM headroom.
SCRAPE_BLOCK_RESOURCES	image,media,font	Comma-separated puppeteer resourceType list aborted before they hit the wire. Set to empty string to load every resource. The default leaves document, stylesheet, script, and xhr intact, so Readability.js sees the same DOM.

og:image is read from the meta tag in HTML, not from a loaded image, so the default block list does not affect topImageUrl for sites that expose Open Graph metadata. Sites without og:image fall back to scanning <img> tags by rendered size; with image loading blocked, all images report 0x0, so the fallback returns the first <img> instead of the largest.