Example: Basic Auth with a Pagination Cursor
curl -sS 'https://api.example.com/v2/orders?limit=50&cursor=eyJwYWdlIjoyfQ' \
-u 'user:password' -i -w '\n%{http_code}\n'
-ubase64-encodesuser:passwordintoAuthorization: Basic .... Converted code must set the header manually in some languages; the converter does this explicitly rather than relying on an auth kwarg that differs per library.- The
-wfooter makes CI assertions easy:test $(curl ...) -eq 200is brittle; a printed code isn't. The flags guide covers-wand--fail-with-body, the two flags that make scripts honest. -ion a paginated endpoint surfacesLink: rel="next"— the actual pagination cursor, which the body usually doesn't duplicate.
What -u sends, byte for byte
Against an echo origin, -u 'user:password' produces exactly one header:
Authorization: Basic dXNlcjpwYXNzd29yZA==
That is base64("user:password") — encoding, not encryption. Two consequences:
- A password containing a colon breaks nothing in curl (it takes the first colon as the separator), but it breaks every human reading the base64 later. If your "password" has structure, it's probably an API key and belongs in a header, not in
-u. -usends the header on the first request. curl does not wait for a401 WWW-Authenticatechallenge. Some servers that advertiseNegotiate/NTLMwill ignore your Basic header and 401 you anyway; add--ntlmor--digestonly when the server actually demands it, and check withcurl -vwhich scheme the challenge named.
Following the cursor without lying to yourself
The Link header is the cursor. A paginated walk that asserts on the response instead of trusting it:
#!/usr/bin/env bash
next='https://api.example.com/v2/orders?limit=50'
page=0
while [ -n "$next" ]; do
page=$((page+1))
hdr=$(curl -sS -D - -o "page-$page.json" -w '%{http_code}' \
-u "user:$PASS" "$next")
code=$(tail -n1 <<<"$hdr")
[ "$code" = 200 ] || { echo "page $page -> $code"; exit 1; }
# rel="next" survives redirects; take only that relation
next=$(grep -i '^link:' <<<"$hdr" | tr ',' '\n' | grep 'rel="next"' \
| sed -E 's/.*<([^>]*)>.*/\1/')
done
echo "$page pages"
- Splitting on
,before matchingrel=matters: some APIs put commas inside the URL (;q=parameters), and a naivesedgrabs the wrong bracket. - If
Linkis absent, the API may use a body cursor ({"next":"..."}) — read the body withjq -r .next. Either way, stop when the page count stops advancing, not when the file is non-empty: an API returning the same first page forever is a real failure mode, and an infinite loop writingpage-9999.jsonis how you find it. -w '%{http_code}'after-D -puts the code on the last line, which is why the snippet tails instead of greps. The assertion pattern is the same one in the flags guide.
Rate limits show up in the headers you skipped
Paginated APIs throttle. The 429 is visible with -i, but the useful part is Retry-After (seconds or HTTP-date) and X-RateLimit-Remaining. A walk that ignores them gets its token revoked for hammering; one that reads Retry-After sleeps the right amount. If the walk runs in CI, print X-RateLimit-Remaining on every page — the drift toward zero is your early-warning, and it costs one grep.
FAQ
Is Basic auth safe over HTTPS? Yes, that's what it was designed for — base64 is encoding, not encryption, so the TLS layer is the protection. Never over plain HTTP.
The cursor in my URL changed to something random. Bug? No — many APIs return opaque cursors (base64 of an offset or keyset). Treat them as opaque; don't construct them client-side. Parse Link: rel="next" instead.
Why does the converter print the Authorization header instead of using auth=(user, pass)? Because kwargs differ across requests/httpx/fetch and silently change behavior. An explicit header converts identically everywhere.