cURL Best Practices: Writing HTTP Requests You Won't Have to Debug Twice

cURLParse Guides · Updated 2026-10-18 · All guides

curl is the only HTTP client that ships with every server on earth. That also makes it the most copy-pasted and most mis-configured one. These are the practices that turn it from folklore into tooling.

Make failures loud

The default curl succeeds on a 500. curl https://api.example.com exiting 0 after receiving {"error":"db down"} has burned entire dashboards. The fix is a flag set, not a habit:

curl -sS --fail-with-body \
  --connect-timeout 5 --max-time 30 \
  -w '\n%{http_code}\n' \
  'https://api.example.com/health'

-sS: quiet progress bar, but real errors still print. (-s alone is how people lose DNS errors.)
--fail-with-body: non-zero exit on 4xx/5xx and the error body still prints. Prefer it over -f, which hides the body — the only interesting part.
--connect-timeout + --max-time: without both, a blackholed TCP handshake waits ~130s and your CI watchdog kills the wrong thing.

Quote like the shell is adversarial

Single quotes around every header and payload that contains no $variables. Double quotes only when you deliberately want interpolation — and never around JSON you didn't write, because $(...) inside double quotes is a command, not a string. The full taxonomy of how this breaks per shell is in quoting hell.

Be explicit about the body's identity

-d silently implies two things: the method becomes POST, and Content-Type becomes application/x-www-form-urlencoded. If you meant JSON, say so — -H 'Content-Type: application/json' — or your "JSON API call" is a form POST wearing a costume. This exact mismatch is the #1 reason converted code fails.

Read the response, not just the output

Add -i (or -D -) until the request is stable. Half of "it worked in curl" reports are people who never looked at the status line and saw a plausible-looking error body. -w '%{http_code}\n' is the assertion-friendly version for scripts.

Convert before you hand-write

When you move a working curl command into Python/Node/Go, don't rewrite from memory — convert it and read the diff. Our converter maps headers, body encoding, and auth explicitly so the diff shows real semantic choices (form vs JSON, session jar vs bare request) instead of library defaults you didn't choose. The five ways that diff usually saves you are enumerated in the debugging guide linked above.

Prefer honest method flags

Delete -X GET (a no-op that can defeat caching). Keep -X POST when the payload is -d — it's redundant to curl but load-bearing for the human reading the command. See the flags that matter for the fifteen options that carry the other 215.

Scripts: exit codes over string matching

code=$(curl -sS -o /tmp/resp.json -w '%{http_code}' --fail-with-body "$URL") || {
  cat /tmp/resp.json  # the error body explains everything
  exit 1
}

--fail-with-body gives you the exit code; -o + -w give you the code without contaminating stdout. This pattern retires every grep -q ok healthcheck you've ever seen.

Retry only what is safe to retry

--retry looks like a free reliability win and isn't. curl retries on transient errors (timeouts, HTTP 5xx) and, by default, only on idempotent requests — but -d makes your request a POST, and --retry on a POST is how a payment endpoint gets charged twice. The safe shape:

curl -sS --retry 3 --retry-delay 2 --retry-connrefused \
  --fail-with-body --connect-timeout 5 --max-time 30 \
  'https://api.example.com/health'

--retry-all-errors extends retries to all errors including the non-transient ones; pair it with --retry-max-time so a total failure still ends inside your CI budget.
--retry-connrefused matters in deploy scripts: the container is up but the port isn't listening yet, and plain --retry treats connection refused as fatal.
Retrying a non-idempotent POST is a design question, not a flag. If the API supports an idempotency key, send one (-H 'Idempotency-Key: ...') and then retry. Without one, the retry converts a timeout into a duplicate.

Rate-limit yourself before the server does

curl --parallel --parallel-max 4 -Z url1 url2 ... fetches multiple URLs concurrently, and the default is 50 parallel transfers — a self-inflicted DDoS against your own API. --parallel-max caps it; --xattr is unrelated and not a typo to fix here. For anything beyond a handful of URLs, read Retry-After/X-RateLimit-Remaining and sleep, as the pagination example does.

Pin the shape of the request you think you're making

The habit that prevents the most production surprises is making the invisible visible once per endpoint:

# what does the server actually receive for my command?
curl -sS -X POST http://localhost:9999/ \
  -H 'Content-Type: application/json' \
  -d '{"a":1}' -v 2>&1 | sed -n '/^> /p'

Printing only the > lines (the request) shows the method, the headers curl invented, and the body byte-for-byte. -d with no -H prints Content-Type: application/x-www-form-urlencoded — proof of the costume problem above, in your own terminal, in ten seconds. A local echo origin makes this a two-minute habit instead of a folklore argument; the transcripts in the flags guide are exactly this exercise.

curlrc: where the defaults should live

Flags every call should carry belong in ~/.curlrc (or $CURL_HOME), one per line, without the leading dash:

# ~/.curlrc
silent
show-error
fail-with-body
connect-timeout = 5
max-time = 30
retry = 2
retry-connrefused

Two cautions. First, --insecure in curlrc is a landmine — it silently disables TLS verification for every script on the machine, including the ones you write six months from now assuming defaults are safe. Second, a curlrc that sets silent breaks scripts that parse the progress meter (rare, but the bug is unforgettable when it happens). Keep curlrc to timeouts and fail-loud flags; keep auth, output, and method per-invocation.

CI: the healthcheck that cannot lie

The pattern from the scripts section, hardened for pipelines:

#!/usr/bin/env bash
set -euo pipefail
req=(curl -sS --connect-timeout 5 --max-time 30 -o /tmp/resp.json -w '%{http_code}')
for i in $(seq 1 10); do
  code=$("${req[@]}" "$URL") || code=000
  [ "$code" = 200 ] && exit 0
  sleep 3
done
echo "$URL returned $code after 10 tries:"; cat /tmp/resp.json; exit 1

The array (req=(...)) keeps the flags in one place; the || code=000 means a transport failure prints as 000 — curl's own convention for "no HTTP response" — instead of aborting the loop under set -e.
The loop polls because deploy-then-check has a race window; --retry alone won't cover a process that starts, crashes, and restarts behind the load balancer.
cat /tmp/resp.json on failure is the whole point: the exit code says that it failed, the body says why, and the flags guide shows why -f alone would have hidden the body.

FAQ

Is -k ever acceptable? In a lab against a self-signed cert, yes, with a comment. In CI or production, no — ship the CA bundle and use --cacert. -k doesn't skip a warning, it skips the security model.

Should I always add --compressed? For interactive use, fine. In scripts, only when the source command had it — the converter keeps it when it changes the response, and your timing measurements are meaningless if half your samples are compressed and half aren't.

What replaces -v for CI logs? --trace-ascii trace.log. Verbose mode interleaves body and headers and truncates; trace is complete and separable.

Is --retry safe on my POST? Only if the endpoint is idempotent or accepts an idempotency key. curl's own retry logic is conservative about which errors count as transient, but it cannot know whether your POST created a row. Default answer: retry GETs, guard POSTs with a key.

Should curlrc contain --compressed? No. Compression changes response bytes and transfer timing; scripts that benchmark or hash responses get inconsistent input. Ask for compression per-request when you want it.