Debugging a cURL Request That Works in curl But Not in Python

It is almost never "the API hates requests." It is almost always one of five things curl did that your code didn't — all of them visible in the converted output on this site, if you know which line to read.

Every transcript below was captured against a local origin that prints what it receives, with curl 8.5.0 and requests 2.31 hitting the same server. Where a blog post and the transcript disagree, the transcript wins.

1. User-Agent

curl sends User-Agent: curl/8.x. Some CDNs prefer it and reject generic script UAs out of hand — you're getting a 403 from the edge, not the app. Fix: set a real UA header in code and test again. If the code works with a UA header, that was it.

curl -sv -o /dev/null https://api.example.com/ 2>&1 | grep -i '^> user-agent'

2. Header case and ordering

HTTP headers are case-insensitive in the spec; badly-written APIs are not. curl's copy-paste sends Content-Type; your requests.post(json=...) sends it automatically, and your hand-rolled dict added content-type — and their middleware checks the exact casing of the one they read. Match the copy from devtools verbatim, once, and normalize later.

3. The body encoding you didn't choose

This is the single most common cause of the complaint in this page's title. curl -d '{"a":1}' sends those exact bytes. requests.post(url, data={"a":1}) form-encodes the dict — the origin receives a=1, the API expected JSON, you get a 400 "invalid json".

# What the server sees for each, verified against one origin:
curl -d 'a=1' http://origin/          # body: b'a=1'   + form-urlencoded content-type

import requests
requests.post(url, data={"a": 1})     # body: b'a=1'   <- matches curl, looks like it worked
requests.post(url, json={"a": 1})     # body: b'{"a":1}' + application/json
requests.post(url, data='{"a":1}')    # body: b'{"a":1}' but content-type stays form-urlencoded

The middle line is the one that ships. The third is the one that fails on a server that sniffs the content type instead of the body. The curl equivalent of json= is data=json.dumps(...) plus the header:

requests.post(url, data=json.dumps({"a": 1}),
              headers={"Content-Type": "application/json"})

Our multipart upload example covers the same trap from the file side.

4. Redirects: the failure mode is the status code, not the client

You will find this stated as a curl-vs-requests difference. It is not. Both clients downgrade POST to GET on a 301, and both preserve it on a 307. Same origin, same payload, four requests:

Your payload disappeared on the 301 row. It disappeared identically in both clients, so "it worked in curl" was never about the client — it was about which redirect the endpoint happened to issue, or about you not passing -L and never following it at all. Three consequences worth acting on:

# force the method through a 301 (note: --post301 carries the body, -X POST alone does not)
curl -L --post301 -d 'a=1' https://api.example.com/start   # POST /final body=b'a=1'
curl -L -X POST  -d 'a=1' https://api.example.com/start    # POST /final body=b''   <- method kept, body gone

-X POST combined with -L gives you the worst of both: the method survives, the body does not, and the endpoint sees an empty POST that can 200 while creating nothing. Use --post301/--post302, or point the client at the final URL.

In code, do not expect requests to preserve the method either:

r = requests.post(url, data='a=1')
assert r.history == [] or r.request.method == 'POST', \
    f"downgraded to {r.request.method} after {[h.status_code for h in r.history]}"

That assertion is 5 lines and catches the class of bug that otherwise costs an afternoon: the redirect is real, the 200 is real, and the record you thought you created does not exist.

5. Timeouts and connection reuse

curl reuses nothing and times out fast. Your code might reuse a keep-alive connection a load balancer already killed (that RemoteDisconnected you keep seeing), or sit on a 130s connect timeout while a CI watchdog blames the API. Explicit timeouts on every call; retry only idempotent verbs.

s = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_maxsize=1)   # or a pool timeout < your LB's idle timeout
s.mount('https://', adapter)
s.post(url, json=payload, timeout=(5, 30))   # (connect, read) — a bare timeout= is only the read half

The debugging move: paste your curl into the converter, read the diff between what it generated and what you actually wrote, and ship the difference. Nine times out of ten that diff is the bug. Read the diff in both directions: a line missing from the generated code can be the converter declining to handle a flag (-u, -b, -F, --data-binary are not parsed), so confirm the omission is yours before you fix it.

FAQ

requests gives 403 but curl gives 200. What first? Set User-Agent to what curl sends (check with curl -v). Cloudflare and Akamai commonly gate on UA alone.

Should I retry on RemoteDisconnected? Only idempotent methods (GET/PUT/HEAD) and only with a fresh connection — disable connection reuse or set a pool timeout shorter than your LB's idle timeout.

Why did my POST body vanish after a redirect? Because the redirect was a 301 or 302, and every client you use downgrades those to GET per RFC behavior — curl -L included. Point the client at the final URL directly, or have the endpoint answer 307/308, which both clients preserve.

Is curl -L unsafe for POSTs? It is unsafe for POSTs that hit a 301, yes, and so is every HTTP client. Prefer --post301 when you must follow, and treat a 200 on a followed redirect as unverified until you confirm the resource exists.