"OpenAI-compatible" is a surface, not a contract

Almost every gateway, proxy and inference host now claims to be OpenAI-compatible, and the claim is usually true in the sense that matters least: you can point an OpenAI SDK at it and get a completion back. What varies, and what costs you a day when it varies, is everything after the happy path.

This is a list of the specific things "compatible" does not tell you, and a short set of requests that answer each one. It takes about twenty minutes and is worth running before you migrate anything, including against us.

1. Which endpoints, not "the API"

The OpenAI API is not one endpoint. Compatibility almost always means POST /chat/completions, sometimes embeddings, and much more rarely anything else: files, batch, moderation, or the newer Responses surface. That is a reasonable place to stop; the problem is that the claim does not say where it stopped.

Test it: call GET /v1/models first. If it 404s while chat completions works, the surface is one endpoint wide, and you now know that before you write code against anything else.

2. Parameters that are accepted and ignored

This is the one that actually hurts, because it fails silently. A request carrying temperature, top_p, seed, response_format or tools can be accepted with a 200 and served as though those fields were never sent. Nothing errors. Your evaluation just quietly measures a different configuration than you think it does.

Test it: send the same prompt twice with temperature at 0 and at 2, several times each. If the spread of outputs looks the same at both settings, the parameter is not reaching the model. Do the same with seed: two identical requests with the same seed should produce more similar output than two without it. For response_format, ask for JSON and send a prompt that would naturally answer in prose.

An implementation that rejects a parameter it does not support is being more honest than one that accepts it: a 400 is information; a 200 is a wrong answer you will not notice.

3. Whether you get usage back

The non-streamed response object carries usage alongside id, choices, created, model and object (reference). If usage is missing or zeroed, you cannot attribute cost per request, which means you cannot attribute it per team or per feature either: you are back to reconciling one bill against a guess.

Streaming is where this most often breaks. Usage is not in the stream by default: you ask for it with stream_options and its include_usage field, and the documented behaviour is that an additional chunk will be streamed before the data: [DONE] message, whose usage field covers the whole request. Plenty of compatible implementations do not implement that chunk.

Test it: make a streaming request with "stream_options": {"include_usage": true} and check whether a usage chunk arrives before [DONE]. If your cost attribution depends on streamed traffic and that chunk never comes, you have found the gap before it becomes a billing dispute.

4. Streaming framing, exactly

Streaming responses are server-sent events, and the stream ends with data: [DONE]. Three things go wrong often enough to check: the terminator is missing, so a client waits for a close that never comes; chunks are buffered somewhere in the middle so the whole response arrives at once — technically streaming, functionally not; or an error mid-stream arrives as a broken frame rather than as anything a client can parse.

Test it: use curl -N, without it curl buffers and you will measure your own client. Watch whether tokens arrive progressively, and that the last line is the terminator. Then make it fail: request a model that does not exist, or send an oversized prompt, and see what a mid-stream failure actually looks like on the wire.

5. Errors: status codes first, envelopes second

The OpenAI SDKs map HTTP status codes to their exception classes. That means status codes are the part of the error contract you can rely on, and the JSON envelope is the part that varies. Write your error handling against the status.

For reference, here is what our own endpoint returns, measured rather than described:

// missing or invalid key -> 401
{"error": {"code": "", "message": "Invalid token (request id: ...)", "type": "runix_error"}}

// unknown path under /v1 -> 404
{"error": {"message": "Invalid URL (POST /v1/nope)", "type": "invalid_request_error",
           "param": "", "code": ""}}

Note the type values differ between the two, and one of them is not an OpenAI-standard string. That is exactly the point: if your retry logic branches on error.type, it is branching on the least stable field in the response. Branch on 401 and 429 instead.

Test it: send a request with no key, with a bad key, to a path that does not exist, and with a model id that does not exist. Four requests, four status codes. Write them down: that table is your error handling.

6. Whether the request id survives

When something goes wrong in production, the only useful question is "what happened to this request". That needs an identifier you can quote and the other side can find. Some implementations return one in a header, some inside the error message, some not at all.

Test it: trigger an error and look for an id anywhere in the response — headers included. If there is not one, every future support conversation starts with both sides guessing.

7. Tool calling, if you use it

Tool and function calling is the compatibility surface most likely to be partial, because it is several behaviours rather than one: the shape of tools in the request, whether tool_choice is honoured, whether arguments come back as valid JSON, and whether parallel tool calls are supported or silently collapsed to one.

Test it: define two tools that could both plausibly be called and see how many come back. Then force one with tool_choice and confirm it is the one you get.

The twenty-minute version

1. GET /v1/models: how wide is the surface?
2. Same prompt at temperature 0 and 2, several times: is the parameter reaching anything?
3. Non-streamed request: is usage present and non-zero?
4. Streamed request with stream_options.include_usage — does the usage chunk arrive before [DONE]?
5. curl -N — do tokens actually arrive progressively?
6. No key / bad key / bad path / bad model: four status codes, written down.
7. Trigger an error: is there a request id you can quote?
8. Two tools, then tool_choice — is tool calling whole?

None of this is exotic, and that is rather the point: the answers are cheap to get and almost nobody gets them before committing. A provider that passes all eight has earned the word compatible. One that passes six has not stopped being useful — you just know which two to design around, which is a different position than finding out in production.

If you want to run this against Runix Router, ask for an evaluation key and do exactly that. We would rather you test the claim than take it.

Parameter names and streaming behaviour above are from the OpenAI chat completions reference, read on 6 August 2026. The error envelopes are what our endpoint returned on the same day. APIs change; re-run the checks rather than trusting a list.