Profile manual

The HTTP engine

Pipe carries plain HTTP — and HTTPS for the hosts it decrypts — request by request. Each request can be rewritten, answered locally, handed to a script, and routed by itself.

[URL Rewrite]
^http://example\.com/old/(.*) https://example.com/new/$1 302
^https?://ads\.example\.com/ - reject

[Header Rewrite]
http-request ^https?://api\.example\.com/ header-replace User-Agent Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X)
http-response ^https?://api\.example\.com/ header-del Set-Cookie

[Body Rewrite]
http-response ^https://api\.example\.com/feed "\"ads\":\[[^\]]*\]" "\"ads\":[]"
http-response-jq ^https://api\.example\.com/config 'del(.tracking)'

[Map Local]
^https://api\.example\.com/banner data-type=text data="{}" header="Content-Type:application/json"

[MITM]
hostname = api.example.com, ads.example.com

What goes through it#

• Every request to Pipe's shared proxy: GET http://host/path HTTP/1.1.
• On the tunnel, and inside a SOCKS5 or HTTP CONNECT tunnel: every connection read ahead — TCP to port 80 (Sniffing) — that turns out to be HTTP.
• Every connection to a host in force-http-engine-hosts, on the port its item names — port 80 for one that names none, every port for :0. A connection there that is not HTTP breaks.
• HTTPS to the hosts [MITM] decrypts, once it is decrypted. Its requests are then https:// URLs to the rules, rewrites, scripts and records alike.

Each request is routed by the rules by itself, with its URL and its User-Agent for URL-REGEX and USER-AGENT and its Host as a name for extended-matching; is carried to its server; and is a record of its own in the app's list of requests, with its method, URL, status, and what was done to it.

Requests and connections#

• HTTP/1.1 and HTTP/1.0.
• The client's connection is kept for its next request. So is the connection to a server, for the next request that goes the same way — the same policy, the same destination. A kept server connection that turns out to have been closed is dialled again, once, for a request that can have been done twice with nothing changing — GET, HEAD, OPTIONS, TRACE, PUT, DELETE, or any method with an Idempotency-Key — and whose body the client is not still sending.
• Heads are read strictly: a request line of method, target and version, one value on a field line, and nothing anywhere in a line that another reader could take for the end of it (a bare CR or LF, a NUL). What does not read that way is refused (400 for a request, 502 for an answer). A head is at most 64 KiB and 256 field lines (431 for a request past either). A header value may hold any bytes — a Latin-1 filename, a GBK cookie — and goes on as it came.
• Bodies go on as they come: by Content-Length, in chunks (kept as chunks, written anew from the sizes they said), or to the end of the connection — unless [Body Rewrite] or a script wants one whole.
• 100 Continue comes back before the body it asks for.
• A client that closes its connection while its answer is on the way — an app giving up on a long poll — ends the request, and the connection to its server with it.
• An upgrade — 101 Switching Protocols, a WebSocket — becomes a plain relay, when the request asked for one; a 101 nobody asked for is refused.
• On the tunnel, a request goes to the address its connection was made to, whatever its Host says — unless a rewrite sends it to another server.

The order of processing#

What the profile does to a request comes in this order:

1. [Header Rewrite]
2. [URL Rewrite]
3. [Map Local]
4. [Body Rewrite]
5. http-request scripts
6. the rules, which see the request as these left it

An answer goes through [Header Rewrite], then [Body Rewrite], then http-response scripts.

Refusals and failures#

A request a rule refuses is answered by Pipe:

A request that cannot be carried — its proxy or server cannot be reached, its server breaks off before answering — is answered 502 Bad Gateway, with a page that says why, unless show-error-page = false.

[URL Rewrite]#

pattern replacement type

[URL Rewrite]
^http://example\.com/old/(.*) https://example.com/new/$1 302
^http://www\.example\.cn http://www.example.com
^https?://tracker\.example\.com/ - reject
^http://router\.example http://{{{GATEWAY_ADDRESS}}} 302

A request whose URL the regular expression pattern finds has the part it found replaced by replacement — $1 is what the pattern's first group captured — and then:

• The first line that matches decides.
• A line is tried on the request's URL, then on the same URL with the host its Host names, then with the name its connection was made to.
• {{{GATEWAY_ADDRESS}}} in a replacement is the router of the network the device is on, as the app last said. While it is not known, a line that uses it does not apply.
• Any other type is warned about, and the line is left out. To answer a request with content of your choosing, use [Map Local].

[Header Rewrite]#

http-request|http-response pattern action field value

[Header Rewrite]
http-request ^https?://api\.example\.com/ header-add X-Client Pipe
http-request ^https?://api\.example\.com/ header-replace User-Agent Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X)
http-request ^https?://api\.example\.com/ header-del Cookie
http-response ^https?://www\.example\.com/ header-replace-regex Cache-Control max-age=\d+ max-age=60

Every line whose pattern finds the request's URL changes the request's head — or its answer's — in order.

• http-request changes requests, http-response answers. A line without either is warned about, and left out.
• A value may hold spaces: it is the rest of the line, or a quoted word.
• header-add and header-replace need a value, and header-replace-regex a regular expression and its replacement: a line without them is warned about, and left out. So is another action.
• A line that would change Content-Length or Transfer-Encoding, which say where a message ends, is warned about and left out.

[Body Rewrite]#

http-request|http-response pattern regex replacement [regex replacement …]

[Body Rewrite]
http-response ^https://api\.example\.com/feed "\"ads\":\[[^\]]*\]" "\"ads\":[]"
http-request ^https://api\.example\.com/log "\"debug\":true" "\"debug\":false" "token=\w+" "token=-"

Every line whose pattern finds the request's URL replaces, in the request's body or its answer's, what each regular expression finds, in turn. ^ and $ are the ends of a line. A word with spaces in it is quoted.

• A body is read whole first, decompressed (gzip, deflate, zstd), rewritten, and sent on by its new length. A request whose answer is to be rewritten asks only for encodings Pipe can undo: its Accept-Encoding is narrowed to gzip, deflate and, off iOS, zstd. A zstd body may use a window of up to 8 MiB; one that needs more, or a body in another encoding (Brotli), goes on as it came.
• A body is read whole up to 4 MiB of an answer and 2 MiB of a request on iOS (10 MiB and 32 MiB elsewhere); for a script, up to its max-size, and at most 4 MiB on iOS (32 MiB elsewhere). A longer one goes on as it came, past the rewrites and scripts, and its record says so. So does one Pipe's memory has no room for, and one that comes slower than Pipe waits for: 30 seconds, and as long again as its length takes at 16 KiB a second.
• A chunked request body, or one that waits for 100 Continue, goes on as it came.
• A body that is not UTF-8 text goes on as it came, and its record says so.
• What the rewrites make of a body comes to at most twice what it may be read at (above). A replacement that would make it longer, or that Pipe's memory has no room for, is not made, and the record says so; the others are.
• A body nested deeper than 128 arrays or objects is not given to a jq program; it goes on as it came. fromjson does not read JSON text nested deeper than that either: the program fails, and the body goes on as it was.
• A line whose regular expressions do not come in pairs is warned about, and left out.

jq#

http-request-jq|http-response-jq pattern 'program'

[Body Rewrite]
http-response-jq ^https://api\.example\.com/config 'del(.tracking) | .features.beta = true'
http-response-jq ^https://api\.example\.com/feed jq-path=strip-ads.jq

A JSON body goes through a jq program — written on the line, quoted, or in a file named by jq-path= — and the first value the program gives is the body that goes on, read whole and sent by its length as above.

• A body that is not JSON, or one the program fails on or gives nothing for, goes on as it was, and its record says why. So does one a program takes more than three seconds on, or more memory than about eight times the body.
• Programs follow jq, but for a few corners: Pipe runs them with jaq, whose documentation lists the differences. They have jq's library, but env, and cannot load modules. A program Pipe cannot run leaves its line out, with a warning.
• A line has one program; a line with more is warned about, and left out.

[Map Local]#

pattern data-type=… data=… status-code=… header=…

[Map Local]
^https://api\.example\.com/banner data-type=text data="{}" header="Content-Type:application/json"
^https?://ads\.example\.com/.*\.gif data-type=tiny-gif
^https://api\.example\.com/down data-type=text data="Not today" status-code=503
^https://api\.example\.com/blob data-type=base64 data=SGVsbG8=

A request whose URL the pattern finds is answered by Pipe, and never sent on.

Without a Content-Type, one is given: by the file's extension for a file (below), text/plain; charset=utf-8 for text, image/gif for tiny-gif, and application/octet-stream for base64.

• A file is read when the profile is compiled. A data URL is not downloaded yet: such a line is left out, with a warning. On iPhone, the app keeps no files beside a profile yet (Files and URLs): answer with text, base64 or tiny-gif there.
• A status-code outside 200 to 999, a header Pipe cannot read, data that is not base64, or a data-type it does not know is warned about, and the line is left out. A word that is not key=value is warned about, and read past; a parameter Pipe does not know is noted, and read past.

Regular expressions#

The patterns in these sections are regular expressions, found anywhere in the URL unless anchored, case counting. They may look around and refer back, as other apps' modules write them; what a header or body rewrite finds in the text may not. A line whose pattern Pipe cannot use is left out, with a warning that says why.

[MITM]#

HTTPS to the hosts hostname names is decrypted, on the tunnel and through the shared proxy (HTTP CONNECT and SOCKS5). The client is answered as the host, with a certificate made for that name and signed by a CA it trusts — the CA's own certificate sent with it — and its requests go through the HTTP engine, each on to the server over a TLS connection of Pipe's own.

[MITM]
hostname = -pinned.example.com, *.example.com, api.example.net:8443
skip-server-cert-verify = false
h2 = true

hostname#

hostname is a host list:

• * and ? are wildcards; - in front leaves a host out. The first item that matches decides, so exclusions go first.
• host:port is another port, host:0 every port. A host without a port means port 443 — and, for a connection whose ClientHello asks for it by name, any port.
• <ip-address>, <ipv4-address>, <ipv6-address> and <simple-hostname> stand for connections made to an address, or to a name without a dot.
• A connection is read for its ClientHello when it may be one to decrypt: on port 443, on a port the list names, or made to a name the list holds (on the tunnel, a fake IP put back to its name).
• An item Pipe cannot read is warned about, and left out.

hostname-disabled takes items out of hostname, as written: an item of hostname that is also in hostname-disabled is not used.

The CA#

The certificates clients are shown are signed by a CA:

• ca-keystore-name: a p12 item of [Keystore]. It comes before ca-p12.
• ca-p12: a PKCS#12 file in base64, with its password in ca-passphrase. It may be encrypted as OpenSSL 3 encrypts one (PBES2, AES), as OpenSSL 1 and the macOS keychain do (RC2 and 3DES), or not at all. The key may be RSA or EC.
• Without either, Pipe's own: made the first time it is needed, kept, and the same from then on.

A CA that cannot be read, or a keystore item that is not there, is warned about, and Pipe's own CA signs instead — which the device must then trust. A CA that is read but cannot sign what clients would take — a key that is not its certificate's, a certificate that does not say it is a CA's, or that may not sign certificates — stops the profile, with a message that says which.

The app shows the CA in use, under HTTPS Decryption, with whether the device trusts it, and installs it: as a configuration profile carrying the certificate, which Safari takes from the app and Settings installs (or which is shared, or saved). iOS lets no app install a root certificate or trust one itself, so it takes these steps, all but the first iOS's own:

1. Install Certificate, in the app; Safari asks to allow the download.
2. In Settings, Profile Downloaded at the top (or General › VPN & Device Management), then Install — the app's Open Settings is a tap from it.
3. Settings › General › About › Certificate Trust Settings: full trust for the certificate, by its name.

Back in the app, the page sees it trusted. Only the certificate leaves Pipe's core; the key stays in it.

Settings#

A key not in the table is noted, and read past. A value that should be true or false and is neither is warned about, and taken as false. In [MITM], a key given on two lines counts once: the later line.

h2 = true offers clients HTTP/2. Each stream is a request of its own through the HTTP engine, routed, rewritten, scripted and recorded as one over HTTP/1.1 is. Servers are offered it too: the requests that go to one that takes it are streams of one connection; those that go to one that does not, connections of their own, kept for the next. With h2 off, clients are offered HTTP/1.1 alone, and a client that offers only HTTP/2 cannot be decrypted. More than HTTP is never decrypted: a client whose ClientHello asks for protocols none of which Pipe speaks as a server — gRPC asking for HTTP/2 with h2 off, MQTT on port 443 — is carried as it came, for decrypted its handshake could only fail. A WebSocket goes over HTTP/1.1 either way.

client-source-address names the clients whose connections are decrypted: addresses and ranges — 192.168.1.0/24, fd00::/8 — with - in front to leave one out. The first item that holds a client's address decides, so exclusions go first, and a client no item holds is not decrypted. Without the setting, every client is. A client is the address it reaches the shared proxy from, or, on the tunnel, a packet's source. QUIC is refused only to the clients that are decrypted. An item Pipe cannot read is warned about, and left out.

Clients that refuse#

A client that refuses the certificate — the CA is not trusted, or the app pins its own certificates — is recorded as such. One that takes it and then asks for nothing is logged, once for each host: that is a sign of pinning too (or of a browser opening a connection ahead, and closing it unused). Leave such hosts out of hostname.

Capture#

Capture keeps the heads and bodies of the requests the HTTP engine carries, and of their answers — the request as it went to the server, the answer as it went to the client — while it is on.

• It is not a profile setting: it is off until it is switched on in the app while Pipe runs, and a reload leaves it as it is.
• Capture holds up to 4 MiB on iOS (64 MiB elsewhere), dropping the oldest entries first. Each message is kept up to 512 KiB on iOS (8 MiB elsewhere).
• The app exports recent requests as HAR 1.2, with what capture kept of each: bodies with their chunks joined, and gzip, deflate or zstd undone. What a HAR holds is decrypted traffic. The header fields that carry credentials are left out — Cookie, Set-Cookie, Authorization, Proxy-Authorization, and every field whose name has auth, cookie, token, secret, password, passwd, session, key, signature, csrf or xsrf in it — and the values of query parameters, and of form (multipart too) and JSON fields, whose names have one of these in them, or are code, sig, otp, pwd or pass, are given as …, in every URL the archive holds. Other bodies, and JSON capture kept only the start of, are as they went: a HAR can still hold what a page or an app sent in them.
• Switching capture off keeps what it kept until newer requests take their place or it is cleared.