Upgrades as a Source of Failure: Why It Broke Right After the Update

An upgrade changes defaults, tightens checks and relocates parameters. Four mechanisms, each of which looks exactly like blocking, and the correct order of upgrading.

2026-08-31 GIGATAP Team #vpn
#vpn#anticensorship#operations#upgrade#troubleshooting

Upgrades as a Source of Failure: Why It Broke Right After the Update

One complaint comes up suspiciously often: “I updated and it stopped working.” It is usually written off as coincidence — the block just happened to land at the same time. Coincidences do happen, but less often than assumed: an upgrade has four mechanisms that break a working configuration, and each of them looks exactly like blocking from outside.

Mechanism 1: a default changed#

The nastiest one, because it is invisible in your configuration.

A parameter you never set had one value, and after the upgrade it has another. Your settings file did not change by a single byte — what changed is the value substituted when you say nothing.

A real example: a server implementation introduced a minimum acceptable client version, and in one build the default became non-zero. The server began rejecting clients older than the threshold. From the user’s side: “it will not connect,” with no explanation. From the server’s side: everything is fine, it honestly rejected an incompatible client.

BEFORE                        AFTER
server: minversion = (none)   server: minversion = X
client 1.8 ──► ✓ accepted     client 1.8 ──► ✗ rejected
client 2.4 ──► ✓ accepted     client 2.4 ──► ✓ accepted

the configuration did not change. the default did.

The diagnostic sign: failure is selective by client version. Some users work, others do not, and they split not by geography but by when they last updated the app.

Mechanism 2: a parameter moved#

The upgrade relocated a setting into a different configuration section. The old location is no longer read.

The key detail: most configuration parsers do not complain about extra fields. A parameter in the old place raises no error — it is simply ignored.

The result: the configuration loads, the service starts, the logs are clean. And it runs without the setting you specified.

What you see What is happening
configuration accepted the extra field was silently dropped
service started there is no error because there is no error
logs are clean ignoring is not logged
behaviour changed the setting was not applied

The check is simple and almost nobody performs it: look at the effective configuration, not the submitted one. If the service can show its parsed state, compare it against what you sent.

Mechanism 3: a check got stricter#

The upgrade became less tolerant of what used to pass.

The typical case is certificates. Moving to post-quantum signature algorithms increases chain size, and the chain may stop fitting inside a limit that was never a problem before. Or the reverse: an implementation started rejecting an algorithm combination it previously accepted.

Here the failure is usually consistent — it breaks for everyone at once. Paradoxically that is good news: a blanket failure is diagnosed faster than a selective one.

Mechanism 4: versions drifted apart#

The most common one in multi-node installations.

The control panel was upgraded, the nodes were not. Or the other way round. A version gap opened between them, and the protocol they speak stopped matching.

CORRECT ORDER               INCORRECT
1. panel                    1. half the nodes
2. verify                   2. panic
3. one node                 3. the remaining nodes
4. verify                   4. the panel
5. remaining nodes          5. unclear what broke

“Control plane first, then the workers, one at a time, with a check in between” is not bureaucracy. It provides what a bulk upgrade cannot: at every step it is clear what exactly changed.

Why does it look like blocking?#

All four mechanisms produce the same observable picture: it worked, now it does not. What separates them is not the failure itself but who is affected and whether a version rollback helps.

Several signs help tell them apart from blocking.

Sign More likely an upgrade More likely blocking
Timing coincides with the update unrelated
Who is affected by client/node version by geography, by carrier
Other addresses also fail some work
Another network does not help often helps
Version rollback helps does not help

The last row is the strongest sign. Rolling back a version does not cure blocking. If the rollback helped, the cause was internal.

A numeric example#

An installation with a panel and nine nodes. After the upgrade, complaints from part of the user base.

Group Affected What they share
Nodes 1–3 (upgraded) 100 % of complaints new version
Nodes 4–9 (not upgraded) 0 % of complaints old version
Users with app ≥ 2.4 0 % of complaints newer than the threshold
Users with app < 2.4 100 % of complaints older than the threshold

Intersecting two rows gives the answer in a minute: only the combination “new node + old client” breaks. That is mechanism 1, not blocking. Geography plays no part at all.

Had this been blocking, the split would have followed carriers and regions rather than version numbers.

What to do before upgrading#

Record what works now. Not “everything is fine” but a measurable state: which routes were checked, with what results. Without it there is nothing to compare against afterwards.

Read the changelog for defaults. Not the new features — the lines reading “default value changed” and “parameter moved”. Those are what break things.

Upgrade one at a time, verifying in between. Control plane first, then one worker at a time.

Keep a rollback path. Not a theoretical one (“we will revert the version”) but a tested one: you know where you are rolling back to and what happens to data when you do.

What to do afterwards#

Compare the effective configuration against the submitted one. Mechanism 2 is caught only this way.

Test with an old-version client. Mechanism 1 is caught only this way.

Repeat the same measurements as before. The difference between “before” and “after” is the only thing that admits a single interpretation.

Conclusion#

An upgrade is a change, and a change can break things. What makes it distinctive is that three of the four mechanisms break things silently: the configuration is accepted, the service runs, there are no errors, the behaviour is different.

The practical rule is short: the absence of errors in the log does not mean what you specified was applied. What must be checked is the effective state, not the submitted one.

And before hunting for a block — try the rollback. If it helped, you have already found the cause.

Terms#

  • Default change - a change to the value substituted for a parameter you never set.
  • Version gap - the state where the control plane and the workers run incompatible versions.
  • Effective configuration - the parsed state the service actually runs on, as opposed to the file you submitted.

FAQ#

How do you tell an upgrade effect from blocking?#

Roll the version back. A rollback does not cure blocking, so if it helped, the cause was inside the installation.

In what order should things be upgraded?#

Control plane first, then one worker at a time, verifying between steps. That way each step shows what changed.

Why is there no error in the logs?#

Because extra fields are usually ignored silently. The configuration is accepted, the service starts, and your setting is not applied.

Further reading#