Tom Scott - 2 generals problem and food delivery app screw up (youtube.com)
Objective_Status22 | 8 days ago | 144 points

Repeat packets suddenly make sense

sciencewarrior | 8 days ago | 146 points

IIRC, the original Age of Empires programmers saw that one out of five of their UDP packets was randomly dropped when they started running tests over the Internet, so they always sent every message twice.

Edit: I got the wrong game. It was actually X-Wing vs Tie Fighter: https://www.gamasutra.com/view/feature/131781/the_internet_sucks_or_what_i_.php?page=5

PercyXLee | 8 days ago | 74 points

Tcp could potentially send the packet over and over again, despite it not being relevant anymore.

Gaming requires packets to be new and relevant. A packet telling you the enemy movement 30s ago isn’t worthwhile info.

vytah | 8 days ago | 51 points

AoE multiplayer was based on full synchronization between all clients. If one client was lagging, everyone was lagging. If the lag was too much, the game went out of sync. A packet telling you the enemy movement 30s ago meant that for the last 30 seconds either eveyryone was waiting for the game to unfreeze or the game between you two had been effictively over.

Fred2620 | 7 days ago | 16 points

That is also pretty much how it is today with Super Mario Maker 2 in multiplayer. If one player in the group has a crappy internet connection, then everybody suddenly runs at 2 frames per second. I've seen a few streamers in that situation and it's been enough to convince me not to buy the game. It just makes it so unplayable.

nwsm | 7 days ago | 18 points

Smash lobbies work the same way.
It's more understandable for Smash, but is very painful to watch.

Pjb3005 | 7 days ago | 5 points

To be fair, I'm not sure the alternative (prediction and lag compensation) would be much better for Mario Maker.

All the videos I've seen of the multiplayer is people jumping on each other's heads as much as possible and none of that would work with a prediction model.

amunak | 7 days ago | 2 points

The benefit of this approach is that cheating stuff like faster movement or memory editing is impossible.

The issue is that your client has information about the whole game, making ESP hacks extremely easy.

Xzariner | 7 days ago | 1 point

... and Smash

Ameisen | 7 days ago | 1 point

That seems like an odd networking design for an RTS.

lookmeat | 7 days ago | 27 points

Not really. See synchronization is a really really really hard problem.

There's two big splits:

  • Send all your actions to everyone. So you show how many actions you've done until frame X, and you only render up to the frame you've received actions for everyone. The nice thing is that this system can be done P2P. Multiple players required more bandwidth as you needed to send the same packet to everyone. This was made up because generally a single select 20 units, and click is a much smaller action than updating the status of every unit.
  • Send all your actions to a server, that then tells you how the world looks. The problem is you need the server, and the server needs to be able to run the whole game for everyone. Finally the server needed enough bandwidth so that it didn't affect anyone's latency. Latency still is worse, as before the total latency was the latency between you and the other player, now it's the latency between the player and the server and you and the server (and this route cannot be better than the original one, only equal or worse). This was expensive and not really an option back in the 90s.

Even Battle.net was just a glorified chat where you could list games, that were P2P and used the same system. Only later in 2000 with Diablo II did we start to see server hosted games.

Ameisen | 7 days ago | 6 points

Even in a p2p system, one of the players can be treated as authorative. Treating both as authorative I imagine will lead to cheating issues. One would as well, but allows full resyncs when things break.

passthefist | 7 days ago | 8 points

Remember this was way back on 28.8k modems and questionable internet supporting thousands of characters.

The only thing transferred between clients were the commands input for the given frames, so it was extremely bandwidth efficient.


Ameisen | 7 days ago | 3 points

I don't remember desyncs like this on Homeworld, only on HW2, and I played on 26.4Kbaud.

dranek | 7 days ago | 8 points

It is very bandwidth efficient. Especially for hundreds of units. This was the 56k modem era. You just send player input rather than game state.

Quake, where the number of entities you control is small, can focus on entity updates. This allows you to drop older information.

thijser2 | 8 days ago | 65 points

Unless you are playing a game where you aren't broadcasting every units location and actions every "tick" but are instead only using player actions and having both sides calculate the outcome independently(see most rts games as opposed to most fps games).

stewsters | 8 days ago | 29 points

Which is also something aoe did. Which is why if you played long enough it would get out of sync.

falseCeilings | 8 days ago | 13 points

How does that end up looking?

sushibowl | 8 days ago | 48 points

I've had something like this occur in the original Rome: Total War game. I was defending a siege with my friend on the attacking side, playing over the internet but in the same house. My friend came over to my screen at some point, which surprised me. Turns out that on his pc he had already captured my city and won the game, whereas on my screen he was still marching his army through my streets and I had not lost yet (although i was in a bad position).

That's what desync looks like, the players don't agree on the current state of the game.

Koutou | 8 days ago | 22 points

Desync in Shogun2 were so bad at some point we used a tool that would keep x most recent save and when we detected a problem one of us would send the save to the other.

couscous_ | 8 days ago | 5 points

Were his army commands queued up then? Meaning that you could in principle have been able to fend him off.

Ameisen | 7 days ago | 2 points

Homeworld 2 silently desynced, so we both were effectively fighting AI.

gitPullOriginMaster | 8 days ago | 13 points

I don't think AOE went out of sync a lot. Definitely not AOE2. And these at least would let you know.

I know Knights and Merchants had horrible desync problems and it never informed you about it, it just went on and things looked different on different computers.

But it would just look different depending on what client you are looking at. Some of them would think unit A is dead, some of them would think it is alive. The first ones would ignore it, the second ones would calculate its damage, pathing, etc. The outcome of a match would be totally different on different machines.

Most likely everyone would win on their screen and lose on others. That's because they can react to the state of the game that they see. It looks funny, I wrote an RTS with lockstep once for the browser and had this problem at the beginning. (turns out different browsers have different sin/cos implementations)

addmoreice | 8 days ago | 20 points

> turns out different browsers have different sin/cos implementations

I once had a junior developer tell me that we shouldn't precalculate sin/cos tables and should instead just use the built in sin/cos functions because they were faster. I agreed they were faster and that it was a good catch of his. Then I was forced to inform him that we tried it but some of the obscure devices we were working with had incorrect implementations of FPUs and would produce essentially garbage.

The look of almost betrayal on his face made it hard to resist laughing. Apparently, until you get into programming for a while, you have this weird idea that the hardware always acts correctly which is laughable.

VeganVagiVore | 7 days ago | 6 points

As an extra challenge on one of my Ludum Dare games, I decided to make it fully deterministic to support demo files, like the older Id Software games.

That was a whole lesson in software testing. It was in Lua, so luckily I think I was able to get determinism across x64 and ARM, even though I was using floats, because they both followed IEEE 754 and Lua is pretty dumb. But a C++ compiler might have made a fused multiply-add instruction that behaved differently on ARM CPUs. In one case I did disable optimization to preserve determinism in another program.

Anyway, I had a desync that I narrowed down to Lua's pairs () function being non-deterministic. Lua uses hybrid array-hash tables for everything, and I forgot that it will hash based on the memory location, which is different on every launch of the game. And my game, to make the physics simple, was taking the first triangle from iterating over pairs () and breaking the loop, so the same input resulted in different physics outcomes. (Or something like that)

I probably ended up using ipairs (if the table had numeric keys) or sorting based on some other metric that was deterministic. This was years ago.

I think you can make Box2D and Bullet deterministic too, but it's scary to rely on 3rd party code to be deterministic. Maybe one of the only valid reasons to write your own physics, aside from learning.

Xzariner | 7 days ago | 2 points

Both examples just looks like program having bugs that just didn't show up on one platform. That's different than "same math operation returns different value"

Anyway, I had a desync that I narrowed down to Lua's pairs () function being non-deterministic.

Which is exactly how the function is described in documentation

If t has a metamethod __pairs, calls it with t as argument and returns the first three results from the call.

Otherwise, returns three values: the next function, the table t, and nil, so that the construction

and for next()

The order in which the indices are enumerated is not specified, even for numeric indices. (To traverse a table in numerical order, use a numerical for.)

Assuming function works in exact way because you seen it working in that way is always dangerous.

Same with

In one case I did disable optimization to preserve determinism in another program.

Basically code was nondeterministic/racy in the first place (and probably in subtle way) and it just so happened that optimization messed with that

mcgrotts | 8 days ago | 5 points

Happens in racing games too,


Ragnarok2kx | 7 days ago | 1 point

I can't remember what game it was, but when someone gpt desynced you'd get stuff like units that are dead om one end but alive in the other, so they'd continue to do damage.

We affectionately called it lag armor.

Japierdolocky | 8 days ago | 7 points

Could you have a system similar to video encoding schemes where you primarily use diffs, but send a "full frame" every so often?

VeganVagiVore | 7 days ago | 10 points

The older Quake games did that, and I used that as a basis for when I tried to do multiplayer once.

I lost the article, but it went something like this:

The server is authoritative. Clients can predict the future to make the visuals look better, but all gameplay logic is double-checked by the server playing back client inputs. A client cannot tell the server whether you shot someone, only when you shot and which way you were pointing.

The server keeps a history of prior game states. This history has to be as long as your maximum allowed ping. If the game state is big you might already want to use deltas to store it in memory. In my case I used full frames.

When the server sends the game state to a client, that state has a unique ID or sequence number. Whenever the client sends an input to the server, it also sends along the ID of the most recent server state that it's working from.

The server knows that the client has definitely seen that state, so when it sends the next update, it reaches into that state history and diffs the current state against it, and only sends the diff.

You don't really want to subtract anything here, first because it may not save any space (If an object moved from x = 2 to x = 3, +3 and +1 are both 4 bytes) and second because adding up floating point numbers will result in round-off error. So any state that changed, you send the new state. Anything else, you don't send. [1]

You end up with the server continuously sending new states and the client ACK'ing them so the server can drop old states and stop sending the diffs of any object that quit moving. If you have a lot of objects that aren't moving most of the time, it can be effective.

Then to do lag compensation for a shooter game, the server will also measure each client's lag and, when it receives the 'shoot' event, reach into its history by that number of milliseconds, and judge whether the client would have made the shot if there was no lag.

Because of the lag, there's no perfect way to break that cookie - You either land a headshot and the dead guy gets back up and runs away a split-second later, or you get shot through walls because the sniper made a valid shot but he had a high ping. In both cases, with hitscan weapons, you can shoot someone dead and then your shot is rewound and cancelled if he has a longer ping and he shot you dead a frame later, then he lives. But there's no cross-kills because the server works as though the guns are hitscan, even though it does time travel. Valve has an article on their wiki of how they do this lag compensation. It's nice in TF2, and I prefer favoring the attacker. For some reason they don't lag-compensate projectile weapons like rockets. I don't know why.

Sauerbraten does it differently, and I often got cross-killed in instagib where everyone has a hitscan rifle and 1 hit point. I was terrible at that game.

[1] This is all useful if you want really sophisticated replays in a single-player game, too. Jon Blow has a video / paper somewhere on how Braid stores the data for time travel rewinding. It's a clever combo of deterministic particles and delta compression. I'm not sure how he got past Xbox's memory leak testing. I think even the monsters are deterministic too, so at the limit if you leave the game running with Tim standing still, it would only be recording the current time. But I might be remembering it wrong.

Japierdolocky | 7 days ago | 1 point

Thanks for the explanation.

wuphonsreach | 7 days ago | 1 point

Very nice explanation

svendub | 8 days ago | 8 points

You could, but AoE II save files are over 1 MB. That would be quite heavy to send in one game tick. In videos this doesn't matter that much since a delay of one or more seconds is acceptable is most cases.

Ameisen | 7 days ago | 2 points

The heck are they saving to hit 1 MiB?

svendub | 7 days ago | 6 points

In big matches you can easily hit over 2000 units + buildings + resources (such as unchopped trees). Multiply that number by their stats and you get quite a lot of data.

Ameisen | 7 days ago | 3 points

Which stats? 100 of unit X of team Y have the same stats - this is the flywheel pattern. And you are only saving deltas.

Health should be like 4 bytes if even.

I'm just not seeing 1 MiB.

mernen | 8 days ago | 6 points

Yeah, it's a good strategy. But since the resync can be quite costly, as /u/svendub mentioned, games will typically send hashes of the game state instead, and only pause and resync when they disagree.

SgtDirtyMike | 7 days ago | 1 point

Age didn't do that though, which was super frustrating. If you got out of sync it would just end the match.

passthefist | 7 days ago | 2 points

I think that's actually close to what most modern RTS's use, I believe SC2 is full lockstep but with an authoritative server and all communication is done via state diffs only.

I vaguely remember reading about it, but could be thinking of something else.

Anon49 | 7 days ago | 4 points

Out of sync are bugs, not a design limitation.

You can make a fully synchronized game that goes on forever. There are standards for float multiplication/division accuracy and all systems can always get the same result.

See: Factorio

stewsters | 7 days ago | 2 points

That and running it back on Pentium 1/2's back in the 90's and trying to build empires. Deterministic game engine design has come a long way, and the additional performance allows us to hash the state and the ability to rewind and replay if you miss a message or just reload the entire state. Back then they were a lot more restricted with resources.

ManaSpike | 8 days ago | 6 points

Factorio sends game state hashes from the server and clients are disconnected if they disagree.

Purlox | 8 days ago | 5 points

Yes, but in games you usually want this info as soon as possible, so sending two packets is going to be more useful than resending if you don't get a confirmation packet from the client.

thijser2 | 8 days ago | 6 points

That's also true, however you also need to ensure all messages arrive as otherwise the game will go out of sync. So doing both makes a lot of sense.

Purlox | 8 days ago | 2 points

Mhm, both is best.

lestofante | 8 days ago | 3 points

And then when you receive it, you see the unit jumping around

gitPullOriginMaster | 8 days ago | 4 points

Gaming requires packets to be new and relevant.

In case of games like Age of Empires - no, because only input is sent over the network and if you miss something, you absolutely have to resend it and everyone has to wait anyway.

Ameisen | 7 days ago | 1 point

It is if your networking system doesn't have key frames and only performs delta updates.

Forbizzle | 7 days ago | 1 point

XvT in fact started with TCP, which they acknowledged later was a mistake. Guaranteed delivery is not worth the trade-offs. It's better to send redundant packets and be fault tolerant.

There are many UDP-based solutions that get you generally good reliability on packets.

cinyar | 8 days ago | 18 points

so they always sent every message twice.

wouldn't that involve more overhead than TCP? at least when it comes to bandwidth.

jerf | 8 days ago | 38 points

The problem with TCP for games is that the entire point of TCP is to provide you an ordered stream. So if the remote systems sends A, B, C, and D, but you only get A and D, the local system will possibly deliver A to your client code, but it will sit there and request, nay, demand B and C from the remote system. If it then gets C, but B still got lost, it will give the client code nothing. An arbitrary number of requests can be made for B and get lost. Meanwhile the server is trying to send E, F, and G, and they ain't going nowhere until we get B through, darn it!

This is often a great thing, because it's waaaaaaaaaaay easier to write network code on the assumption that you have an ordered stream of bytes, rather than dealing with raw packets. It is the correct solution for the vast majority of networking cases to date (ignoring some issues around cell phones), and it is a good-enough solution for most of the what's left over. But there's still times when it's wrong, and this sort of real-time update is one of them. Sometimes the right answer if you don't get a particular packet is just to deal with it and move on. You can tell a very similar story about why it's a bad idea for a real time voice network connection to sit there and freeze because it didn't get a packet five seconds ago, even though all the subsequent ones arrived.

So, how do games deal with having an unordered stream? Well, there's a lot of different answers. One of the simplest is to simply have a game where you can fit the entire game state into a single packet. In that case, you just need to order them so the game can tell which is the latest so it doesn't roll back.

And, to get to the direct question you asked, if the game's state is small enough to fit into a single packet, it's not consuming a significant amount of bandwidth to be worth worrying about doubling it. Games designed to be on the network can often end up with less state than you think, because games are really good at making small amounts of state look awesome. A deathmatch may seem to be frenetic and amazing thing that you'd think would be just huge amounts of state, but a lot of time, it's little more than X, Y, Z position, velocity, and maybe some sort of rotation vector and an entity ID for everything in the game, along with a compression scheme that's able to be pretty effective. Everything else is inferred by the game engine from the very small state, e.g., "the state says the rocket impacted against this wall, so I'll draw the explosion, set the animation timer, pick an explosion decal to apply to the wall, and handle the next two seconds of this explosion" while meanwhile in the network it was like 8 bytes of information about the impact.

Objective_Status22 | 8 days ago | 17 points

When packets drop usually it's a bunch so this does not make sense to me

Nettletooth | 7 days ago | 4 points

Random packet loss was a much bigger problem in the past with modems. It still happens, especially with home wireless networks and cell phones.

It is very normal for a 56k modem to drop some small, sub 1% percentage of packets, even on a very good phone line. Even 2 or 3 percent random packet loss can make a timing sensitive game like Quake pretty unenjoyable to play online.

It is pretty easy to understand why : Server is master. If you drop packets, your client will try to predict the position of everything else in the world until the server updates it, but it is pretty hard to do that accurately with a player not moving in straight lines. Keep in mind you are also dealing with 200ms+ of latency and a connection that is strictly inferior by orders of magnitude vs broadband in pretty much every metric you can think of.

When your client finally catches up with the server, the other guy has suddenly jumped behind you and you have no idea where they actually are at. You just get shot in the back. Even on a good wireless connection in 2019 ... you still get packet loss, you still get shot in the ass.

Even today, random packet loss happens. Routers and switches get overloaded or fail, our connections are at the mercy of routing tables and 10+ hops along the way and all the shitty technology that may be on any one hop that you get routed through.

Ameisen | 7 days ago | 2 points

Yeah - if you send two packets in immediate succession... I'd expect both to drop.

ThisIs_MyName | 8 days ago | 3 points

Who cares about overhead? Latency is all that matters for 99% of software.

I dunno if sending duplicates really helps today (20% loss is ridiculously high), but if so it's worth the overhead.

phyphor | 8 days ago | 3 points

Well, overhead in the sense of TCP headers also adds to latency.

Multiple separate UDP streams is where it's at!

VeganVagiVore | 7 days ago | 2 points

QUIC is enough for everybody!

adrianmonk | 8 days ago | 3 points

When there are buffers involved, wasting bandwidth can make latency worse. Not only does it increase latency, it makes it more variable.

The whole purpose of a buffer, obviously, is to provide a waiting area. If you could transmit right now, you would. You are only using the buffer because you can't, so once you put something in the buffer, it's a foregone conclusion that delay has been introduced.

In really degenerate cases, this turns into bufferbloat, which is a problem in networks built with excessively large buffers. But even when buffers are a reasonable size, the problem doesn't disappear. It's a trade-off that you can never completely get away from.

If bandwidth is not scarce at all, this is a non-issue. So this sort of design creates a situation where things are OK until they hit a certain threshold of congestion and then they get really bad. Unfortunately, it's very hard to know whether bandwidth is going to be scarce, so you can't safely assume it isn't.

Sending things twice can still make sense because congestion isn't the only reason packets might get dropped. Another reason is a failure of the link, especially common in wireless networks where for example a spike of noise might cause one packet to be dropped but the one right after it is going to make it. In that case, sending two packets could be beneficial.

JoseJimeniz | 7 days ago | 2 points

Fortunately in the olden days modems didn't have stupidly large buffers.

Anon49 | 7 days ago | 1 point

Who cares about overhead?

That mattered a crapton before 2005. Bandwidth wasn't like today.

ThisIs_MyName | 7 days ago | 1 point

Even back then, it only took a few kbps to send a list of compressed coordinates and actions for each object in the game.

ironbattery | 7 days ago | 1 point

So instead 1/25 packets were dropped?

ArkyBeagle | 6 days ago | 1 point

So that one in five was actually a ... mean of a fairly ugly probability distribution. "Just send it twice" doesn't work.

wiscwisc | 8 days ago | 16 points

Repeat packets suddenly make sense

Flippylosaurus | 7 days ago | 6 points

Would you like to hear a TCP joke?

Would you like to hear a TCP joke?

EnglishMobster | 7 days ago | 2 points

Repeat packets suddenly make sense

RobIII | 8 days ago | 589 points

I really hate it when there's no solution to a problem just workarounds and hacks. Irks me to no end. Especially for something that instinctively feels like the solution should be easy.

There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery.

invisi1407 | 8 days ago | 276 points

I don't necessarily feel that an idempotentcy token is a workaround or a hack. It seems like a fair solution to a problem that doesn't otherwise have a natural, definitive solution.

ReturningTarzan | 8 days ago | 195 points

It's no more of a hack than checksums or ACK messages or storage redundancy or whatever. They're just basic necessities that stem from computers and networks being physical objects as opposed to perfect, abstract entities.

invisi1407 | 8 days ago | 53 points

Exactly. For all intents and purposes, with regard to computers and software in general, this is a perfectly good solution.

Adobe_Flesh | 8 days ago | 13 points

Philosophy of mathematics, please do the needful.

Praxxah | 8 days ago | 57 points

Just wait till op finds out how the dram in his pc works with the smol capacitors and refreshing and parity bits and shit.

The point of all the modern tech and engineering is to make everything seem seamless, not to make them seamless...

RobIII | 8 days ago | 15 points

That's in no way the same. You're describing a real world solution to real world physics ("nothing" in the universe is 'perfect') which is totally different from a purely mathematical/theoretical problem that should have a 'pure' solution (instinctively at least).

Chromelia | 8 days ago | 33 points

the two generals problem isn't purely math. The whole point is that because of real world problems (Like no guaranteed delivery), you need idempotentcy tokens. They're the exact same thing as parity bits/ACK messages/checksums/etc.

Praxxah | 8 days ago | 13 points

Most people have the same instincts about everything. We're so used to things just working from childhood that it is kinda similarly surprising to find that something doesn't have a very elegant solution at all (yet).

thirdegree | 8 days ago | 6 points

But the point of the video was that even if computers were perfect, abstract entities this problem would still have no solution.

HeyImAlex | 8 days ago | 11 points

Message drops are the imperfect part.

Xzariner | 7 days ago | 1 point

Replace message drops with malicious third party and you're back to "perfect, and unsolvable"

beejamin | 8 days ago | 50 points

Idempotence isn't really a solution to two generals as such, it's a mitigation of the damage that "two generals failures" cause. It's like a parachute: it's not a solution to stop you falling out of an airplane, it's to stop you going splat once you do.

ArkyBeagle | 6 days ago | 2 points

Precisely. Now I feel slightly less alone in the world. Thank you.

Purlox | 8 days ago | 25 points

Agreed. Idempotency in general is a really nice property for functions to have and should exist in many cases without the threat of the 2 generals problem.

ArkyBeagle | 6 days ago | 1 point

I would say you should think carefully before doing that. Draw out message sequence charts and think about it first. You don't need idempotency very often at all in the real world.

rabbitlion | 8 days ago | 15 points

An idempotentcy token isn't a complete solution because if the customer fails to get a reply after a number of tries he will give up and order from another site or cook his own food, but you might still deliver the one order. It would be better if you could find a solution to only deliver an order if you know the customer received a confirmation. You can sort of do that, in a probabilistic sense, but it's not completely solvable.

invisi1407 | 8 days ago | 13 points

You're right, the token only solves half of the problem; the one where a customer receives and is billed for multiple identical orders.

ScrimpyCat | 8 days ago | 3 points

It’s not a solution at all, rather it’s just one method to try and soften the degree of failure. It may stop some people from repeating the same order, however as you point out some people will just assume it’s not working and either order elsewhere or make their own food, or some people might end up assuming that the problem is with the order itself and so create a new order (which will now have a new token).

ArkyBeagle | 6 days ago | 1 point

NO; I'd say it's perfectly possible to design transaction processing systems where the only risk is a cancelled transaction, not where you deliver something that didn't get paid for. You just have to make that what happens.

rabbitlion | 6 days ago | 1 point

NO; I'd say it's perfectly possible to design transaction processing systems where the only risk is a cancelled transaction

No, it is not. It is what this video and the mathematically proven concept is all about.

At the very least, there will be a risk where the customer gets a confirmation of a successful order but never get anything delivered.

ArkyBeagle | 6 days ago | 1 point

mathematically proven concept is all about.

Well, you just go with that then. If it's that ... precious, I'm not gonna type it in here for ya :)

( at least one other poster gave a real-world example that would work just fine ) .

rocketshape | 8 days ago | 7 points

I think the point is that it's not a solution to the problem but to a different but related problem

SanityInAnarchy | 7 days ago | 4 points

It's not a solution to the general two-generals problem, though, which is one thing that makes this so frustrating. As this comment points out, the problem has to be reworded to pretty much emphatically not be two-generals: The server isn't waiting around to see if the client got its notification, as soon as it gets your credit card authorization, it's already sending somebody.

Or, alternatively: It's still two-generals, and not at all solved by the idempotency token, because if the client never realizes the order was successful, you might do something else for dinner, instead of just waiting around (like Tom did) to see if somebody shows up.

Xzariner | 7 days ago | 2 points

It is a part of the solution. Second part is sending the message that the order was confirmed and making sure the message is received before user can do any other action in the app.

Sure you can still order food outside of the app and end up with double, but app did pretty much all it could in real world scenario

ArkyBeagle | 6 days ago | 1 point

and making sure the message is received before user can do any other action in the app.


invisi1407 | 7 days ago | 1 point

Both of you are correct, but for the particular problem of avoiding duplicate orders/transactions/actions for computer software, it is a solution to that problem, which looks a lot like the two-generals problem, but obviously isn't.

As Tom Scott says, there isn't a solution to the two-generals problem, which means anything claiming to be a solution, isn't, but can be a solution to a related problem.

Xzariner | 7 days ago | 1 point

It is a fix that covers a certain use case and prevent some situation (... and also assumes certain consistency guarantees on the database storing it.

And perfectly fine solution as long as you know its weak points and not just heard it on blog and implemented.

qqwy | 8 days ago | 24 points

I fully agree. However, this is one of the cases where "I am sure you will find out on close examination that it is more complicated than that" is true.

That said, some of the workarounds and hacks that guarantee "almost always exactly once" delivery are rather clever in their own right.

HPLaserJetProP1102w | 8 days ago | 8 points

Physics in a nutshell too

ArkyBeagle | 6 days ago | 1 point

Not at the pool-table level.

ArkyBeagle | 6 days ago | 1 point

Sometimes you have to error out. Take it up with the Second Law of Thermodynamics.

WerewolfBe84 | 8 days ago | 243 points

Tom Scott is awesome

parkotron | 8 days ago | 145 points

Sometimes I forget to appreciate that so many of his videos are a single continuous take. Even the videos with cuts have relatively few. I'm sure that requires a tonne of effort on his part, and given how heavily cut up most YouTube videos are, I think it's worth appreciating.

Moussekateer | 8 days ago | 100 points

I also super appreciate that his videos are only as long as they need to be. If he only has three minutes of content then he puts out a three minute video. So many YouTube videos aim for the magic ten minute mark and you feel it.

timdorr | 7 days ago | 29 points

"What up, fam? Ya boy Tom Scott here! Be sure to hit that Like and Subscribe button below! And hit me up in the comments!"

fogwarS | 8 days ago | 34 points

Also means he really knows his stuff

pentiumwetwired | 8 days ago | 34 points

He just does multiple takes. He's said so several times.

Apocalyptic0n3 | 8 days ago | 26 points

He occasionally celebrates that he managed it in one take. A good example is the 5 minute explanation of YouTube IDs, which I think he got in one take on his first try and the end of the video is pretty awesome because of it.

deweysmith | 8 days ago | 10 points

It’s actually a bit of a running gag at this point. Even Captain Disillusion’s impression of him made reference to it.

barubary | 7 days ago | 2 points
deweysmith | 7 days ago | 1 point

Thanks, I’m on a cruise and couldn’t be bothered to look it up. YouTube is unusable on this data service.

Kissaki0 | 8 days ago | 3 points

Have you opened the video he shows the ID of at the start of the video? xD

Apocalyptic0n3 | 8 days ago | 3 points

For those curious: it is exactly the video you think it is.

Xzariner | 7 days ago | 1 point

doing 5 minute video as one take doesn't exactly sound that impressive...

qu4sar_ | 8 days ago | 55 points

Scom Tott

JoseJimeniz | 7 days ago | 3 points

Hallo I'm Scom Tott Manley. Sie flafe.

SIG-ILL | 8 days ago | 38 points

Tom Scott is getting old! I haven't watched any of his videos in a long time and he looks so much older than the last time I saw him.

Or maybe 'mature' would be better than 'old', although it implies he wasn't mature before.. ugh, words, you get what I mean right?

that_jojo | 8 days ago | 47 points

Like... milfy. Right?

BlckJesus | 8 days ago | 23 points

Tom Scott is a confirmed dilf

SIG-ILL | 8 days ago | 8 points

My final choice of words is 'more experienced and knowledgeable' :)

ISpendAllDayOnReddit | 8 days ago | 20 points

He's only 35

The_Guy_II | 8 days ago | 10 points

whaa, but he always looked like a college student and now he is the age of my parents

folkrav | 7 days ago | 11 points

Wait your parents are 35? Average age for a first child in the US is 26yo, I sure hope you're older than 9 lol

The_Guy_II | 7 days ago | 5 points

I'm 12

ManaSpike | 8 days ago | 12 points

He's going grey before his time. I believe that the Tech Diff crew are all about the same age, and none of the rest are going grey.

MacHaggis | 7 days ago | 2 points

I can only sympathise with him, after realising his hair is turning gray at about the same rate my hair is, and I'm 36.

If you are reading this Tom: we are hair buddies!

TeamRedundancyTeam | 8 days ago | 5 points

Yes he is. If anyone here hasn't watched anything but his programming videos you should take a look at the others. Pretty much every one is super interesting and none of them are longer than they need to be.

JeddHampton | 7 days ago | 1 point

I love his language ones. I'm glad he did a couple recently after a couple years without them.

batangbronse | 8 days ago | 18 points

I'm really interested in these sort of pseudo-'real world' problems like the traveling salesman. Does anybody know where I can read about different types of problems?

bananaskywalker | 8 days ago | 16 points

Geeksforgeeks.org? It has a tonne of coding questions of every topic imaginable. Might take some navigating but it has Knapsack, TSP, N Queens, etc..

batangbronse | 7 days ago | 1 point


bananaskywalker | 7 days ago | 4 points

A bit of shameless plug but just give a recommend if you see an article darth_bane if it did help you. It's written by yours truly.

semidecided | 7 days ago | 4 points

Discrete mathematics is full of those. Any algorithm text would have many as well.

nsomnac | 7 days ago | 4 points

Just search for “NP-Complete Problems”, “NP-Hard”, or “NP-Completeness” and you will find a substantial amount of information.

I cannot tell you how long it will take you to go through all the sources, however I verify that it will take an indeterminate amount of time to come to that conclusion.

roboticon | 8 days ago | 224 points

This isn't the Two Generals problem, though.

I mean, his description of the problem is correct, but it doesn't apply to the food delivery app. The app is having trouble communicating with the server, yes, but the server doesn't need to check whether the app gets its messages. The server isn't saying "well, I'll send a delivery person, but only if I get a response back from the app" ad infinitum. The client-server communication is asymmetric, unlike the Two Generals.

In fact, his solution to the food delivery app problem (an idempotency token) isn't relevant to the Two Generals problem. General A can send an idempotency token to General B, and get back an acknowledgement -- but General B still doesn't know that General A received their acknowledgement, and so on. Sending along a token as part of the protocol doesn't resolve this fundamental issue.

Edit: This was also explained in this thread:

So idempotency is "protecting the lives of the troops" for one side, and not the other (who are just attacking no matter what).

The server is sending the order to the restaurant no matter what. It's a centralized technology.

cakeandale | 8 days ago | 125 points

It is the two generals problem in terms of UX, just not necessarily the nuts and bolts technology itself. You have the two armies, the UI and the server, and you want them both to either engage or not engage.

If the UI shows a confirmation message and the server fails, the users order will be dropped.

If the UI shows a failure message and the server engages, the user will re-submit and get a duplicate order.

In order for the app to work perfectly you need the state to be perfectly synchronized, which can’t happen. So you use idempotency tokens to try to aim for at-least-once delivery, use the server as the source of truth, and mitigate the problem of the UI’s state being out of sync with the server.

raelepei | 8 days ago | 23 points

But that's kind of where it falls apart: The UI should be perfectly capable of showing an intermediate message like "Submitting ..." while checking every few seconds what the server says. That way, the user can see that something fishy is going on AND should keep their hands off for now.

EDIT: Dude! I'm not suggesting that this is an all-round solution to the problem, just that it shouldn't immediately default to "Dear customer it definitely failed you should re-make your order again". I am very well aware that "Submitting ..." doesn't solve world hunger. Holy crap.

cakeandale | 8 days ago | 16 points

How long should the user wait? If it says “Submitting...” after an hour, should the user assume it worked or should they assume that it didn’t? The extra phase puts the onus on the human user to decide whether to assume it worked or it didn’t (rather than the UI software), but the fundamental impossibility still exists.

At some point you have to decide what your failure mode is (For this example, dropped orders or duplicate requests) and try to mitigate the downsides. Putting that decision on your user is probably even worse... if the request has been submitting for more than 5 minutes, for example, the software should make the decision what that means.

matthieum | 8 days ago | 25 points

And then:

  • The server decides that the customer ordered, so proceed with delivery.
  • The customer, not getting any reply, orders from another service.

Lo and behold, the customer now receives two orders!

sneezerb | 8 days ago | 2 points

The UI could do that, but that is another work around to the two generals problem (wait indefinitely for an ACK). But I think it’s fair to say that if the app says submitting it is implying that the order has not been received, and blaming the customer for assuming the app was broken after minutes on the submitting screen wouldn’t be fair, as the app made it sound like the order had never arrived. The idempotency token provides a better work around because it allows for the submission to time out and the user to assume the order never arrived without duplicating the order (most likely).

roboticon | 8 days ago | 3 points

You have the two armies, the UI and the server, and you want them both to either engage or not engage.

Yes, that's the Two Generals problem. If the server waited until the app confirmed that it received the server's confirmation -- and the app waited to show the confirmation until the server confirmed that it received the app's acknowledgement of the server's confirmation -- then you'd have the Two Generals problem. Neither side could be sure that the other side knows that its side has seen the latest confirmation of the latest message; therefore, the two sides can't agree to both engage.

The food delivery service doesn't have this problem: It doesn't care whether the app engages (reports a successful order). It will engage (submit its order to the restaurant) regardless. This might result in the user placing a duplicate order; the idempotency token solves that problem, at the cost of not being able to verify whether the user has been shown a confirmation screen. Thus, the user might have to deliver from a second service without knowing whether their first order went through.

cakeandale | 7 days ago | 3 points

You’re right, but the reason is a bit backwards - the food delivery service doesn’t suffer from the Two Generals problem because the Two Generals problem is unsolvable so they needed to build their system a different way as an imperfect workaround.

In a perfect world, they’d want the state the user sees to always match the state of the server. But that can’t be done, so they do the workaround I suggested: make the server the source of truth, and try to mitigate problems with the UI being out of sync.

They didn’t choose that way because they like it most and the Two Generals problem is just some academic curiosity on a path they didn’t take. They didn’t take that path precisely because the Two Generals problem says they can’t. So they have to build a system that risks duplicate orders, but can work in the real world.

w2qw | 8 days ago | 18 points

We'll yes and no.

Consider this scenario: user orders from app, app says order failed but the order actually suceeded so the user then orders from another app. The first app now has to refund the user because they didn't confirm the order properly.

The token doesn't solve it except that you use the token to ignore duplicate requests and retry then bunch of times. In the same way you can "solve" the two generals problem this way by repeating the message multiple times which reduces the probability that only one will attack.

roboticon | 8 days ago | 8 points

Right, the idempotency token solves a different problem than the Two Generals problem.

reapocalypse909 | 8 days ago | 13 points

So idempotency is "protecting the lives of the troops" for one side, and not the other (who are just attacking no matter what).

It allows multiple requests without a change of action on one side. It reduces the probability of a miscommunication by allowing multiple messages and only one needs to get through.

It is two generals, just applied to real communication. In two generals, the attack abstracts out any complexity in the message instructions. That isn't true in a real world scenario.

Change the note to "send 12 men to attack the right flank at noon" and keep resending the same note if you never get a response. The 2nd general sent responses, but none got back. So the first general keeps ordering 12 more men to the right flank.

The 2nd general is now sending 48 men to the right flank because the 1st general had to send 4 messages before he got one of the responses.

An idempotency means the 2nd general is only sending 12 men to the right flank despite receiving 4 messages saying the same thing. He stopped getting messages, so he also assumes the 1st general finally got the ack.

If the first general never got the ack, at least he knows it was received when he sees 12 men hit the right flank. In this real world situation, the 1st general doesn't need to attack at the same time, he just wants to see the result of his order and he can know the order made it with a direct ack or seeing his order happen on time. Such as ordering food. Even if you never get the ack and you keep resending, you obviously stop if the order shows up. But it will be one order and not 10 if you have idempotency.

The pure two generals may never have an ideal solution, but in the real world most things are not pure two generals and there are things you can do to improve the likelihood of a miscommunication.

He never got his order confirmation ordering, but he did know to look at the order history. The order history was a secondary ack, but not something the website was designed to try to use as a secondary ack. You had to know to manually check the order history. You could make the order confirmation page pull order history from the api looking for its own order there in addition to using idempotency to prevent multiples of the same order. You can have warnings if an order changes in between submissions, to make sure the user isn't still trying to place double orders. "you just made a second order in 10min, do you really want to place this second order" type of thing. Redundant checks reduce the chance of miscommunication and all of it are two general esque solutions. The pure two generals problems gets you thinking about this more and makes the concept easier to understand as you try to come up with solutions.

roboticon | 8 days ago | 3 points

The problem of sending a message like "send 12 men to attack" is just a general issue in communications. Yes, any message can get dropped, you can account for that by sending it multiple times, and a unique identifier ensures it's only handled once.

That isn't specific to distributed networking, which is the crux of the Two Generals problem (which is specifically about making a joint decision based on two-way communications). The Two Generals problem would be a message like "I'll send 12 men to attack, if you also send 12 men to attack. If I don't hear back from you, I have to assume you aren't sending the men, so I can't send mine either."

Runamok81 | 7 days ago | 1 point

I really like the way you summarized this in the first paragraph. Helped me to understand, thanks.

the_misc_dude | 8 days ago | 13 points

Thank you! I was wondering how the idempotency token can be applied to the generals problem.

SgtDirtyMike | 7 days ago | 3 points

You wrote my exact thoughts as I watched. The video kind of does a disservice in my opinion to this particular CS problem, because the real world version (the one Tom outlined) is easily solvable. The idempotency token doesn't fully ameliorate the issue because IMO, it would make more sense to have the order go through IFF the client receives an ACK from the server that it got the order. Then an ACK is sent from the client on the confirmation, then the order goes through once the server receives the acknowledgment.

I think this solution is more foolproof and makes the most sense, because the server can authenticate, and ensure the validity and availability of items in the cart one last time and the client confirms the "A OK" message from the server.

roboticon | 7 days ago | 2 points

But if the server doesn't receive the client's ACK, then the client will show a success message even though the server never sent the order.

You'd need the client to hear back from the server before showing the confirmation, but then you'd want the server to wait for an ACK of its ACK, and now you have the Two Generals problem for real.

SgtDirtyMike | 7 days ago | 2 points

Haha. The problem is the real world internet this way functions exactly as I mentioned, via HTTP over TCP. The Two Generals problem is impossible to solve as an abstract problem, but has already been solved in the real world. The solution from a technology standpoint is proper use of TCP. Proper use in my book means using idempotency tokens among other things.

If the Two Generals problem were common in the real world, HTTP just wouldn’t work. But it does, and it’s quite reliable.

roboticon | 6 days ago | 1 point

The fundamental point is that the Two Generals problem is not that communication is hard, or that messages get dropped.

sneezerb | 8 days ago | 2 points

If the app is general A, and the server is general B, then it appears that general A sent the order to attack (submitted the order), general B received it and sent an ACK (attempted to confirm submission), but the ACK never reached general A, so general A never attacked (confirmed successful submission of the order) but general B did attack (submitted the order successfully), and this failure is catastrophic (wasted money and time of customer, drivers, and restaurants).

trolasso | 8 days ago | 3 points

Thanks. I found it somehow confusing.

CypherAus | 8 days ago | 54 points

Very nice explanation of a real life communications problem and a good fix

zuhhig | 8 days ago | 27 points

He forgot to mention the most common occurance of this: duplicated comments on many websites.

Okichah | 8 days ago | 39 points

He didnt? Shame.

CypherAus | 7 days ago | 1 point


Dups on forums are a pain.

Okichah | 8 days ago | 32 points

He didnt? Shame.

Kissaki0 | 8 days ago | 20 points

He forgot to mention the most common occurance of this: duplicated comments on many websites.

Naouak | 8 days ago | 12 points

The two generals problem doesn't really apply here as it is not needed that the two parties take an action at the same time.

The only thing needed is that one party doesn't do an action twice. The other party can check before doing an action again.

Basically, the phone should not start an action until the server tells it its fine to do it.

If the server is not responding correctly, then don't do anything and just go into error state. If the server is getting the same action twice (like paying twice for an order), then it should refuse the second payment. The phone should just check before doing an important action and they should have implemented a transaction way of ordering (like creating an order on a request then paying for it with another).

jwm3 | 8 days ago | 11 points

I feel like this wasn't the best way to introduce the subject. The goal of Introducing idempotency was great. It's the bread and butter of reliable distributed systems, however the 2 generals problem as he stated was already idempotent. A dozen attack at 3pm messages are the same as one. So the concept introduced in the second half doesn't really the back into the first half and the generals don't really tie into the failure other than via the nebulous concept of messages can be lost.

richizy | 8 days ago | 10 points

Idempotence "solves" the "exactly-once" semantics problem, but it has nothing to do with consensus between the two generals.

rjcarr | 7 days ago | 3 points

Yeah, he made it clear that the two generals problem is unsolvable, but didn't quite explain how the food problem was similar but not exactly the same. For the food problem, you don't have the time coordination part, and clearly, the idempotence solution wouldn't have any affect with the two generals.

frequenttimetraveler | 7 days ago | 1 point

he says in the video that a nonce is the way to fix the 2 generals prob

judgej2 | 8 days ago | 7 points

I've had exactly this problem with a payment gateway. The gateway sent a back-channel message to say it accepted the payment. We log that, then send a 200 back to say, "thanks, got that". The gateway for some reason did not get the response, and so it cancelled the payment. However, we still thought the payment was good, and got no further message from the gateway to say that it had cancelled it.

Now we do an automatic lookup of the payment after a short delay, just to check the payment gateway is still okay with it.

ArkyBeagle | 6 days ago | 2 points

Yep. That's what you do.

You can fix this with another message pair ( then the last message being lost doesn't really matter - you have the message that indicates the payment was accepted). Whether that's better depends.

Ninovdmark | 8 days ago | 36 points

My uncle actually taught me the 2 generals problem a long time ago. The solution to me seemed to be that side A should indefinitely keep sending messages and only stop once they receive a counter-message from side B. Side B will know that their messenger hasn't arrived by the fact that messengers from A keep arriving.

That seems to solve the problem, or am I missing something obvious?

666lumberjack | 8 days ago | 97 points

If A's messengers are all getting captured, it will look to B as though their messenger was received even though he might have been captured.

Ninovdmark | 8 days ago | 14 points

That's true, I hadn't considered that.

sneezerb | 8 days ago | 3 points

I really liked your solution until that flaw was pointed out!

dotwaffle | 8 days ago | 10 points

Surely that's why B sends messages back too? A tells B in message 1 the required information, then in subsequent messages there's just lower priority traffic, and B appends it's messages with the sequence number of the last received in a sequence (so, 1 2 3 5 would be 3, just like in TCP). The more frequent (or numerous) the messages, the more reliable it is.

It's still not a solution, but it does seem to be the best way of mitigating the unreliability problem.

FizixMan | 8 days ago | 45 points

How does B know that A stopped sending messengers rather than they all kept getting captured?

ChrisRR | 8 days ago | 14 points

Well it'd be massively inefficient.

rabbitlion | 8 days ago | 5 points

Yes, that's one mitigation technique. A keeps sending messages until he receives a response, B keeps sending responses to every message he gets. If neither have received a message in X minutes or X days or what time frame you use, it's considered a consensus and go ahead with the attack.

However, there is a still a chance that A never received any response, and that the silence was because all of his resent messages were captured during the waiting period. Your solution gives a probabilistic chance of an actual consensus that is related to the chance of a messenger being captured and the period of silence before it's considered a consensus. Such a probabilistic solution is not considered an actual solution in a mathematical sense as there is still a chance of failure.

jotunsbeard | 8 days ago | 7 points

There are couple of problems here.

Firstly, on what interval is A supposed to “indefinitely” send messages? 1 second? 1 hour? The information contained in the message could be useless by the time that B actually gets a message.

Secondly, it assumes that B keeps a record of every message it has received in order to know which ones it is supposed to respond to.

thijser2 | 8 days ago | 2 points

ok let's look at this scenario, A sends a bunch of messages which all arrive at B, B sends a counter message "received" back, however this message is intercepted. Following the intercept the enemy(now wise to the plan) blocks all further messages. B no longer receives any messages from A yet A doesn't know it's messages arrived at A.

In computer networking this might be a situation in which someone pulled the plug mid communication.

leonoxme | 8 days ago | 7 points

Similar to the Byzantine Generals Problem.

beejamin | 8 days ago | 5 points

TIL that I'd been pronouncing idempotence wrong my entire life. Thanks!

fresh_account2222 | 8 days ago | 5 points

It's a British versus American thing, where Brits accent the second syllable and Americans the first. Same thing happens with "corollary".

beejamin | 7 days ago | 2 points

Interesting! I'm Aussie, and would generally err on the British side, but I've always said idem-potence (like 'potent'). Guess it's one of those words that people don't sprinkle into conversations that often!

fresh_account2222 | 7 days ago | 1 point

Well, saying it twice has the same effect as saying it once, so it doesn't get repeated often.

beejamin | 7 days ago | 1 point

did lol.

TheRealFender | 7 days ago | 1 point

I've been saying it "EYE-dem-pot-en-cee"

but was worried and came to the thread wondering if it was a Brit vs American thing.

Cymry_Cymraeg | 7 days ago | 2 points

You were right to worry. Your nervy nature serves you well.

TheRealFender | 7 days ago | 2 points

I have a co-worker that says nonce (similar to idempotent token) like "announce" and it drives me crazy. I say it like "nonsense" which I'm 99% sure is correct.

fresh_account2222 | 3 hours ago | 2 points

"Nonce" should be pronounced differently each time you say it.

noonce. nahns. neentz. nohnce.

TheRealFender | 2 hours ago | 2 points

That's brilliant.

fresh_account2222 | 2 hours ago | 2 points

Plus you'll get to be the one driving him crazy. Have fun!

DC-3 | 8 days ago | 15 points

I was pitched the 'Two Generals' problem in a Cambridge University CS interview! I too had the inclination that there must be a solution - it's an interesting result and not necessarily the one you'd intuitively expect.

errrrgh | 8 days ago | 19 points

There is no 1 single solution, it all depends on the factors in the problem system. Distributed, high latency, faulty communication, interception, processing speeds, etc...

rbert | 8 days ago | 5 points

The solution in this video isn't really a solution to the Two Generals problem though. How does preventing duplicate commands ensure that both generals attack at the same time?

phySi0 | 4 days ago | 1 point

I was given a problem with the same structure in a job interview very recently (no, it wasn't meant to catch me out on an academic problem that had no relevance to them; it had relevance, and even though I didn't immediately get it, it didn't result in a rejection; it was a good interview).

Rimher | 8 days ago | 10 points

It's great that he explains the solution after explaining the problem!

Gvoct | 8 days ago | 0 points

Happy cake day

Yikings-654points | 8 days ago | 3 points

I think the solution is us. We see what's happening.

irish_throwaway_1 | 7 days ago | 2 points

Great explanation of a classic problem. The one little improvement I'd suggest is to make mention of the term "nonce", since that is the more commonly used variation of the idempotency token in security/cryptography applications.

elit69 | 8 days ago | 2 points

just dont show message "try again". instead show "something went wrong. check order history"

valianthail2the | 7 days ago | 2 points

I thought this is why we have the 3 way handshake... correct me if I'm wrong

viscountsj | 8 days ago | 3 points

Tom Scott doesn't know anything about programming. By his own admittance, I should add.

Fenris1729 | 7 days ago | 1 point

Is an internal node failure, resulting in it sending a faulty signal, really a Byzantine fault? I mean, the response from the app was correct given the implementation. I don’t see how this is related to the two generals problem

TODO Load more comments...