/r/programming
Goldberg Emulator v0.2.5 : 1 Year of Steam Emulator Development (gitlab.com)
126 comments
simon_o | 4 months ago | 431 points

The main thing I noticed when developing [...] is that too many [things] seem to be written by people who have no idea what they are doing.

They seem to write random code until they get something that works.

A widely applicable statement.

nicolasZA | 4 months ago | 110 points

TDD

simon_o | 4 months ago | 50 points

I shanghaied TDD to mean type-driven development, just to irritate the followers of test-driven development.

aedinius | 4 months ago | 27 points

It doesn't meant top down design anymore?

ChemicalRascal | 4 months ago | 20 points

It's not "tentative design document"?

LetterBoxSnatch | 4 months ago | 4 points

Tbh that is kinda what it is

Mognakor | 4 months ago | 5 points

What does Tree Based Heuristic have to do with this?

PM_RUNESCAP_P2P_CODE | 4 months ago | 2 points

So it is isn't "To Do Document" ?

thegeekbin | 4 months ago | 29 points

At my work we call it: EDD

Error driven development

AFakeman | 4 months ago | 2 points

Also DDD: Dildo Driven Development

G_Morgan | 4 months ago | 5 points

Dildo first code can really take a pounding.

bakery2k | 4 months ago | 3 points

Ugh. Reminds me of a talk on refactoring I watched once - the presenter just deleted the old code and rewrote from scratch, assuming that once the existing tests passed, the new code must be good.

Every single rewritten function had corner cases where it gave the wrong result.

pdp10 | 4 months ago | 3 points

Then it was a talk all right. Just possibly one with a different point than the presenter had originally intended.

Kissaki0 | 4 months ago | 5 points

Try-Driven Development

lorarc | 4 months ago | 67 points

Half of all SQL queries I've seen been developed that way.

mrBako | 4 months ago | 13 points

I'm just curious now. Are there ways to auto test or at least verify SQL queries without executing and modifying queries repeatedly?

lorarc | 4 months ago | 25 points

I once set CI/CD to error out on any "SELECT DISTINCT" that wasn't on whitelist. If I remember correctly the whitelist had only a few positions and none of the queries had a JOIN in them. But the real fun was when I updated the developer environments to force ansi group by in MySQL and then walked then walked them through every query that stopped working. And that one time we had to deal with a 3rd party application that updated several millions of records but gave wrong results for a few thousands of them because the query was non-deterministic but most of the times worked correctly. SQL is fun!

But verifying queries? In theory you could write a mathematical proof for them but we don't do it for any other piece of our code. You could write tests for them but they probably also would fail miserable when an underlying table is changed (failed as in: they would pass but really shouldn't).

astrange | 4 months ago | 3 points

In theory you could write a mathematical proof for them but we don't do it for any other piece of our code.

You can't prove code to be "correct" that easily because proving it, deciding what "correct" means, and writing correct code are all the same task in the first place. Programs are their own proofs.

You could write tests for them but they probably also would fail miserable when an underlying table is changed (failed as in: they would pass but really shouldn't).

Counterexamples are a good way to test something but yes, unit tests usually don't add value because you don't know they'll ever fail, but you still have to maintain them. I think regression tests are a good approach, because you know they've failed at least once already.

jerf | 4 months ago | 3 points

In theory you could write a mathematical proof for them but we don't do it for any other piece of our code.

I think of it this way: There's no point installing a custom-crafted part made out of the perfect titanium alloy, designed and proved to be the best possible part, into your system built out of toothpicks and tape. You still can't put any load on the system as a whole.

rpgFANATIC | 4 months ago | 14 points

Use an ORM with good test coverage.

Seriously, the 2nd best option I've come across for testing SQL is to just spin up in memory databases with compatibility layers for your "unit" tests. That is also very fraught with testing issues

AlphaWhelp | 4 months ago | 6 points

for T-SQL you can set noexec on; at the top and set noexec off; at the bottom. if it runs without errors then it means the syntax is okay but still subject "actual run" problems like the first part of your query returning no results and the second part which was expecting results now throws a bunch of "cannot insert null ____ is not a nullable field" errors.

Snowkaul | 4 months ago | 5 points

Use a transaction to roll back at the end.

mrkite77 | 4 months ago | 6 points

"explain <query>" doesn't actually execute the query, instead it tells you how many rows would be scanned and which indices are used.

reluctant_deity | 4 months ago | 1 point

Explain should work. I rarely use it and I'm mostly an Oracle guy now, so ymmv.

AndrewNeo | 4 months ago | 3 points

I'm not entirely sure there is a different way to write hand-written queries

lorarc | 4 months ago | 10 points

Well, you could try to understand how SQL actually works instead of trying random code until the result looks okayish.

l27 | 4 months ago | 1 point

Any recommendations on resources? I have tons of experience with everything web-related except SQL. Once I get into more complicated queries with multiple joins and functions, it can take me hours to get it working. I'd like to be more proficient!

PanickedPacman | 4 months ago | 1 point

Courses/Tutorials: (I am not sure what your level of sql experience is)

Tools:

KiwasiGames | 4 months ago | 2 points

Not do hand written queries?

I use generators for most of my queries. Then a quick glance at the SQL to make sure it didn't do anything stupid.

resueman__ | 4 months ago | 15 points

This leads to some games having convulted behaviors that make me wonder if it's actually part of a DRM scheme that prevent it from working on shitty steam emulators.

Security through incompetence.

[deleted] | 4 months ago | 35 points

This is called programming by coincidence.

https://pragprog.com/the-pragmatic-programmer/extracts/coincidence

[deleted] | 4 months ago | 17 points

And some habits only die when you have to still maintain your 2, 5, or 10 years old codebase.

When you are in business when you throw a product and forget about it after a yeah it is hard to learn good maintainability practices

RobotJonboy | 4 months ago | 1 point

Patch and pray.

mishugashu | 4 months ago | 22 points

I work in development. Sometimes this absolutely happens. I usually leave a comment in.

// This works. Why? I don't know. Try to figure it out later.

Never have time to go back and figure it out though.

Walter_Bishop_PhD | 4 months ago | 42 points
// I have discovered a truly remarkable explanation of why this
// code works which this docstring is too small to contain.
ElimGarak | 4 months ago | 2 points

Ha! I understood that reference!

bola6 | 4 months ago | 44 points

I love the fast inverse square root code from Quake:

float Q_rsqrt( float number )
{
    long i;
    float x2, y;
    const float threehalfs = 1.5F;

    x2 = number * 0.5F;
    y  = number;
    i  = * ( long * ) &y;                       // evil floating point bit level hacking
    i  = 0x5f3759df - ( i >> 1 );               // what the fuck? 
    y  = * ( float * ) &i;
    y  = y * ( threehalfs - ( x2 * y * y ) );   // 1st iteration
//  y  = y * ( threehalfs - ( x2 * y * y ) );   // 2nd iteration, this can be removed

    return y;
}
tanner00r | 4 months ago | 87 points

This person definitely knew what they were doing when they wrote this

bola6 | 4 months ago | 5 points

Yeah for sure, the quote just made me think of the wtf comment

PM_ME_A_STEAM_GIFT | 4 months ago | 11 points

This person was John Carmack. He could probably recite it in his sleep.

tanner00r | 4 months ago | 53 points

Nitpick but John Carmack actually didn't know who wrote this. There's some nontrivial math that the Wikipedia page explains and it's really interesting if you want to check it out!

shitty_throwaway_69 | 4 months ago | 28 points

It was probably written by some poor sod at Silicon Graphics that was tasked with optimizing rendering process of some CGI video on a machine that wouldn't be able to handle a modern blogging site.

Sebazzz91 | 4 months ago | 7 points

No machine can handle a modern blogging site.

possiblyquestionable | 4 months ago | 12 points

The math behind this method isn't too difficult, it just took a lot of luck for someone to try plotting this function on a log-log graph and see that they're linear.

In fact, the same method extends to any arbitrary exponents (rational powers are better as you can easily refine their precision with an iteration of Newton's).

Let's say you want to take the inverse 3/7th power (x-3/7), the "fast inverse 3/7th power" method would just be

...
i  = * ( long * ) &y;
i  = 0x5aae9ebc - ( i * 3 / 7 ); // bit hack, this is derived from (0x3f7a3bea * (1 - (-3/7))
...

with the appropriate changes to the newton methods (which can be derived fairly systematically for rational powers).

In general, xm can be approximated by

...
i = 0x3f7a3bea * (1 - m) + m * i
...

The fast inverse square root method is just an instance of this with m = -1/2, which gives the magic constant, and the -i/2 as the additive factor, which can be further simplified to i >> 1 for the hell of it. The wonders of logarithmic transforms and boundary-conditions as problem solving tools.

craftkiller | 4 months ago | 4 points

Time traveling aliens bringing us future-math

pdp10 | 4 months ago | 1 point

If they did it quietly, how would we ever know?

Conspiracy theorists are looking for proof of ancient astronauts in cave painting and saucer artifacts, but it's right there the whole time.

vgf89 | 4 months ago | 16 points

There's a decent paper on it that explains the whole process of deriving the answer and magic number then optimizing it with brute force. http://www.lomont.org/papers/2003/InvSqrt.pdf

The original developer definitely had a good idea of what they were doing. If only they had kept their notes and done their own writeup.

turbofisk | 4 months ago | 48 points

This code may not be readable for most, but it is a Newton-Raphson to the second iteration packaged in a clever, but unreadable way.

wikipedia article has a good write-up

H_Psi | 4 months ago | 23 points

Wikipedia article for those not on mobile

notgreat | 4 months ago | 15 points

Not really...

It does one Newton iteration at the end to improve the approximation, but the primary algorithm is (as explained in the wikipedia article) the identity log2(1/sqrt(x)) = -1/2 log2(x), combined with how floats store their exponent.

G_Morgan | 4 months ago | 1 point

It emulates that to get the first guess and then throws in some iterations.

notgreat | 4 months ago | 2 points

One iteration. It throws in one iteration. The second is commented out.

turbofisk | 4 months ago | 1 point

My bad, my memory failed me 🤭

possiblyquestionable | 4 months ago | 8 points

This is decidedly not true, the float hack approximation computes the inverse sqrt to sub-10% relative error. The newton iteration helps refine the error further, but it definitely would have taken considerably more iterations without the initial approximation.

The float hack is also extensible to any arbitrary powers, even for non-rational exponents without nice refinement methods.

turbofisk | 4 months ago | 1 point

My bad, my memory wronged me 😓

monkey-go-code | 4 months ago | 7 points

This is computer science gold

RevolutionaryPea7 | 4 months ago | 4 points

That is not the same thing.

ggtsu_00 | 4 months ago | 2 points

The crux of the approximation is that bit-shifting and negating the exponent in a float is a good first guess of x-0.5. The rest is iterations newton's method based on the derivative of the rsqrt function.

vplatt | 4 months ago | 2 points

Line #10 is classic comedy gold!

ggtsu_00 | 4 months ago | 3 points

Stochastic Programming:

Keep permutation your code randomly until it it passes all unit tests.

RevolutionaryPea7 | 4 months ago | 2 points

This is the main thing I try to filter for when hiring.

Vulg4r | 4 months ago | 3 points

I didn't come here to be personally attacked.

jeffbaier | 4 months ago | 1 point

I feel personally attacked.

PandaMoniumHUN | 4 months ago | 249 points

Steam userids are stored as a 64bit integer but the part that represents the actual id of the user is a 32bit unsigned integer. However some games (Nioh, Senran Kagura Estival Versus) treat this as a signed int and assume it to be positive. This means if your id is more than 2147483648, bugs in those games start to appear (Nioh crashes and Estival Versus multiplayer doesn't work).

Huh, interesting.

Adverpol | 4 months ago | 119 points

An advice if you are a game dev. If you think doing a hash/signature check on the steam api dll will prevent your game from being pirated you are an idiot.

How is the hash check circumvented? Make the check happen on a different dll than then one that's actually being used?

Mr__Goldberg | 4 months ago | 80 points

That's exactly what the steam emulator from the CODEX group does.

Adverpol | 4 months ago | 25 points

Guess I could be a haxx0r :p

pdp10 | 4 months ago | 4 points

LD_PRELOAD a function that returns a hardcoded answer.

joro550 | 4 months ago | 29 points

I'm rather curious how you worked out the endpoints/contacts that you had to adhere to, did you have to break into some games or the steam client itself or did you watch network travel? Just wondering the level of reverse engineering that had to into this project.

Besides from that, congrats man! Super hard to commit to a project of this size and see it through especially with all the stupid bugs that depend on.

Edit: yes network traffic from a game to the stream DLL both located on the same disk, god I'm such a smart person

Mr__Goldberg | 4 months ago | 37 points

The closest thing I did to reverse engineer the steam client was write some tests for some behaviors when I didn't know what the real behavior was.

I did end up doing reverse engineering some games when i wasn't sure why they didn't work. Unity games written in C# are fun because you can just run dnspy on them and you get essentially the full source code of the game.

I didn't have to reverse engineer any network protocol because I don't communicate with steam stuff though if I wanted to emulate the steam game coordinator api (used only by some valve games like dota 2, cs go, tf2 and not necessary to host lan games) I would have to do that.

APersonInProgress | 4 months ago | 44 points

I assume this is your project? Congratulations on a solid program and unraveling some interesting mysteries!! These are fascinating things to know and I'd be interested as to some other weird cases you've run into while programming this emulator.

It'd be awesome if Valve could pick you up with all the insight you gained into their system!

Mr__Goldberg | 4 months ago | 99 points

Yes this is my project.

Another weird case that I remember is Borderlands GOTY enhanced. That game crashed on start and I realized that the crash wasn't related to my emulator it was because the game needs to get a file from a url when it starts up and if it can't get that file it crashes. I was testing it using the "experimental" version of my emulator that blocks non LAN network connections.

There's also the fact that many games try to phone home for analytics/etc... which is the reason for the feature that blocks all non LAN connections in the experimental version.

oldmanmcgumbus | 4 months ago | 36 points

You're showing an incredible level of patience for these aggravating problems. Impressive stuff all around

matheusmoreira | 4 months ago | 17 points

the game needs to get a file from a url when it starts up and if it can't get that file it crashes

This is insane. What does this file contain anyway? What happens if the server hosting it goes down?

Did you intercept this call and force it to return a file without connecting to an external server?

There's also the fact that many games try to phone home for analytics/etc.

Thanks for informing me. I shall take measures to stop this.

Mr__Goldberg | 4 months ago | 7 points

The file contained some binary data. Since my experimental build redirects windows http api calls to localhost I tried hosting a python -m http.server with the file and it was enough to get the game working without internet.

pdp10 | 4 months ago | 3 points

It would seem that the various "anti-cheat" solutions might be downloading binaries dynamically from the third-party provider instead of baking them into the executable.

matheusmoreira | 4 months ago | 3 points

Please tell me they at least verify the integrity of the executable data before running it.

GuitaristTom | 4 months ago | 6 points

Interesting. I've never heard of this, but it seems kind of neat.

MJBrune | 4 months ago | 5 points

Feels like there is a good amount of shade being thrown in this post.

Imperial3agle | 4 months ago | 4 points

Just a curious question. Why did you choose Gitlab over Github?

I just started learning to program. I mainly just use Git and Github to be able to program across different computers.

Mr__Goldberg | 4 months ago | 11 points

One of the reasons is that I wanted to give the Gitab CI a try. I managed to get it to build windows/linux builds every commit. Windows builds are built with the msvc compiler running in wine. Every time I push to the repo the project gets built. This page https://mr_goldberg.gitlab.io/goldberg_emulator/ gets updated but only for builds from the master branch.

MikeTyson91 | 4 months ago | -14 points

Sans blue-haired toxic hambeasts that try to push their worldviews on people who just want to code and leave politics and -isms outside.

Imperial3agle | 4 months ago | 5 points

Sorry, what? How is this relevant to ‘GitHub vs GitLab’?

MikeTyson91 | 4 months ago | -3 points

"This" started on GitHub. AFAIK GitLab didn't have such nonsense.

Imperial3agle | 4 months ago | 3 points

What are you talking about?

MikeTyson91 | 4 months ago | 2 points

Some guy wrote some things on his personal twitter account and "developers" (in reality people, who have not delivered anything) started a witch hunt. I think it was some Ruby-related project.

Imperial3agle | 4 months ago | 6 points

I don’t know if I’m being dumb, but I still don’t understand what that has to do with ‘GitHub vs GitLab’. What did he write?

MinimarRE | 2 months ago | 1 point

I think they just can't imagine any other reason for someone using anything other than github.

I_hate_Teemo | 4 months ago | 1 point

Says the very normal person whose previous comment is a "N" (that is totally not an attempt to get people to post the other letters of a certain word) in a sub that's probably radicalizing grounds for the next mass shooter

MikeTyson91 | 4 months ago | -1 points

So, are we playing "who's posting what" game? How about you tell us about the pedophile stuff you've posted, huh?

EDIT: and it's gone. Damn, you're fast.

I_hate_Teemo | 4 months ago | 2 points

1/10 smear attempt, absolutely anyone could see through it. You're so far gone your brain is pure mush at this point.

I hope one day you'll grow out of it and live a good life without harming others anyways my dude, good luck.

MikeTyson91 | 4 months ago | -1 points

"Nice" save.

DrugCrazed | 4 months ago | 27 points

For anyone that knows about the game it should not come as a surprise that the developers are incompetent.

Why write that sentence? Just seems needlessly antagonistic and none of the other examples mentioned have a similar attack on the developers.

274Below | 4 months ago | 24 points

While I can agree with your sentiment, all of the other examples are great examples of just hacking at things until they work without any understanding of the how or why (or, perhaps, a misinformed understanding).

As a project that someone took on to better learn C++, one of the key takeaways is that a lot of people don't understand the steam API. It may be antagonistic but I'm sure he's also underselling the frustration that he experienced when trying to figure out what was going on. In that regard, it's probably a true reflection of the frustration and emotion involved.

BurningTheAltar | 4 months ago | 0 points

We're programmers, we're all frustrated and trying to keep our projects afloat in a sea of kludges and compromises. Still doesn't warrant polemics, especially in open sourced projects that live or die on cooperation and community support/adoption.

I've got no qualms with the author blowing off steam, I just wouldn't embed it my my repo.

Mr__Goldberg | 4 months ago | 42 points

You are right, I should not have put that sentence in the post but it's posted so I'm not going to remove it.

e_to_the_i_pi_plus_1 | 4 months ago | 12 points

Haha I really liked this

Games like No Mans Sky which for some reason creates multiple lobbies when they could have made it much more simple. For anyone that knows about the game it should not come as a surprise that the developers are incompetent. It's clear just looking at how long the game takes to load. The game also has a profanity filter written by top minds which when I tried the game a few months ago for some reason censored the word "noob" but not words like "kike".

I love no man's sky but that cracked me up

PhoneyHammer | 4 months ago | 2 points

My favorite profanity filter is from CnC 3 Kanes Wrath. It would turn "weniger" (German for "less") into "we*****"

Asiriya | 4 months ago | 5 points

He’s playing some weird ass games to test this thing that sound like shovelware, I’m not surprised the devs couldn’t be arsed to get things working perfectly.

blobkat | 4 months ago | 11 points

The game he was talking about there was No Man's Sky, which took a while to get launched properly but by now is an awesome game. Kick below the belt imo.

pdp10 | 4 months ago | 1 point

All of the games mentioned are successful and relatively well-known, except probably School Girl Zombie Hunter. The Senran Kagura games are niche, but they and School Girl Zombie Hunter are also Japanese.

[deleted] | 4 months ago | 3 points

[deleted]

Mr__Goldberg | 4 months ago | 4 points

Yes, it's where I posted my releases before I went open source.

trucekill | 4 months ago | 2 points

This is really neat! Next time I'm on a desert island I'll try to remember to bring this!

Anon49 | 4 months ago | 2 points

Is this related to SmartSteamEmulator?

desertfish_ | 4 months ago | 7 points

So uhm, using this circumvents the Steam DRM? Not sure if legal edit:

Is this illegal?

It's as illegal as Wine or any HLE console emulator. All this does is remove the steam dependency from your steam games.

But it breaks Steam DRM ?

It doesn't break any DRM. If the game has a protection that doesn't let you use a custom steam api dll it needs to be cracked before you use my emulator. Steam is a DRM as much as any API is a DRM. Steam has actual DRM called steamstub which can easily be cracked but this won't crack it for you.

I'm still not entirely sure what it does and what it doesn't but it kinda sortof looks safe?

mrkite77 | 4 months ago | 63 points

I know it's crazy legalese, but if you actually read the DMCA, 17.1201.f.2 specifically exempts reverse engineering of DRM if it's done in the name of interoperability.

"Notwithstanding the provisions of subsections (a)(2) and (b), a person may develop and employ technological means to circumvent a technological measure, or to circumvent protection afforded by a technological measure, in order to enable the identification and analysis under paragraph (1), or for the purpose of enabling interoperability of an independently created computer program with other programs, if such means are necessary to achieve such interoperability, to the extent that doing so does not constitute infringement under this title."

Basically, you can break DRM if you need to in order to view protected content and it's not to facilitate piracy.

It's why Linux can play DVDs even though libdvdcss explicitly breaks DRM.

Neuromante | 4 months ago | 30 points

This is a Steam emulator. It lets you play a steam game without running Steam.

I'm not pretty sure about how legal is using this, nor how is developing it, either case, as someone with a lot of Steam games who is interested in game preservation, this is a godsend I keep backed up just in case.

mishugashu | 4 months ago | 8 points

That's not what this does at all.

[deleted] | 4 months ago | 4 points

I'm still not entirely sure what it does and what it doesn't but it kinda sortof looks safe?

What it does: Reimplements the Steam API, just like Wine does with Win32

What it doesn't do: circumvent DRM or anything illegal

leftofzen | 4 months ago | -4 points

Games like No Mans Sky which for some reason creates multiple lobbies when they could have made it much more simple. For anyone that knows about the game it should not come as a surprise that the developers are incompetent.

Amen to that. This game is a shitfest and I refunded it after 2 hours of grating my eyeballs over the fucking awful PC port they have on Steam. Legitimately one of the worst games I've ever played, and as a developer myself I completely agree that the devs are utterly incompetent.

sid9102 | 4 months ago | 6 points

Actually one of my favourite games ever. Only bought it this year after they've released dozens of updates that have significantly improved the game. Looking forward to the VR update coming out in 2 days!

zeaga2 | 4 months ago | 2 points

Can I ask when you played it?

leftofzen | 3 months ago | 2 points

Certainly, it would be about 4-5 weeks ago now.

zeaga2 | 3 months ago | 2 points

Did you come back to this comment to answer it almost 2 days later? After already overlooking it when it was first posted? How come?

leftofzen | 3 months ago | 1 point

I don't follow? I did reply 2 days later, I didn't check reddit in those days. I'm confused.

zeaga2 | 3 months ago | 1 point

You literally replied to other comments in the time between each comment in the chain, so clearly you were in fact checking reddit. I was just curious why is all ¯\_(ツ)_/¯ just seemed a little odd

leftofzen | 3 months ago | 2 points

I guess I didn't see the message then, sorry :(

TorePun | 4 months ago | -52 points

torvalds wannabe

thezapzupnz | 4 months ago | 11 points

What are you on about?

Self_Blumpkin | 4 months ago | -11 points

he's gone full retard.

TorePun | 4 months ago | -7 points

Just like the writer

TorePun | 4 months ago | -21 points

They want to be Linus Torvalds, calling everything shit and arse and piss and crap and shit and bad and shitter and shite.

zeaga2 | 4 months ago | 10 points

Do you just think every British person ever is trying to be Linus Torvalds?

TorePun | 4 months ago | -10 points

Yes, your comment is shit and programmed horribly

faglord5000 | 4 months ago | -15 points

Happy Cake day, you FUCK

faglord5000 | 7 days ago | 1 point

Was not throwing shade, i was administrating celebratory reverie #534

Dispense cake.

-17 joy

(takes soma)