y

So I decided to try NodeJS and Heroku.

Heroku is a service that hosts your applications. NodeJS is a… well, you know what it is, right? You create apps using javascript, that’s what it is.

There’s a lot of hype around these two, so I was curious to try them; I also come from a very different environment (desktop apps written in C#) so I expected nothing – other than my own incompetence :) I knew I needed to learn this from scratch basically, so I tried to enter this world with no preconceptions. So, these are my subjective observations coming from a standalone developer perspective.

The project was simple: an online highscore-storing service for a Unity game. The Unity part is important: I wanted to access the service using C# code, which meant I needed an encryption routine that worked for both. I thought this was going to be the most difficult part, but I was wrong.

Let’s start with Heroku. I found it to be a very friendly service, it has a nice onboarding/tutorial page which guides you through the setup phase and runs a basic app. It works locally and on the web, too.

I’ve run into a minor problem; Heroku thought I had two projects in the local folder, despite having only one. I have no idea how did I end up there, but I eventually solved it by deleting a rogue git alias. It was a risky move because it’s not clear what aliases does Heroku expect; now I have one remote called “heroku” and it works.

Other that that, it’s smooth sailing. You have a git repo, you have a Heroku account, Heroku runs whatever is in you repo – that’s it.

Aside: out of curiosity, I asked some of my more experienced friends how much time would this project take; the answers ranged from “an hour” to “half a day”. I think that’s reasonable, as I only need two endpoints, a very simple database and some encryption. I was prepared to spend a few days on it to get the hang of things.

So, onto NodeJS! The default Heroku sample uses a framework called “Express”. I had no idea what it did. It’s an “application framework”, which is meaningless, and their website isn’t very helpful either. After some fumbling around it turns out it has some handy tools for serving HTTP websites, so I decided to stick with it. I asked around and it’s one of the most popular webserver frameworks, so there’s probably no harm in using that instead of writing a lot of boilerplate code. It still strikes me odd that its functionality isn’t really explained on their site.

Handling a request is very simple; it’s similar to writing to a console:

app.get('/', function(req, res) {
  res.send("<html><body><h1>ayy</h1></body></html>");
});

I guess this is the part when I would have to look up a template engine to serve nice HTML content, but for this project I just need to pass data, so I’m skipping that. Anyway, this ran flawlessly on Heroku, and seeing my own little NodeJS webserver running live is very promising! Note that I’m about two hours in, including installing Heroku and NodeJS. So far, so good, I’m progressing quite fast.

Next, I’ll need a database to work with some “real” data. The local pgsql install process takes some time (around an hour), and syncing it with Heroku is supposed to be simple. However, I screwed this up, someehow lost my root password, I had to reinstall, and after that Heroku wasn’t so cooperative. I guess an environment variable got stuck somewhere; I remember seeing it on a stackoverflow answer? It’s not clear what the problem is and how should I go about it. I have no doubt that it would work nicely if I don’t fuck it up, but I did, and it’s very hard to find out how to make it work again, because I have little idea what’s going on.

Eventually, I solve this too, but it takes a considerable amount of googling, and the solution turned out to be an unofficial post somewhere.

Once working, Heroku can connect to a db seamlessly – the same code and config works locally and in the cloud. (Well, not for me, but you get the idea.)

I had to look up the await and async keywords for javascript; these make it possible to run – yeah right – asynchronous code. I got a bit confused with async though. First, I forgot to declare my anonymous function as async, so I got a runtime error. This feels very awkward compared to compiled languages, but that was sort of expected. I wonder though if I could do a static analysis to catch these errors while deploying? This seems to be a major difference when developing web apps.

Of course, I wanted to catch errors too. I looked into best practices for this, and I found conflicting opinions. Some say that try/catch doesn’t handle exceptions coming from async code, some say they do. There’s no definitive list of exceptions a function can throw; developers are advised to always use Error when throwing, as it is the only exception that’s widely supported. There are error events for Express. There are “Promises” with .catch() clauses…

The most striking statement is that after an error there is no way of ensuring the correctness of the application state, so you are to “gracefully restart” the application. This is unthinkable in the standalone world, but I guess it has something to do with a stateless environment and a large number of concurrent connections? Either way, there is a scary lack of documentation regarding this, and it’s especially weird to work in an environment where you can never guarantee the correctness of the application. To be honest, it all seems to be a cop-out: nobody knows what’s happening, so you just restart the app.

I’d like to emphasise that this is my subjective observation as a Node-rookie. Maybe I just haven’t found the appropriate docs, or I haven’t spent enough time looking?

Perhaps an experienced developer could tell me what the best option is, but I’d expect this topic to be well-covered, even for – especially for – beginner developers. The presence of “opinions” just highlight the lack of an official knowledge base.

Getting data

Ending this rant, after a few hours I settled on using try/catch-ing the database code. The result, I have to admit, is super-clean, simple, and well-readable:

app.get('/getscores', async function(req, res) {
  try
  {
    const client = await pool.connect();
    const result = await client.query('SELECT * FROM scores ORDER BY score DESC LIMIT 20');

    res.write(JSON.stringify(result.rows));
    res.end();

    client.release();
  }
  catch (err){
    console.error(err);
    res.status(500).send('Error: ' + err);
  }
});

Almost trivial. The “getscores” subpage returns a JSON containing the first 20 scores. I can’t shake the feeling that this can crash for unknown reasons (why isn’t client.release() in a finally block? I don’t know), but let’s hope for the best. I’ve inserted some test scores into the database, and it works! Way simpler than I thought it would be.

Uploading data

As simple as the project is, I wanted to make score uploading somewhat secure. I have some experience with encryption, so I didn’t have to learn this from scratch, and there is a nice library called “CryptoJS” that does the heavy lifting for me. A wonderful stackoverflow answer by user usselite basically solves the whole problem: encrypt some data using C#, decrypt it using javascript. Read it here: https://stackoverflow.com/questions/47891104/compatible-aes-encryption-and-decryption-for-c-sharp-and-javascript

Implementing this was surprisingly easy. I had to make some modifications to work with NodeJS (this included looking up what the bafflingly-named “atob” function does and replacing it using Buffer objects), but in less than an hour, I made a console app in C# that generated some encrypted data, a debug page to post my data, and the real endpoint that receives the encrypted payload, decrypts it, and writes it to console.

Now, here’s the catch.

The payload decrypting was easy; inserting it into the database is easy. Since the payload is encrypted, it’s hard to tamper with, but I still wanted to validate the data before doing any database operations.

So I got stuck on trying to verify if the score is a valid integer. Amazingly, this took hours.

Let’s start with the solution, the one I settled with at least:

if (!Number.isInteger(payload.map) || (payload.map <= 0))
{
  console.log('map invalid');
  res.status(400).end('error');
}

But getting there? That was surreal. The requirement is simple: the payload contains this data, and it should be a natural number (some values, like the level index, can be zero). There were all sorts of complications: how to check if the JSON (converted to an object after decryption) has a member? Converting it to bool doesn’t work if the number is 0. There’s a function for that; if we have the member, how to check if it’s a number? Check its type? Which equality operator do you use for that? What’s a number anyway? (Seriously, look that up.)

There are dozens of ways to go around this, each of them slightly different, there are hidden conversion going on in the background with all sorts of results (number, string, undefined, NaN…). And there is not one that is definitive.

This is baffling. Should I use a validation library? Seems overkill for such a simple task. It seems like I need more experience to answer this question? That this is not a trivial problem?

Conclusions

I’d like to remind you again that I spent about 8 hours on this project, so I’m definitively not an expert, and please consider this as first-look impressions.

Heroku is a very nice service. It just works, it’s free, and does a lot of things for you so you can concentrate on development. It would be very interesting to see how it scales upwards, distributing work between multiple dynos (a VM, sort of), but… I’m not going to do that :) It is amazingly friendly – you don’t even have to remember git credentials?! That’s the most streamlined service I ever saw.

Developing stuff in NodeJS is very fast. You can get your online thing running in a day, which is a lot faster tha I expected. If you need something, there’s a good chance that there is an npm module for that, so you’re just a command line entry away from implementing your high-level vision.

It is very hard, however, to write correct code. Debugging is very restricted, so you better log everything. I presume it also drastically increases the number of unit tests you need to write. Error handling is very weird – it seems like nobody knows how it works, and nobody cares. This means that even if you want to write correct code, chances are some of your dependencies will behave unpredictably.

Documentation is, well, dare I say, non-existent? There is no single source of truth. There are, however, conflicting “opinions”. Most of your time will be spent on Stackoverflow and various personal blogs on Medium, with no apparent way to verify what you read (other than trying it). It’s not unlikely that a developer will recommend a solution because they “like it the most”.

Suprisingly, nobody thinks of this as a concern.

I find this worrying, and I wouldn’t recommend NodeJS for anything other than really small, insignificant projects. Maintaining even medium size projects would be a nightmare; my project is about 200 lines of code, and I have no idea if it’s correct or not, nor have I any means of checking it. Does Express throw an exception somewhere? Or CryptoJS? Is it enough to put my database code in try/catch? Will it even catch errors in async functions? The only way to find out is to try it – which would take quite some time in itself.

What happens when an exception is thrown somewhere I didn’t expect? Okay, I know this one – the whole app will crash. How do I avoid that? I have no clue. It is telling that there are a fair number of tools that will automatically restart your app if it crashes.