Blog: Torchlight 3 – Data fixup war stories

The next weblog submit, until in a different way famous, used to be written by way of a member of Gamasutra’s neighborhood.
The ideas and reviews expressed are the ones of the creator and now not Gamasutra or its mum or dad corporate.


Right through the continued construction of Torchlight III, we would like gamers to get up-to-the-moment details about what is going on with the sport & dev crew. This month’s Developer Replace comes from Man Somberg, Lead Programmer of Echtra, with lend a hand from Jill Sullivan, Senior Neighborhood Supervisor.


In recreation construction, now not each drawback that you simply remedy is efficiency, options, or capability that the gamers get to look.  Every now and then, you will have one thing messy that must be constant, and also you simply wish to dive in and fasten it in order that paintings can get accomplished.

There’s a word that we use to speak about this kind of paintings: “yak shaving”.  Initially from a connection with the TV program “Ren and Stimpy”, it now refers to paintings that seems to be utterly unrelated to the tip objective, however which you need to accomplish so as to achieve it.  For instance – “I’m looking to construct a stone bridge over this creek.  I’m shaving this yak in order that I will be able to business the fur to a yarn maker, who in trade will let me borrow the cart in order that I will be able to take it to the quarry to pick out up some stones.”  Shaving the yak isn’t essentially a very powerful a part of construction that bridge, however you gained’t have the ability to make any development whilst the yak nonetheless has its coat!

It is a choice of simply one of the instances that we’ve had yak-shaving issues that had to be “simply constant”.


Supply keep an eye on techniques are probably the most basic equipment that recreation builders (and, in reality, just about all builders) use.  This can be a database containing all the historical past of each record that makes up our recreation – supply code, belongings, sounds, you title it.

About 3 years in the past, we switched our supply keep an eye on from one gadget to any other.  It doesn’t subject from what to what.  The gadget we have been the usage of used to be breaking down underneath our load, so we wanted a brand new one.  We did our homework, tested the choices, and made a decision.

Now, while you transition supply keep an eye on techniques, there are widely talking two techniques to head about it.  The easier manner is to fasten everyone out of supply keep an eye on, take a snapshot of the newest stuff, import it into the brand new gadget, tweak it to adapt to the brand new gadget’s thought of ways the universe will have to paintings, after which flip the brand new gadget on for other folks.  This has the benefit that it “simply works”, but it surely loses all the supply historical past from prior to the changeover.  In those environments there may be frequently a unmarried second in supply historical past that claims “Imported the whole lot.  If you wish to have historical past prior to this, cross glance within the different supply keep an eye on gadget.”  That’s superb, as long as the opposite gadget remains round otherwise you in a different way have get right of entry to to it, however frequently the commits from previous are misplaced eternally.

The extra complicated manner of doing that is to in reality import the historical past from the previous gadget into the brand new one.  Maximum supply keep an eye on techniques can help you do that, however it’s time-consuming, error inclined, and nonetheless calls for some guide intervention to adapt to the idiosyncrasies of the brand new gadget.  Even supposing it’s extra paintings up-front, it’s helpful after all to have all of your supply historical past to be had.

We opted for the historical past import, which – no less than at the floor – looked as if it would cross simply superb.  We noticed the historical past, we noticed the recordsdata, and we have been in a position to poke round and check that the whole lot regarded proper.  Some recordsdata had bizarre spacing problems, which didn’t appear to be that gigantic a deal.

However then we attempted to assemble, and all of it crumbled down.  The Visible Studio compiler complained about “Mac line endings” and refused to assemble anything else.

What?!  Why would that be?

A little of background right here: when a pc needs to constitute a personality, it has to make a choice an encoding.  The most typical encoding in use these days is known as UTF-Eight, which will encode all English-language characters, commonplace punctuation, or plenty of keep an eye on codes right into a unmarried byte of knowledge.  (The usage of more than one bytes, you’ll encode knowledge in as regards to any language, however that’s any other dialogue.)

Two of those keep an eye on codes are the Carriage Go back (CR) and the Line Feed (LF) characters, which hearken again to the times the place computer systems have been hooked as much as automatic typewriters slightly than displays.  In the ones days, you may inform the printer carriage to return to its house column by way of sending it a CR code, and you may have the paper roll to the following line by way of sending it an LF code.  Thus, when you sought after to begin typing originally of a brand new line, you may ship the series CR LF.

When the transfer to fancy graphical shows took place, this CR LF conference remained for backward compatibility.  On the other hand, builders of latest techniques – just like the sparkly new Apple Macintosh laptop and the Unix gadget at Bell Labs – weren’t hindered by way of backward compatibility and have been loose to make other possible choices.

It seems that the 3 maximum commonplace techniques on the earth these days all made other possible choices: DOS used CR LF, Macintosh used CR, and Unix used LF.  Home windows inherited its line endings from DOS, and MacOS now makes use of LF (the similar as Unix).

Over the years, the diversities looked after themselves out.  Device is usually in a position to function in “textual content mode” and give you the person with no matter line endings they want for his or her gadget to render it as it should be.  The main points of those variations leak via each so frequently, however generally aren’t a large deal.

All of this background flashed via our minds after we noticed the mistake about Mac line endings.  What used to be it speaking about?  We broaden on Home windows, so all the line endings will have to were Home windows (CR LF) line endings – or, on the very least a mixture of Home windows and Unix (LF) line endings.  The place have been those lone CR characters coming from?

After which we remembered the bizarre spacing problems – all of our supply code gave the impression to be double-spaced.  The place did the ones additional areas come from?

That is the place someone had the speculation to take a look at the record in a hex editor – a device which permits us to look the binary illustration of the textual content recordsdata by way of exhibiting every byte’s price in hexadecimal.  Ordinarily, on a record with Home windows line endings, you are expecting to look a line of textual content, then a CR (13, or 0D in hexadecimal illustration) and an LF (10, or 0A).  For some reason why, at the damaged traces, we noticed a CR (0D), then any other CR (0D), after which an LF (0A), giving us 0D0D0A.

Come what may, all over the conversion procedure from one supply keep an eye on program to any other, the conversion program made up our minds that the record had Unix line endings, then went via and did a blind seek/exchange each LF with CR LF, despite the fact that it already had a CR!  That defined the whole lot.  Our editor used to be completely satisfied to render the CR as a clean line, and knew the way to convert the CRLF right into a clean line, which used to be why our code gave the impression to be double-spaced.  Contrariwise, the Visible Studio compiler used to be satisfied to interpret the CR LF combo as a newline, however errored out at the previous CR.

I constant this by way of writing a bit of program in C++.  It might iterate over our supply code listing, open each textual content record, in finding patterns of 0D0D0A and exchange them with 0D0A.  We don’t be expecting to switch supply keep an eye on techniques once more, so the code for this software is misplaced to the sands of time.  (Ed. – Or, so we idea!  A force containing the supply for this program used to be came upon after this newsletter used to be written, so we now have uploaded the code to our repository for posterity.)

There have been most effective two or 3 people who labored in this explicit factor, however you’ll get any people to twitch a bit of simply by pronouncing “oh doh doa”.

Octothorpe Fixer

A few years in the past our sound dressmaker and our composer took a shuttle out to Bratislava, Slovakia to document a reside orchestra for a few of our track.  It used to be an important shuttle (so I’m informed), they usually were given so much accomplished over the few days that they have been there.

One of the vital outputs of this shuttle used to be a collection of content material that we name “vzory” – Slovakian for “trend”.  Those are small orchestral chunks of track that may be mixed in myriad techniques to create new track, and are recorded in more than a few combos of keys and notes.  The result is that we have got a specific trend in G, in G#, in F, in F#, and so on.

Our composer did the herbal factor – he spent a number of time chopping and organizing all of this content material, doled it out into folders and recordsdata matching the be aware that they have been recorded at, after which imported the entire suite into the audio software that we use, FMOD Studio.  FMOD is connected to our supply keep an eye on gadget, and it fortuitously added all the new recordsdata after which checked them in.

Thus far so just right.  Till other folks began to get mysterious warnings about filenames after they were given the newest code and knowledge via our supply keep an eye on gadget.  They have been simply warnings – they weren’t combating any one from running – but it surely used to be indubitably one thing that we didn’t need to stick round.

Those vzory tracks have been tracked down because the wrongdoer.  It seems that our supply keep an eye on gadget doesn’t find it irresistible when you test in recordsdata with an octothorpe (‘#’, also known as a pound signal, hash mark, hash tag, quantity signal, or more than a few different issues) within the filename.  It is going to settle for them, however bitch loudly.  It seems that our composer named the listing and matching recordsdata for the vzory tracks in the important thing of A pointy with the title “A#” – naturally!  (The opposite sharp keys have been arrange this manner as neatly.)

The supply keep an eye on gadget used to be maximum displeased with this selection.

Renaming the recordsdata wasn’t sufficient, as a result of FMOD helps to keep observe of record and listing metadata in XML recordsdata – every one with a GUID (a chain of letters, numbers, and dashes) because the filename.

As soon as once more, code to the rescue.  This time it used to be a program written in C# (sarcastically) that might iterate over all the recordsdata and subdirectories within the given trail, in finding ones with an octothorpe within the title, and rename them to switch the ‘#’ with the phrase ‘sharp’.  So, ‘A#’ was ‘Asharp’.  Then it could iterate over the XML recordsdata within the trail, in finding any that had an octothorpe within the record contents (which have been subsequently metadata in regards to the recordsdata or directories that have been renamed), and exchange the ‘#’ in that line with the phrase ‘sharp’.

Rather then telling our people “don’t do this”, there’s now not a lot that we needed to do to forestall this from going down once more.  This time we stored the supply code, so if we make that exact mistake once more then the software to mend it’s in a position handy.


Localization and internationalization are essential portions of any recreation undertaking.  We use the Unreal engine, which has a collection of localization equipment integrated.  Through the usage of a specific knowledge construction in our knowledge recordsdata, Unreal can in finding all the localized traces within the recreation.  We will be able to then export them right into a standardized layout known as a “transportable object” (.po) record, utilized by the GNU gettext equipment, amongst others.

It is a layout that our translators have equipment to care for.  They snatch the recordsdata, translate the traces, and ship them again.  We will be able to then import them to a specific locale after which Unreal will render the textual content.  All very neat, as long as you colour throughout the traces and practice the best way that Unreal expects you to paintings.

Naturally, we now have constructed a few of our personal stuff which lives inside Unreal’s techniques and performs properly with them, however is satisfactorily “off to the facet” that it’s invisible to a few of Unreal’s different techniques.  A type of portions is the localized string gadget, which didn’t see any of our fancy belongings.

We wrote a device that makes them visual, and known as it an afternoon.  Our first large batch of localization went out to the translators.  We went to import it…most effective to search out that none of our strings were given imported!

What came about?

Unreal lets you establish every localized string by way of a couple of textual content strings: a class and an access inside of that class.  For those who don’t supply both of the ones entries, it is going to generate them for you.  It seems that the software that we had written to make our belongings visual to the translators generated a brand new class and access for each localized textual content string each time it used to be run, which intended that each textual content line would get a unique code each time we ran the import or the export.

Oh, expensive.  We constant the underlying drawback and made the class/access pairs be constant throughout runs, however we had this large drop of strings in all the languages that used to be incompatible with the fixed-up knowledge!  We had to determine the way to run a one-time fixup on those strings to cause them to fit.

Thankfully, every string got here with a large number of metadata about its context and provenance.  A lot of this metadata didn’t exchange, or no less than modified in a predictable model.  This metadata grew to become out to be sufficient that lets examine an imported line and a newly-exported line and fit up the strings.

As prior to, writing some code used to be the solution right here.  We wrote a program (C++ once more) to learn within the translated record (containing the previous, fallacious class/access pairs) and a newly-exported English-language record (containing the brand new, right kind class/access pairs), fit up the metadata, after which write out a hard and fast model of the record containing the translated textual content with the corrected class/access pairs.

Here’s one scenario the place merely solving up the knowledge used to be inadequate.  We had to remedy the underlying drawback first prior to lets write the software to mend the knowledge.


Those issues all had a commonplace theme: via some series of occasions – human error or device error – a number of essential recordsdata seemed that have been all damaged by some means.  In the long run, to the people who find themselves running with the knowledge it doesn’t truly subject why any of these items came about.  They only need to take their damaged recordsdata and fasten them.  It’s all the time a profitable endeavour to determine a root motive and save you a subject matter from happening once more, however once in a while you simply wish to get to paintings.  The entire postmortem research and preventative paintings on the earth gained’t lend a hand other folks get their jobs accomplished with the damaged recordsdata that they have already got.

The examples I’ve mentioned right here have been all essential paintings that had to get accomplished, however all 3 of the ones systems were given run precisely as soon as.  It seems that lots of the techniques that we construct are complicated, and those varieties of problems crop up as a typical section or construction as we find one of the edge instances.

Every now and then, you want to write down equipment that you simply run precisely as soon as, and that’s not an issue – you simply wish to grit your tooth and shave that yak.

– Man Somberg

Leave a Reply

Your email address will not be published. Required fields are marked *