There is a specific kind of bad day that starts with a file full of é where every accented letter
used to be, or a spreadsheet whose names have all become question marks, or a database column that
looks correct in one application and like line noise in another. The word for the mangled result is
mojibake, and the cause is almost never corruption. The bytes are usually fine. What went wrong is
that somebody decoded them under the wrong assumption.
Understanding why requires separating two things that everyday language runs together: characters, which are what you mean, and bytes, which are what is stored.
A file does not know what encoding it is
This is the fact everything else follows from, and it is genuinely counter-intuitive.
A plain text file contains no field stating its encoding. It is a run of bytes. When a program opens it, that program must decide how to turn those bytes into characters, and it makes that decision from its own defaults, the operating system’s locale, a heuristic sniff of the content, or an out-of-band declaration such as an HTTP header or an XML prologue. Two programs can make different decisions about the same file, and both will happily show you the result of their decision without mentioning that a decision was made.
So “the file is broken” is usually the wrong diagnosis. The file is a sequence of bytes that means one thing under one interpretation and something else under another.
What Latin-1 does, and why it never fails loudly
Latin-1, formally ISO-8859-1 and closely related to the Windows-1252 codepage that most Western European systems actually used, is a single-byte encoding. Every character is exactly one byte, and there are 256 of them. That buys simplicity and costs coverage: you get the unaccented Latin alphabet, digits, punctuation, and a selection of accented letters covering Western European languages, and nothing else. No Greek, no Cyrillic, no Chinese, no emoji.
The critical property, the one that makes it dangerous rather than merely limited, is that a single-byte encoding cannot fail. Any byte at all maps to some character. Feed a Latin-1 decoder a file that was actually UTF-8 and it will not raise an error, because there is no byte sequence it considers invalid. It will simply produce the wrong characters, silently and confidently.
This is where the classic mojibake pattern comes from. In UTF-8, an accented letter is stored as two bytes. A Latin-1 decoder sees two bytes and produces two characters, so one letter becomes a pair of them, and the first is very often the same capital A with a tilde that everybody recognises from broken text. The pattern is so consistent that experienced readers can name the original encoding just by looking at the damage.
What UTF-8 does differently
UTF-8 is variable width. A character occupies one, two, three or four bytes depending on which character it is, and the design has two properties that matter enormously in practice.
The first is that the entire ASCII range, meaning the unaccented English letters, digits and common punctuation, is encoded as a single byte with exactly the same value it had in ASCII. That makes a pure-English UTF-8 file byte-identical to the same file in ASCII or in Latin-1, which is why encoding problems have a habit of staying hidden until the first accented name or curly quotation mark arrives. Every test passed because every test used English.
The second is that multi-byte sequences follow a strict pattern, and not every byte sequence is valid. A UTF-8 decoder can therefore detect that its input was not UTF-8, rather than producing plausible nonsense. That asymmetry is why “read as UTF-8 and see whether it complains” is a reasonable first diagnostic, while the reverse is not.
Where the boundary keeps catching people
The interesting failures are not in files but at the joins between systems, and three come up constantly.
Text on the web has to survive a URL, and a URL is not allowed to contain arbitrary characters. Percent-encoding is the mechanism: unsafe characters are replaced by a percent sign and their byte values in hexadecimal. Note the word bytes. Percent-encoding operates on bytes, not characters, so an accented letter has to be turned into bytes first, and the result differs depending on which encoding did that. This is why the same query string can work against one server and fail against another, and it is the layer our URL encoder and decoder makes visible: seeing the percent sequences is often enough to identify which encoding produced them.
Text passing through a channel that only carries printable characters hits the same boundary from the other side. Base64 exists to solve that, and it converts bytes, not characters. A naive implementation in a browser trips on exactly this: the platform’s built-in function treats each JavaScript string code unit as if it were a single byte, so any character outside the single-byte range either throws an error or silently mangles. Our own Base64 encoder converts the text to its real UTF-8 bytes first and encodes those, which is the difference between a tool that works on English and a tool that works on your actual data.
The third join is the spreadsheet. A CSV exported from a system in one locale and opened in a spreadsheet application configured for another is the single most common source of mojibake in ordinary office work, because neither end declares anything and both ends have opinions.
Case conversion is not as simple as it looks
Encoding is only half the story, because once you have the right characters you still have to operate on them, and character operations are language-dependent in ways that catch people out.
Uppercasing is the obvious example. In most languages the uppercase of the letter i is I, which is the rule almost every piece of software applies. In Turkish it is not: the dotted and dotless i are distinct letters with distinct uppercase forms, and applying the English rule produces a genuinely wrong word. German has a sharp s whose traditional uppercase form is two characters, so uppercasing can change a string’s length. These are not edge cases invented by specification authors; they are ordinary words in ordinary languages.
A case converter is therefore a text tool rather than a byte tool, and worth reaching for rather than reimplementing, particularly for the formats people actually want, such as title case, which has its own conventions about which small words stay lowercase.
Counting is language-dependent too
The same applies to counting, which sounds like it should be the least ambiguous operation available and is not.
“How many characters is this” has at least three defensible answers: how many bytes it occupies, how many code points it contains, and how many things a reader would call characters. They diverge the moment you leave plain English. An accented letter may be one code point or two, depending on whether it was composed or decomposed. A family emoji is a single visual glyph made of several code points joined together. A flag is two.
Our word and character counter counts by code point rather than by storage unit, which is the answer that matches intuition most of the time and is the one most social platforms use for their limits. The important thing is not which definition a tool picks but that it picks one deliberately and tells you, because a length limit enforced under one definition and measured under another is exactly how a form rejects text that looks perfectly short.
The practical rules
Use UTF-8 for everything you control, and declare it wherever a declaration is possible: the HTTP header, the meta tag, the database column and connection, the file export dialog. Most encoding problems are two systems that never agreed, and a declaration is the agreement.
When something does go wrong, resist editing the visible mess. Find the point where the wrong decoding happened and fix it there, because the mojibake is a symptom and the bytes upstream are usually still correct. Fixing the symptom means re-encoding the damage permanently, which turns a reversible problem into an irreversible one.
