• Hudell@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    0
    ·
    3 months ago

    The characters are all in your own pc. The text data is actually just numbers, referencing the index of each character in a reference table.

    Early on someone thought “let’s create a bunch of different reference tables and each country uses the one that is best for them so we don’t have to include every character in the world”.

    But that thinking has a critical problem: when you write some text that will only be read within the country, you don’t need to keep track of which table you used because everyone will be using the same. Soon you forget that there are other tables for other countries so when you do send an international text using your table as a reference, the person on the other side will be parsing it using their own table and the resulting text will be different. And sometimes when this mixup happens, the index referenced by the text in the other table may actually be some internal control character that is not meant for rendering.

    These days the problem is “mostly fixed” by the near-universal adoption of a single reference table that proposes including verything you may ever need (even a lot of emojis) - but this large table means that each character in a text may need more digits to represent the intended index so the total file size for the same text is larger than it would be with the non-universal table.