This can get confusing, so from now on when I say encoding I’m talking about encoding code points to a byte stream. So we encode symbols to numbers, but then we also encode those numbers to byte streams. Although ASCII is encoded in bytes it only uses 7 bits for each character, and so only allows for 128 characters.Ĭonfusingly, the term encoding is also used in the more general form to refer to how numbers map to symbols. Similarly, we don’t have to decode it when reading back in to memory. With ASCII we can simply write out byte chars directly to a file without doing any encoding work. The simplest encoding is ASCII where each code point maps to a single byte.
There is a push for C++ stl to treat all strings as encoded. For example, the C string functions assume strings are arrays of code points (no encoding). Whether an array of bytes in memory is an encoding is determined by how it is treated. Just because 2 bytes may be used to store each code point doesn’t mean that it is an encoding. It’s important to distinguish the difference between a text file encoding and how each code point is stored in memory. A text encoding is basically a file format for text files. To read it back in we have to know how it was encoded and decode it back into memory. An encoding is typically used when writing text to a file.
EncodingĪn encoding converts a sequence of code points to a sequence of bytes. Obviously if less than 4 bytes are used not all code points can be represented. A code point can be stored in 1 to 4 bytes in memory. For programmers we could normally think of them as characters, but a code point is the more general term and doesn’t imply any information about how it is stored on a computer. There are many standards for text encodings, some are misnamed (ANSI format), some have very few standards (text file formats) and some things are just plain wrong, but have been wrong for so long that they have been incorporated into standards.Ī code point is the terminology used for a symbol that is represented as a number. Here I want to explain the relationship between text encoding, unicode, wide chars and text file formats. It’s not until you start supporting multiple languages, or come across a file in a weird format that you have to understand how text encodings really work. Unfortunately it’s not always that simple. Most of the time text is just a sequence of chars, chars are just numbers that map to letters, numbers and symbols, and to save text to a file you just write out a sequence of chars, or let whatever API you are using handle the encode/decode. Many programmers get by without having to worry about text encodings at all.
0 kommentar(er)
