scribe/spec/classes
Edward Loveall 7d0bc37efd
Fix markup errors caused by UTF-16/8 differences
Medium uses UTF-16 character offsets (likely to make it easier to parse
in JavaScript) but Crystal uses UTF-8. Converting strings to UTF-16 to
do offset calculation then back to UFT-8 fixes some markup bugs.

---

Medium calculates markup offsets using UTF-16 encoding. Some characters
like Emoji are count as multiple bytes which affects those offsets. For
example in UTF-16 💸 is worth two bytes, but Crystal strings only count
it as one. This is a problem for markup generation because it can
offset the markup and even cause out-of-range errors.

Take the following example:

💸💸!

Imagine that `!` was bold but the emoji isn't. For Crystal, this starts
at char index 2, end at char index 3. Medium's markup will say markup
goes from character 4 to 5. In a 3 character string like this, trying
to access character range 4...5 is an error because 5 is already out of
bounds.

My theory is that this is meant to be compatible with JavaScript's
string length calculations, as Medium is primarily a platform built for
the web:

```js
"a".length // 1
"💸".length // 2
"👩‍❤️‍💋‍👩".length // 11
```

To get these same numbers in Crystal strings must be converted to
UTF-16:

```crystal
"a".to_utf16.size # 1
"💸".to_utf16.size # 2
"👩‍❤️‍💋‍👩".to_utf16.size # 11
```

The MarkupConverter now converts text into UFT-16 byte arrays on
initialization. Once it's figured out the range of bytes needed for
each piece of markup, it converts it back into UTF-8 strings.
2022-01-30 11:53:22 -05:00
..
embedded_converter_spec.cr Proxy GitHub gists with rate limiting 2022-01-23 15:05:46 -05:00
gist_scanner_spec.cr Proxy GitHub gists with rate limiting 2022-01-23 15:05:46 -05:00
gist_store_spec.cr Proxy GitHub gists with rate limiting 2022-01-23 15:05:46 -05:00
markup_converter_spec.cr Fix markup errors caused by UTF-16/8 differences 2022-01-30 11:53:22 -05:00
page_converter_spec.cr Improve title parsing 2021-10-03 18:14:46 -04:00
paragraph_converter_spec.cr Proxy GitHub gists with rate limiting 2022-01-23 15:05:46 -05:00
rate_limited_gist_store_spec.cr Proxy GitHub gists with rate limiting 2022-01-23 15:05:46 -05:00