Skip to content

Unicode: transcoding errors for \u0361 #150

Description

@moyogo

reported on Launchpad https://bugs.launchpad.net/openlibrary/+bug/1020006

There are a lot of transcoding errors from imported external data regarding \u0361 LIGATURE TIE/COMBINING DOUBLE INVERTED BREVE. Ligature tie is used in romanization of Cyrillic.

External data is often using encodings where U+0361 is split into U+FE20 and U+FE21:

  • i︠a︡ [i + U+FE20 + a + U+FE21] should be converted to i͡a [i + U+0361 + a].

Furthermore if there is a dot [U+307] above the ligature tie [U+0361], there should be a CGJ U+034F to prevent Unicode canonical re-ordering:

But the problem is external data often contains errors instead.

  1. U+FE20 has apparently been converted to U+0361 but U+FE21 is still there.

See this fix for example
http://openlibrary.org/books/OL727005M/Zvukovo%C4%AD_sostav_chechenskogo_literaturnogo_i%CD%A1azyka_i_voprosy_usovershenstvovanii%CD%A1a_alfavita_i_orfogr?m=diff&b=4

  1. U+FE20 and U+FE21 have not been converted to U+0361.
  2. U+FE20, U+FE21 and other diacritic have not been converted to U+0361 + CGJ + other diacritic
  3. U+0360 is used instead of U+0361

See http://openlibrary.org/works/OL1070139W which seems to have a bit of everything.

I think most of the errors can be fixed they seem to follow these patterns and they should match a specific set of transliteration digraphs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Affects: DataIssues that affect book/author metadata or user/account data. [managed]Lead: @mekarpelesIssues overseen by Mek (Staff: Program Lead) [managed]Module: ImportIssues related to the configuration or use of importbot and other bulk import systems. [managed]Priority: 3Issues that we can consider at our leisure. [managed]Theme: UnicodeIssues and pull requests related to Unicode charactersType: BugSomething isn't working. [managed]

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions