-
-
Notifications
You must be signed in to change notification settings - Fork 2k
Unicode: transcoding errors for \u0361 #150
Copy link
Copy link
Open
Labels
Affects: DataIssues that affect book/author metadata or user/account data. [managed]Issues that affect book/author metadata or user/account data. [managed]Lead: @mekarpelesIssues overseen by Mek (Staff: Program Lead) [managed]Issues overseen by Mek (Staff: Program Lead) [managed]Module: ImportIssues related to the configuration or use of importbot and other bulk import systems. [managed]Issues related to the configuration or use of importbot and other bulk import systems. [managed]Priority: 3Issues that we can consider at our leisure. [managed]Issues that we can consider at our leisure. [managed]Theme: UnicodeIssues and pull requests related to Unicode charactersIssues and pull requests related to Unicode charactersType: BugSomething isn't working. [managed]Something isn't working. [managed]
Description
Metadata
Metadata
Assignees
Labels
Affects: DataIssues that affect book/author metadata or user/account data. [managed]Issues that affect book/author metadata or user/account data. [managed]Lead: @mekarpelesIssues overseen by Mek (Staff: Program Lead) [managed]Issues overseen by Mek (Staff: Program Lead) [managed]Module: ImportIssues related to the configuration or use of importbot and other bulk import systems. [managed]Issues related to the configuration or use of importbot and other bulk import systems. [managed]Priority: 3Issues that we can consider at our leisure. [managed]Issues that we can consider at our leisure. [managed]Theme: UnicodeIssues and pull requests related to Unicode charactersIssues and pull requests related to Unicode charactersType: BugSomething isn't working. [managed]Something isn't working. [managed]
reported on Launchpad https://bugs.launchpad.net/openlibrary/+bug/1020006
There are a lot of transcoding errors from imported external data regarding \u0361 LIGATURE TIE/COMBINING DOUBLE INVERTED BREVE. Ligature tie is used in romanization of Cyrillic.
External data is often using encodings where U+0361 is split into U+FE20 and U+FE21:
Furthermore if there is a dot [U+307] above the ligature tie [U+0361], there should be a CGJ U+034F to prevent Unicode canonical re-ordering:
See the examples in section 7.9 of http://www.unicode.org/versions/Unicode6.1.0/ch07.pdf
But the problem is external data often contains errors instead.
See this fix for example
http://openlibrary.org/books/OL727005M/Zvukovo%C4%AD_sostav_chechenskogo_literaturnogo_i%CD%A1azyka_i_voprosy_usovershenstvovanii%CD%A1a_alfavita_i_orfogr?m=diff&b=4
See http://openlibrary.org/works/OL1070139W which seems to have a bit of everything.
I think most of the errors can be fixed they seem to follow these patterns and they should match a specific set of transliteration digraphs.