Print Unicode, UTF-8

[SMcNeill]

A couple of quick routines which I worked up to print unicode for us, and to convert UTF-8 codepages to unicode format, so that they'll print properly for us. 

Code: QB64: [Select]

1. SCREEN _NEWIMAGE(800, 600, 32)
2. f = _LOADFONT("cyberbit.ttf", 16, "monospace")
3. _FONT f
4.  
5.  
6. FOR j = 0 TO 65000 STEP 1000
7.     CLS
8.     COLOR _RGB32(255, 255, 0)
9.     PRINT "UNICODE Values"; j; " to"; j + 999
10.     COLOR -1
11.     FOR i = 0 TO 999
12.         PrintUniCode i + j
13.     NEXT
14.     SLEEP
15. NEXT
16.  
17.  
18. CLS
19.  
20. OPEN "Petr Text.txt" FOR BINARY AS #1
21. DIM a(3) AS _UNSIGNED _BYTE
22. DIM i AS _UNSIGNED INTEGER
23. DIM b AS _UNSIGNED _BYTE
24.  
25. b = 1
26. DO UNTIL EOF(1)
27.     p = p + b
28.     FOR j = 0 TO 3
29.         GET #1, p+j, a(j)
30.     NEXT
31.     UTF8 = ConvertUTF8toUniCode(a(), b)
32.     PrintUniCode UTF8
33.  
34.     SLEEP
35. LOOP
36.  
37.  
38. FUNCTION ConvertUTF8toUniCode (a() AS _UNSIGNED _BYTE, b AS _UNSIGNED _BYTE)
39.     'b tells us how many bytes we used in this conversion,
40.     'so we know how far to move a pointer when reading from a file or UTF-8 encoded string
41.     SELECT CASE a(0) 'first byte is the control byte which tells us how many bytes we need to make our UTF-8 character
42.         CASE IS < 194: ConvertUTF8toUniCode = a(0): b = 1 'one byte character
43.         CASE 194 TO 223: ConvertUTF8toUniCode = (64 * (a(0) - 194) + a(1)): b = 2 'two byte character
44.         CASE 224 TO 239: b = 3: '3byte UTF-8 symbol  (I haven't sorted out the conversion for these yet)
45.         CASE 240 TO 255: b = 4 ' 4byte UFT-8 symbol
46.     END SELECT
47. END FUNCTION
48.  
49. SUB PrintUniCode (code AS LONG)
50.     IF code < 1 OR code > 65535 THEN EXIT SUB
51.     _MAPUNICODE code TO 0
52.     PRINT CHR$(0);
53.     _MAPUNICODE 0 TO 0
54. END SUB

Note: This uses some text that Petr shared on the forums here, in the past, so I hope he won't mind me using it for testing purposes. 

I haven't quite sorted out the work needed to convert 3 and 4 byte UTF-8 values to Unicode, but just being able to read and convert 1 and 2-byte characters will cover 99% of all documents which the general public would use normally.  It covers all the general Latin pages, and seems to have all the general symbols used in text.

I'm thinking 3-character UTF-8 characters are Japanese Kanji/Chinese characters, and such, and I have no idea what the 4-character UTF-8 characters represent.  Maybe some of the Emoji, or scientific symbols/notations?  I honestly don't know, as I personally tend to type exclusively in pure old unaccented American English, and the old 7-bit ANSI codes are usually enough for my needs...

Still though, I've been breaking my brain working with the keyhit remapper, so I thought I'd swap gears and take a break for a small bit...

So I swap over to trying to decode Unicode, UTF-8, and ANSI/ASCII encodings instead.  LOL!!


Anywho...  If you folks who tend to speak and write in those languages with these extended code sets could test things out, I'd appreciate it.  As far as I can tell, it works with what I've tested it with so far, but my eyes are old, tired, and I don't know the languages good enough to tell an A-forward slash from an A-backward slash from an A-circle over top...   Hell, I don't even know what the heck you actually call those accents...

I hope it works as it should, but if not, just let me know what's wrong with it, and I'll try and sort it out, as long as somebody is willing to be patient and work with me and tell me exactly what the heck is off with the conversion processes so I can fix them.  ;)

[Petr]

Hi Steve,
it does not work properly. If I remember it well, the point there on thread, was a move of characters over 128, which contain non-English diacritics back down to letters compatible with English, ie characters up to 127? To do this, you would need a function that returns the local national lunguage code number  LCID  (Powershell and get culture can be used, or a similar command for Linux) and then use DATA to move high characters from an ascii table down to English characters. Accents will not be displayed correctly, but the text will be readable. I think this was the point?

I enclose the output of your program and also my ascii table, as is set by default for my language.

 

 

[Petr]

I can work on this program tomorrow with you, because it's almost midnight here and I'm going to work tomorrow. I must go sleeping.

[SMcNeill]

Hi Steve,
it does not work properly.…

It’s not using the ASCII table, or character pages for us.  Instead, we should be reading the UTF-8 directly and then converting and using the unicode characters in the cyberbit.ttf which comes with QB64. 

Does the unicode characters display properly at the start, or is it something wrong with them to begin with, which isn’t working for you?

[SMcNeill]

You're right Petr; it's not working properly there.

It's odd, because I get the correct outputs here:

Code: QB64: [Select]

1. SCREEN _NEWIMAGE(800, 600, 32)
2. f = _LOADFONT("cyberbit.ttf", 16, "monospace")
3. _FONT f
4.  
5.  
6. GOTO skip_test
7. FOR j = 0 TO 65000 STEP 1000
8.     CLS
9.     COLOR _RGB32(255, 255, 0)
10.     PRINT "UNICODE Values"; j; " to"; j + 999
11.     COLOR -1
12.     FOR i = 0 TO 999
13.         PrintUniCode i + j
14.     NEXT
15.     SLEEP
16. NEXT
17.  
18. skip_test:
19. CLS
20.  
21. OPEN "Petr Text.txt" FOR BINARY AS #1
22. DIM a(1) AS _UNSIGNED _BYTE
23. DIM i AS _UNSIGNED INTEGER
24.  
25. DO UNTIL EOF(1)
26.     p = p + 1
27.     GET #1, p, a(0)
28.     IF a(0) > 193 THEN
29.         GET #1, p + 1, a(1)
30.         p = p + 1
31.         '       i = a(0) * 256 + a(1)
32.         SELECT CASE a(0) '194 to 223
33.             CASE IS < 194: PrintUniCode i
34.             CASE 194 TO 223: PrintUniCode (a(1) + 64 * (a(0) - 194))
35.             CASE ELSE: PRINT "U:"; a(0), a(1) 'it's a 3 or 4 byte unicode
36.         END SELECT
37.     ELSE
38.         PrintUniCode a(0)
39.     END IF
40.     SLEEP
41. LOOP
42.  
43.  
44. SUB PrintUniCode (code AS LONG)
45.     IF code < 1 OR code > 65535 THEN EXIT SUB
46.     _MAPUNICODE code TO 0
47.     PRINT CHR$(0);
48.     _MAPUNICODE 0 TO 0
49. END SUB

So apparently, when I converted it to a Function, rather than a direct method (as above), I glitched something out.  The issue seems minor enough -- I just need to figure out why the function is giving me different results than the direct version is.  I'm on it!  :P

[SMcNeill]

Scratch that.  It's working exactly as it should.  I simply wasn't sending it the proper sequence of bytes to convert.

The original code:

Code: QB64: [Select]

1. DO UNTIL EOF(1)
2.     p = p + b
3.     FOR j = 0 TO 3
4.         GET #1, p, a(j)
5.     NEXT
6.  

The glitch?

        GET #1, p + j, a(j)

We need to get up to 4 bytes and send them to the convertor -- not send it the SAME byte 4 times in a row! 

When you get a chance again later, give it a try one more time and see how it does for you.  I've updated the code in the original post, and I think it's got the issue corrected properly for us.  :)

[luke]

Here's a UTF-8 decoder I had in my back pocket that does all the characters:

Code: [Select]

DEFLNG A-Z
REDIM SHARED __utf8d(0) AS _BYTE

__utf8dl:
DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
DATA 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9
DATA 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7
DATA 8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
DATA &Ha,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H4,&H3,&H3
DATA &Hb,&H6,&H6,&H6,&H5,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8
DATA &H0,&H1,&H2,&H3,&H5,&H8,&H7,&H1,&H1,&H1,&H4,&H6,&H1,&H1,&H1,&H1
DATA 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,0,1,1,1,1,1,1
DATA 1,2,1,1,1,1,1,2,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1
DATA 1,2,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,3,1,3,1,1,1,1,1,1
DATA 1,3,1,1,1,1,1,3,1,3,1,1,1,1,1,1,1,3,1,1,1,1,1,1,1,1,1,1,1,1,1,1

SUB utf8print (in$)
    IF UBOUND(__utf8d) = 0 THEN
        REDIM __utf8d(0 TO 399) AS _BYTE
        RESTORE __utf8dl
        FOR i& = 0 TO 399
            READ __utf8d(i&)
        NEXT i&
    END IF
    o$ = ""
    FOR i& = 1 TO LEN(in$)
        c~%% = ASC(in$, i&)
        typ~%% = __utf8d(c~%%)
        IF s~%% <> 0 THEN
            cp = (c~%% AND &H3F) OR (cp * 64)
        ELSE
            cp = (255 \ (2 ^ typ~%%)) AND c~%%
        END IF
        s~%% = __utf8d(256 + s~%% * 16 + typ~%%)

        IF s~%% = 0 THEN
            _MAPUNICODE cp& TO 0
            PRINT CHR$(0);
        ELSEIF s~%% = 1 THEN
            ERROR 2
        END IF
    NEXT i&
    _MAPUNICODE 0 TO 0
END SUB

FUNCTION fromhex$ (in$)
    IF LEN(in$) MOD 2 THEN ERROR 2
    o$ = SPACE$(LEN(in$) / 2)
    FOR i& = 1 TO LEN(in$) STEP 2
        c$ = MID$(in$, i&, 2)
        MID$(o$, (i& + 1) / 2) = _MK$(_BYTE, VAL("&H" + c$))
    NEXT i&
    fromhex$ = o$
END FUNCTION
If you do something like

Code: [Select]

utf8print fromhex$("D09220D187D0B0D189D0B0D18520D18ED0B3D0B020D0B6D0B8D0BB20D0B1D18B20D186D0B8D182D180D183D1813F20D094D0B02C20D0BDD0BE20D184D0B0D0BBD18CD188D0B8D0B2D18BD0B920D18DD0BAD0B7D0B5D0BCD0BFD0BBD18FD18021") it works nicely.

Going up higher you have something like

Code: [Select]

utf8print fromhex$("E0A490") which does work but you'll need a different font because cyberbit doesn't support it.

Unfortunately you can't do

Code: [Select]

utf8print fromhex$("F0908080") because _MAPUNICODE is crappy and doesn't support code points above U+10000.

[SMcNeill]

Nice to see Luke is just as human when it comes to typos!  :)

There's a problem in the line here:             _MAPUNICODE cp& TO 0

Remove that &, and keep your variable type consistent inside that sub.  The only reason why it's working at the moment, is because of the DEFLNG A-Z at the top of code keeping the types matching for the program.  :)

[luke]

Eugh, it was originally just a decoder function and I added that bit to make it a printer.

Maybe instead of a typo we'll call it emphasis? :)

[Petr]

Hi Steve! I test it again using your upgraded first source code and now it works correctly, just font is clipped down, but it doesn't concern this program but it's a known font bug.

Good job!

 

[SMcNeill]

Sorted out my own little decoder, which ends up giving the exact same results as Luke's, from what initial testing I've tried it with.

Code: QB64: [Select]

1. REDIM SHARED __utf8d(0) AS _BYTE
2. DIM a(3) AS _UNSIGNED _BYTE
3. DIM i AS _UNSIGNED INTEGER
4. DIM b AS _UNSIGNED _BYTE
5.  
6. SCREEN _NEWIMAGE(800, 600, 32)
7. f = _LOADFONT("cyberbit.ttf", 16, "monospace")
8. _FONT f
9.  
10. FOR j = 0 TO 65000 STEP 1000
11.     CLS
12.     COLOR _RGB32(255, 255, 0)
13.     PRINT "UNICODE Values"; j; " to"; j + 999
14.     COLOR -1
15.     FOR i = 0 TO 999
16.         PrintUniCode i + j
17.     NEXT
18.     SLEEP
19. NEXT
20.  
21.  
22. CLS
23.  
24. OPEN "Petr Text.txt" FOR BINARY AS #1
25.  
26. b = 1
27. DO UNTIL EOF(1)
28.     p = p + b
29.     FOR j = 0 TO 3
30.         GET #1, p + j, a(j)
31.     NEXT
32.     UTF8 = ConvertUTF8toUniCode(a(), b)
33.     PrintUniCode UTF8
34.  
35.     SLEEP
36. LOOP
37. CLOSE
38.  
39.  
40. Luke$ = fromhex$("D09220D187D0B0D189D0B0D18520D18ED0B3D0B020D0B6D0B8D0BB20D0B1D18B20D186D0B8D182D180D183D1813F20D094D0B02C20D0BDD0BE20D184D0B0D0BBD18CD188D0B8D0B2D18BD0B920D18DD0BAD0B7D0B5D0BCD0BFD0BBD18FD18021")
41.  
42. CLS
43.  
44.  
45. b = 1: p = 0: l = LEN(Luke$)
46. DO UNTIL p >= LEN(Luke$)
47.     p = p + b
48.     FOR j = 0 TO 3
49.         IF p + j > l THEN a(j) = 0 ELSE a(j) = ASC(Luke$, p + j)
50.     NEXT
51.     UTF8 = ConvertUTF8toUniCode(a(), b)
52.     PrintUniCode UTF8
53. LOOP
54. CLOSE
55. PRINT
56. PRINT
57. PRINT "Luke's Translation:"
58. PRINT
59. utf8print Luke$
60.  
61. _DELAY 3
62.  
63. __utf8dl:
64. DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
65. DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
66. DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
67. DATA 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
68. DATA 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9
69. DATA 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7
70. DATA 8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
71. DATA &Ha,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H3,&H4,&H3,&H3
72. DATA &Hb,&H6,&H6,&H6,&H5,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8,&H8
73. DATA &H0,&H1,&H2,&H3,&H5,&H8,&H7,&H1,&H1,&H1,&H4,&H6,&H1,&H1,&H1,&H1
74. DATA 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,0,1,1,1,1,1,1
75. DATA 1,2,1,1,1,1,1,2,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1
76. DATA 1,2,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,3,1,3,1,1,1,1,1,1
77. DATA 1,3,1,1,1,1,1,3,1,3,1,1,1,1,1,1,1,3,1,1,1,1,1,1,1,1,1,1,1,1,1,1
78.  
79. SUB utf8print (in$)
80.     IF UBOUND(__utf8d) = 0 THEN
81.         REDIM __utf8d(0 TO 399) AS _BYTE
82.         RESTORE __utf8dl
83.         FOR i& = 0 TO 399
84.             READ __utf8d(i&)
85.         NEXT i&
86.     END IF
87.     o$ = ""
88.     FOR i& = 1 TO LEN(in$)
89.         c~%% = ASC(in$, i&)
90.         typ~%% = __utf8d(c~%%)
91.         IF s~%% <> 0 THEN
92.             cp = (c~%% AND &H3F) OR (cp * 64)
93.         ELSE
94.             cp = (255 \ (2 ^ typ~%%)) AND c~%%
95.         END IF
96.         s~%% = __utf8d(256 + s~%% * 16 + typ~%%)
97.  
98.         IF s~%% = 0 THEN
99.             _MAPUNICODE cp TO 0
100.             PRINT CHR$(0);
101.         ELSEIF s~%% = 1 THEN
102.             ERROR 2
103.         END IF
104.     NEXT i&
105.     _MAPUNICODE 0 TO 0
106. END SUB
107.  
108.  
109.  
110. FUNCTION fromhex$ (in$)
111.     IF LEN(in$) MOD 2 THEN ERROR 2
112.     o$ = SPACE$(LEN(in$) / 2)
113.     FOR i& = 1 TO LEN(in$) STEP 2
114.         c$ = MID$(in$, i&, 2)
115.         MID$(o$, (i& + 1) / 2) = _MK$(_BYTE, VAL("&H" + c$))
116.     NEXT i&
117.     fromhex$ = o$
118. END FUNCTION
119.  
120.  
121.  
122.  
123.  
124.  
125.  
126. FUNCTION ConvertUTF8toUniCode& (a() AS _UNSIGNED _BYTE, b AS _UNSIGNED _BYTE)
127.     DIM first AS _UNSIGNED _BYTE, second AS _UNSIGNED _BYTE, third AS _UNSIGNED _BYTE, fourth AS _UNSIGNED _BYTE
128.     'b tells us how many bytes we used in this conversion,
129.     'so we know how far to move a pointer when reading from a file or UTF-8 encoded string
130.     SELECT CASE a(0) 'first byte is the control byte which tells us how many bytes we need to make our UTF-8 character
131.         CASE IS < 128
132.             ConvertUTF8toUniCode& = a(0)
133.             b = 1 'one byte character
134.         CASE 194 TO 223
135.             first = a(0) MOD 32
136.             second = a(1) MOD 64
137.             ConvertUTF8toUniCode& = 64 * first + second
138.             b = 2 'two byte character
139.         CASE 224 TO 239 '3byte UTF-8 symbol
140.             first = a(0) MOD 16
141.             second = a(1) MOD 64
142.             third = a(2) MOD 64
143.             ConvertUTF8toUniCode& = 4096 * first + 64 * second + third
144.             b = 3
145.         CASE 240 TO 255 ' 4byte UFT-8 symbol
146.             first = a(0) MOD 8
147.             second = a(1) MOD 64
148.             third = a(2) MOD 64
149.             fourth = a(3) MOD 64
150.             ConvertUTF8toUniCode& = CLNG(262144 * first + 4096 * second + 64 * third + fourth)
151.             b = 4
152.     END SELECT
153. END FUNCTION
154.  
155. SUB PrintUniCode (code AS LONG)
156.     IF code < 1 OR code > 65535 THEN EXIT SUB
157.     _MAPUNICODE code TO 0
158.     PRINT CHR$(0);
159.     _MAPUNICODE 0 TO 0
160. END SUB
161.  

My little decoder looks like this:

Code: QB64: [Select]

1. FUNCTION ConvertUTF8toUniCode& (a() AS _UNSIGNED _BYTE, b AS _UNSIGNED _BYTE)
2.     DIM first AS _UNSIGNED _BYTE, second AS _UNSIGNED _BYTE, third AS _UNSIGNED _BYTE, fourth AS _UNSIGNED _BYTE
3.     'b tells us how many bytes we used in this conversion,
4.     'so we know how far to move a pointer when reading from a file or UTF-8 encoded string
5.     SELECT CASE a(0) 'first byte is the control byte which tells us how many bytes we need to make our UTF-8 character
6.         CASE IS < 128
7.             ConvertUTF8toUniCode& = a(0)
8.             b = 1 'one byte character
9.         CASE 194 TO 223
10.             first = a(0) MOD 32
11.             second = a(1) MOD 64
12.             ConvertUTF8toUniCode& = 64 * first + second
13.             b = 2 'two byte character
14.         CASE 224 TO 239 '3byte UTF-8 symbol
15.             first = a(0) MOD 16
16.             second = a(1) MOD 64
17.             third = a(2) MOD 64
18.             ConvertUTF8toUniCode& = 4096 * first + 64 * second + third
19.             b = 3
20.         CASE 240 TO 255 ' 4byte UFT-8 symbol
21.             first = a(0) MOD 8
22.             second = a(1) MOD 64
23.             third = a(2) MOD 64
24.             fourth = a(3) MOD 64
25.             ConvertUTF8toUniCode& = CLNG(262144 * first + 4096 * second + 64 * third + fourth)
26.             b = 4
27.     END SELECT
28. END FUNCTION

No need for a whole string of DATA statements, which I found myself completely lost as to what they were actually doing, this routine only relies on a few simple MOD statements and basic math to get us to the same solution. 

You've got to love that there's always multiple ways to come to the same solution to a problem!  ;D

[Petr]

I do not have enough knowledge about the construction of UNICODE that I would ever think of such a solution.

[SMcNeill]

I do not have enough knowledge about the construction of UNICODE that I would ever think of such a solution.

Take a look at this little program and see if it doesn't come close to handling any conversion needs you might need:

Code: QB64: [Select]

1. REDIM SHARED __utf8d(0) AS _BYTE
2. DIM a(3) AS _UNSIGNED _BYTE
3. DIM i AS _UNSIGNED INTEGER
4. DIM b AS _UNSIGNED _BYTE
5.  
6. SCREEN _NEWIMAGE(800, 600, 32)
7. f = _LOADFONT("cyberbit.ttf", 16, "monospace")
8. _FONT f
9.  
10. _CONTROLCHR OFF
11. FOR j = 0 TO 255
12.     ASCIItoUTF8 j, a(), b
13.     PRINT j; ")";
14.     FOR i = 0 TO b - 1
15.         PRINT a(i);
16.     NEXT
17.     PRINT CHR$(j),
18.     PrintUniCode UTF8toUniCode(a(), b)
19.     PRINT
20.     SLEEP
21. NEXT
22.  
23. FUNCTION ASCIItoUnicode& (ASCII AS _UNSIGNED _BYTE)
24.     ASCIItoUnicode& = _MAPUNICODE(ASCII)
25. END FUNCTION
26.  
27. SUB ASCIItoUTF8 (ASCII AS _UNSIGNED _BYTE, a() AS _UNSIGNED _BYTE, b AS _UNSIGNED _BYTE)
28.     UnicodeToUTF8 _MAPUNICODE(ASCII), a(), b
29. END SUB
30.  
31. SUB UnicodeToUTF8 (Unicode AS LONG, a() AS _UNSIGNED _BYTE, b AS _UNSIGNED _BYTE) 'a() should be an array from 0 to 3 which we want to get the value back from.
32.     FOR i = 0 TO 3: a(i) = 0: NEXT 'reset all values to 0
33.     SELECT CASE Unicode
34.         CASE 0 TO 127
35.             a(0) = Unicode
36.             b = 1 'we return 1 byte
37.         CASE 128 TO 2047
38.             a(0) = 192 + Unicode \ 64
39.             a(1) = 128 + Unicode MOD 64
40.             b = 2 'we return 2 bytes
41.         CASE 2048 TO 65535
42.             a(2) = 128 + Unicode MOD 64
43.             r = Unicode \ 64
44.             a(1) = 128 + r MOD 64
45.             a(0) = 224 + r \ 64
46.             b = 3 'we return 3 bytes
47.         CASE 65536 TO 1114111
48.             a(3) = 128 + Unicode MOD 64
49.             r = Unicode \ 64
50.             a(2) = 128 + r MOD 64
51.             r = r \ 64
52.             a(1) = 128 + r MOD 64
53.             a(0) = 240 + r \ 64
54.             b = 4 'we return 4 bytes
55.         CASE ELSE
56.             PRINT "Invalid Value passed to UniCodeToUTF8 convertor!"; Unicode, HEX$(Unicode)
57.             ERROR 5
58.     END SELECT
59. END SUB
60.  
61. FUNCTION UTF8toUniCode& (a() AS _UNSIGNED _BYTE, b AS _UNSIGNED _BYTE)
62.     DIM first AS _UNSIGNED _BYTE, second AS _UNSIGNED _BYTE, third AS _UNSIGNED _BYTE, fourth AS _UNSIGNED _BYTE
63.     'b tells us how many bytes we used in this conversion,
64.     'so we know how far to move a pointer when reading from a file or UTF-8 encoded string
65.     SELECT CASE a(0) 'first byte is the control byte which tells us how many bytes we need to make our UTF-8 character
66.         CASE IS < 128
67.             UTF8toUniCode& = a(0)
68.             b = 1 'one byte character
69.         CASE 194 TO 223
70.             first = a(0) MOD 32
71.             second = a(1) MOD 64
72.             UTF8toUniCode& = 64 * first + second
73.             b = 2 'two byte character
74.         CASE 224 TO 239 '3byte UTF-8 symbol
75.             first = a(0) MOD 16
76.             second = a(1) MOD 64
77.             third = a(2) MOD 64
78.             UTF8toUniCode& = 4096 * first + 64 * second + third
79.             b = 3
80.         CASE 240 TO 255 ' 4byte UFT-8 symbol
81.             first = a(0) MOD 8
82.             second = a(1) MOD 64
83.             third = a(2) MOD 64
84.             fourth = a(3) MOD 64
85.             UTF8toUniCode& = CLNG(262144 * first + 4096 * second + 64 * third + fourth)
86.             b = 4
87.     END SELECT
88. END FUNCTION
89.  
90. SUB PrintUniCode (code AS LONG)
91.     IF code < 1 OR code > 65535 THEN EXIT SUB
92.     _MAPUNICODE code TO 0
93.     PRINT CHR$(0);
94.     _MAPUNICODE 0 TO 0
95. END SUB

If you notice, we have a loop from 0 to 255, which we use to get ASCII codes from: FOR j = 0 TO 255   

Then we convert ASCII to UTF8 with:     ASCIItoUTF8 j, a(), b

The next segment prints the UTF-8 codes for us:
    FOR i = 0 TO b - 1
        PRINT a(i);
    NEXT

Then, to prove that things are working as they should, we print the CHR$ character itself:
    PRINT CHR$(j), 

And, for comparison, we then print the unicode version from the UTF8-translation, just so we can compare and make certain that all of the pieces match and look identical for us: 
    PrintUniCode UTF8toUniCode(a(), b)   


I'm thinking there's enough conversion routines in the toolkit here now, that people should be able to quickly/easily convert back and forth between ASCII, UTF8, and Unicode, as their program needs.  :)

If I'm missing something, just let me know, but I think this, when inserted into a program with my key remapper, will allow me to create an easily international input/output system for use with my QB64 programs in the future -- with no real reliance with _MAPUNICODE and code pages being required in the future.
 

[Unma]

Just to join the club, I also have QB64 function that converts UTF-8 strings into UNICODE number.

Not knowing for Your efforts, I have done this as aid to (hopefully automated) translating among many languages.
Once again I did something needless :-)

It is written in such a way that after 8 years, even I can understand what I was doing today. And that is not always the case :-)
Well, it my be helpful to somebody unfamiliar with UNICODE standard.

Code: QB64: [Select]

1. '------------------------------
2. ' INPUT:  UTF-8 string
3. '------------------------------
4. ' OUTPUT: ERROR   UTF2UNICODE < 0
5. '         ---------------------
6. '         OK      UTF2UNICODE => 0 AND UTF2UNICODE =< &H10FFFF
7. '                 Recoginsed unicode character is removed from the begining of argument. This can be turned off (see at the bottom).
8. '------------------------------
9. FUNCTION UTF2UNICODE& (txt$)
10.     DIM chlen AS INTEGER
11.     DIM hb AS LONG
12.     DIM db AS LONG
13.     DIM result AS LONG
14.  
15.     IF LEN(txt$) = 0 THEN
16.         UTF2UNICODE& = -1 'Invalid argument
17.         EXIT FUNCTION
18.     END IF
19.     result = -2 'Unspecified error
20.     chlen = 1
21.     hb = ASC(txt$) 'head-byte only
22.  
23.     IF (hb AND &B10000000) = 0 THEN ' ? 0xxx xxxx  TRUE=byte is ASCII character
24.         result = hb
25.     ELSE
26.         IF (hb AND &B11100000) = &B11000000 THEN ' ? 110x xxxx  TRUE=byte is 1st of two bytes
27.             'head-byte + data-byte
28.             ' 110xxxxx   10yyyyyy
29.             '---------------------
30.             chlen = 2
31.             'result = (hb AND &B00011111) * &B01000000
32.             result = (hb AND &H1F) * &H40 '            head-byte  shifted left 6 places  result | 0000 0000  0000 0000  0000 0xxx  xx00 0000 |
33.             db = ASC(MID$(txt$, 2, 1)) '                     data-byte
34.             'result = result OR (db AND &B00111111)
35.             result = result OR (db AND &H3F) '              data-byte  copied                 result | 0000 0000  0000 0000  0000 0xxx  xxyy yyyy |
36.         ELSE
37.             IF (hb AND &B11110000) = &B11100000 THEN ' ? 1110 xxxx  TRUE=byte is 1st of 3 bytes
38.                 'head-byte + data-byte1 + data-byte2
39.                 ' 1110xxxx   10yyyyyy     10zzzzzz
40.                 '-----------------------------------
41.                 chlen = 3
42.                 'result = (hb AND &B00001111) * &B 0001 0000 0000 0000
43.                 result = (hb AND &HF) * &H1000 '               head-byte   shifted left 12 places  result | 0000 0000  0000 0000  xxxx 0000  0000 0000 |
44.                 db = ASC(MID$(txt$, 2, 1)) '                     data-byte1
45.                 result = result OR ((db AND &H3F) * &H40) '  data-byte1  shifted left 6 places   result | 0000 0000  0000 0000  xxxx yyyy  yy00 0000 |
46.                 db = ASC(MID$(txt$, 3, 1)) '                     data-byte2
47.                 result = result OR (db AND &H3F) '           data-byte2  copied                  result | 0000 0000  0000 0000  xxxx yyyy  yyzz zzzz |
48.             ELSE
49.                 IF (hb AND &B11111000) = &B11110000 THEN ' ? 1111 0xxx  TRUE=byte is 1st of 4
50.                     'head-byte + data-byte1 + data-byte2 + data-byte3
51.                     ' 11110xxx   10yyyyyy     10zzzzzz     10wwwwww
52.                     '------------------------------------------------
53.                     chlen = 4
54.                     'result = (hb AND &B00000111) * &B 0000 0100  0000 0000  0000 0000
55.                     result = (hb AND &H6) * &H400000 '             head-byte   shifted left 18 places  result | 0000 0000  000x xx00  0000 0000  0000 0000 |
56.                     db = ASC(MID$(txt$, 2, 1)) '                      data-byte1
57.                     result = result OR ((db AND &H3F) * &H1000) ' data-byte1  shifted left 12 places  result | 0000 0000  000x xxyy  yyyy 0000  0000 0000 |
58.                     db = ASC(MID$(txt$, 3, 1)) '                      data-byte2
59.                     result = result OR ((db AND &H3F) * &H40) '   data-byte2  shifted left 6 places   result | 0000 0000  000x xxyy  yyyy zzzz  zz00 0000 |
60.                     db = ASC(MID$(txt$, 4, 1)) '                      data-byte3
61.                     result = result OR (db AND &H3F) '           data-byte3  copied                  result | 0000 0000  000x xxyy  yyyy zzzz  zzww wwww |
62.                 ELSE
63.                     'Not a head-byte.
64.                     result = hb
65.                 END IF
66.             END IF
67.         END IF
68.     END IF
69.     IF chlen < LEN(txt$) THEN txt$ = MID$(txt$, chlen + 1) ELSE txt$ = "" ' By commenting this line, function will leave string-argument unchanged.
70.     UTF2UNICODE& = result
71. END FUNCTION
72.  

I intend to ad some more "error codes" (negative returns) at some later point. Right now I have to deal another issue.
It is obvious that there is a bad blood between QB64 IDE and my OS (Linux Mint).
By

[RhoSigma]

And a cross reference to a similar topic: https://qb64forum.alephc.xyz/index.php?topic=2248