As one of the few terminal users of Arabic in this site (emphasis on the terminal part), I want to thank Behdad and others for harfbuzz and other great libraries. His contributions have been significant, and BiCon Arabeyes project was one of the first and only ways to show Arabic in the terminal. This was even before SIGWINCH was a signal you could use IIRC, so that I cannot even run it in tmux, screen, and other dynamically-sized terminal emulators without a massive, massive headache. But before that, there was not much of anything. And things have come a LONG, LONG way since then, even if there is plenty more road ahead.
I started using Linux almost ten years ago, and the winter for RTL languages were just starting to thaw.
Since I have come to using Arabic more often in the terminal, you would be suprised how often this causes problems as text rendering is done at many different layers and there is no harmony between them. Not only does it mean a special terminal emulator (mlterm for the win; esoteric but the only one that handles Arabic properly and handles Chinese, many Indic languages, and Hebrew unlike anyone else) but has all minaskinds of edge cases. Text rendering happens at so many layers and with different libraries, you have to make runtime changes to get certain programs to behave with each other. As a heavy Emacs user, Emacs will conflict with its own (default on) bidi implementation using m17n-lib (as mentioned in the article). So what happens? Arabic and other RTL languages go back to the original order, English-style, which is not what you want, as the operation happens twice since mlterm also uses this library (m17n-lib), but is not aware the bidi reordering happened before at the application layer. I finally got so sick of this my most important Emacs keyboard macro is to toggle off the Emacs internal bidi ordering on and off. Why you ask? Because despite mlterm and Emacs using m17n-lib on my computer, I need GUI emacs to use its own internal Bidi scheme, and the terminal run Emacs (emacsclient -nw) to run without out, and let mlterm pick up from there. I have tried other terminal emulators (someone wrote Perl scripts using, IIRC Text::FriBidi based one of the aforementioned library, GNU FriBidi) for urxvt to handle this), but it does not work as well: you get the correct ordering, but Arabic letters do not connect (this is really glyphs, not letters, but I digress). So if I do not toggle the Emacs macro I made, I cannot have a terminal that will show Arabic file names properly at the same time outside of Emacs, and a simple `ls` is all borked up.
GUI programs are not so much better. I run i3 with my own customizations, including heavy use of the wonderful dmenu utility for searching files indexed with Recoll, connection to my password manager, and running programs on the fly. Probably is dmenu specifically does not support Unicode at all, let alone RTL languages like Arabic, so you get a mess. I love suckless tools; and they avoided Unicode for a reaosn, but suckless, a very common group of power user tools ignoring Unicode for their own good and embracing their mantra of well manicured C code tells you to pity developers who do take the time and bear lots of whining about this crap being inconsistent and having crazy edge cases.
These edge cases will not stop me from using free software, because I had no idea how complicated Arabic rendering was until using Linux, and a lot of this crap flat out is not possible in Windows anyway (try listing Arabic file names in Command Prompt), and is still terrible when you get around to it working. Do we need a "GNOME3 of text rendering" as others quoted in the threads here? I am not sure we do. What we need is standardization of libraries or generalizing backends to choose what you need, a la Freedesktop initiatives in other spheres. Ironically: every terminal emulator and app developer/maintainer retorts when present RTL Arabic problems: this is an application layer concern and out of my hands. And what they ignore app developers with their application layer concerns apply with different layers of libraries unevenly, hence the aforementioned hacks. To resolve this means removing a lot of what makes Open Source with forced standardization, and contributions like that of Behdad wonderful: choice. So I prefer to stick with more choice, or promoting/encouraging different text rendering applications to support different backends and more importantly properly detect the existence of each other when used. I am not a C programmer, but I think that is already where we are in the FOSS world. It is more a matter of awareness and interest: like other software markets (commerical or not), FOSS software with any significant user base will catch up. Find me Bengali Windows and I will laugh at you, but I have seen real impressive work with regionalized Linux distros, with full documentation for Indian communities that nothing proprietary comes close to because there is no interest.
Behdad, thanks again for your work. This is the second time your stuff up on HN in recent memory. I remember emailing you about BiCon years ago and you are still one of my open source heros with HarfBuzz and FriBidi and all that stuff. Keep up the good work and thank you for making FOSS software international.
Thanks for the nice words. I wrote BiCon over ten years ago and never touched it again. It definitely wasn't before SIGWINCH was introduced as I was barely four years old then :).
I checked the code, looks like code for SIGWINCH was there, but not working. I believe this got broken during the pty rewrite we did back then. At any rate. I just fixed that, and fixed initial window size. Enjoy, and in the future, feel free to shoot me an email if something I maintain bothers you!
Ah. Well then correction, you told me SIGWINCH was not implemented in BICON and you did not have the time to work on it, which is definitely understable.
I started using Linux almost ten years ago, and the winter for RTL languages were just starting to thaw.
Since I have come to using Arabic more often in the terminal, you would be suprised how often this causes problems as text rendering is done at many different layers and there is no harmony between them. Not only does it mean a special terminal emulator (mlterm for the win; esoteric but the only one that handles Arabic properly and handles Chinese, many Indic languages, and Hebrew unlike anyone else) but has all minaskinds of edge cases. Text rendering happens at so many layers and with different libraries, you have to make runtime changes to get certain programs to behave with each other. As a heavy Emacs user, Emacs will conflict with its own (default on) bidi implementation using m17n-lib (as mentioned in the article). So what happens? Arabic and other RTL languages go back to the original order, English-style, which is not what you want, as the operation happens twice since mlterm also uses this library (m17n-lib), but is not aware the bidi reordering happened before at the application layer. I finally got so sick of this my most important Emacs keyboard macro is to toggle off the Emacs internal bidi ordering on and off. Why you ask? Because despite mlterm and Emacs using m17n-lib on my computer, I need GUI emacs to use its own internal Bidi scheme, and the terminal run Emacs (emacsclient -nw) to run without out, and let mlterm pick up from there. I have tried other terminal emulators (someone wrote Perl scripts using, IIRC Text::FriBidi based one of the aforementioned library, GNU FriBidi) for urxvt to handle this), but it does not work as well: you get the correct ordering, but Arabic letters do not connect (this is really glyphs, not letters, but I digress). So if I do not toggle the Emacs macro I made, I cannot have a terminal that will show Arabic file names properly at the same time outside of Emacs, and a simple `ls` is all borked up.
GUI programs are not so much better. I run i3 with my own customizations, including heavy use of the wonderful dmenu utility for searching files indexed with Recoll, connection to my password manager, and running programs on the fly. Probably is dmenu specifically does not support Unicode at all, let alone RTL languages like Arabic, so you get a mess. I love suckless tools; and they avoided Unicode for a reaosn, but suckless, a very common group of power user tools ignoring Unicode for their own good and embracing their mantra of well manicured C code tells you to pity developers who do take the time and bear lots of whining about this crap being inconsistent and having crazy edge cases.
These edge cases will not stop me from using free software, because I had no idea how complicated Arabic rendering was until using Linux, and a lot of this crap flat out is not possible in Windows anyway (try listing Arabic file names in Command Prompt), and is still terrible when you get around to it working. Do we need a "GNOME3 of text rendering" as others quoted in the threads here? I am not sure we do. What we need is standardization of libraries or generalizing backends to choose what you need, a la Freedesktop initiatives in other spheres. Ironically: every terminal emulator and app developer/maintainer retorts when present RTL Arabic problems: this is an application layer concern and out of my hands. And what they ignore app developers with their application layer concerns apply with different layers of libraries unevenly, hence the aforementioned hacks. To resolve this means removing a lot of what makes Open Source with forced standardization, and contributions like that of Behdad wonderful: choice. So I prefer to stick with more choice, or promoting/encouraging different text rendering applications to support different backends and more importantly properly detect the existence of each other when used. I am not a C programmer, but I think that is already where we are in the FOSS world. It is more a matter of awareness and interest: like other software markets (commerical or not), FOSS software with any significant user base will catch up. Find me Bengali Windows and I will laugh at you, but I have seen real impressive work with regionalized Linux distros, with full documentation for Indian communities that nothing proprietary comes close to because there is no interest.
Behdad, thanks again for your work. This is the second time your stuff up on HN in recent memory. I remember emailing you about BiCon years ago and you are still one of my open source heros with HarfBuzz and FriBidi and all that stuff. Keep up the good work and thank you for making FOSS software international.