Unicode hyphen not displayed properly [9]

I found that a Unicode hyphen, represented in a man page file as \(hy , does not display correctly in XShell in most fonts. It either displays as a space, or as a nonsense character like a square.

However, if I copy and paste the text containing the unicode hyphen from XShell into Windows Notepad, and choose any of those fonts, then the unicode hyphen displays correctly.

Here are the fonts that I have tested under Windows XShell, with 'yes' if the unicode hyphen displays correctly, and 'no' if it does not display correctly. Also, the type of font is list as OT=OpenType, TT=TrueType, or Raster.

yes Consolas OT
no Courier Raster
no Courier New OT
yes DejaVu Sans Mono TT
no Fixedsys Raster
no Inconsolata TT
no Liberation Mono TT
no Lucida Console OT
no Terminal Raster

Note that of the two fonts that worked, Consolas is OpenType and DejaVu Sans Mono is TrueType.


Program Ver. : Xshell 4

Thank you for your thorough explanation. To help us duplicate the problem in our lab, can you tell us the followings?

1. Result of 'echo $LANG'
2. Which 'man' page are you experiencing this problem?
3. Screenshot of the problem

---
Technical Support

1) en_US.UTF-8

2) The specific man page is not particularly relevant, only that it is using a Unicode hyphen \(hy rather than the more common minus sign \- , but the name of the program is SnapRAID

3) here is a screenshot with Courier New font: http://i.imgur.com/Q7DXuzF.png

I have set the encoding to match yours and try opening the following manual pages:

top, ls, copy, grep, xterm, xargs

hyphens in these manuals displayed fine.

Please make sure you have selected Unicode (UTF-8). It is on the Globe icon in the Standard toolbar.

If the problem persists, can you capture the steps using the psr.exe (standard Windows applications) and send it to us?

---
Technical Support

None of those man pages you tried use \(hy

I thought I already explained this clearly. The problem is specifically with \(hy which is the Unicode encoding for a hyphen. Most man pages do not use that character, but rather use \-

And of course the Translation is set to Unicode (UTF-8). If it were not, then
the Unicode hyphen would not display correctly in any fonts, rather than
working with some fonts.

Yes. You made the point very clear. But, unicode has multiple hyphens and I am not sure which one this specific case is related to.

http://www.cs.tut.fi/~jkorpela/dashes.html
http://en.wikipedia.org/wiki/Hyphen#Unicode

Also, when I report this problem to our developers, I need to explain how we can duplicate the problem so that it can be fixed faster.

Can you tell us which man page includes the unicode hyphen?



---
Technical Support

I already told you which man page it is, SnapRAID. You can download the source code and man page if you want. But that is hardly necessary.

As I already said, the man page (i.e., groff) encoding for the character in question is \(hy

That is the plain Unicode hyphen. U+2010 Or HTML ‐

You can easily make your own man page with this character.

Copy any *.1 man page file and edit it to include the man page encoding (i.e., groff) for a Unicode hyphen which I already explained is \(hy

If you like, look for \- in the man page you copied, which is the minus sign that works fine in XShell. And replace a few of them with \(hy

Then you can view the man page with: man -l yourfile.1

It looks like some fonts does not know how to represent Unicode hyphen. We have tested with few other terminal emulators and the problem exists there as well.

We might be able to map unprintable characters to alternative characters in the given font set but will have to do more research on this.

Thank you for reporting this problem.



---
Technical Support

I also noticed that this problem does not exist under terminals running in linux under Gnome (gnome-terminal, xterm). Either linux fonts just include more characters than Windows fonts, or else the linux terminals are doing the trick you mentioned of substituting other characters for missing characters. Either way, the linux Gnome terminals I tried had no trouble displaying the Unicode hyphen with all fonts.

I see that gnome-terminal can display unicode hyphen even with the fonts that do not have the character in the font. It has been reported to the developers.

Thank you for your time and effort on trying to explain this. We will do our best to include the support in the near future.



---
Technical Support