Fw: ZSH history file VS. UTF-8 data

Fw: ZSH history file VS. UTF-8 data

[Peter Stephenson]

I got this.  I haven't investigated.  Replies will need to be copied
directly.


Begin forwarded message:

Date: Wed, 2 Mar 2011 11:49:29 +0900
From: ıɥɔıɐʇ ɐʇɐqɐʍɐʞ <kawabata.taichi@gmail.com>
To: <pws@csr.com>
Subject: ZSH history file VS. UTF-8 data


Dear sir,

I'm sorry to disturb you.

I've been a user of ZSH  for a long time and I really
appreciate for the developer of this great software.

Recently, I've got one trouble in ZSH history file, and as I tried to
solve it by
surveying Webs and Sources, with no help.  As of it, knowing that this
disturbs you,
I'm trying to ask for any help.

The trouble is as follows:

When I try to use UTF-8 file name in shell command, ZHS history file
seems to save
it with "meta code".

For example, executing
 $ ls \346\226\207\345\255\227  (octal expression of "ls 文字”）
results in histfile
 $ ls \346\203\266\203\247\2\345\255\203\267

That is, when 0x80-0x9F characters are used, then always 0x83 Meta
character is inserted and following character is bit shifted, resulting
garbage history.

Any way to avoid this situation?

Any help is really appreciated,

Sincerely,

-- 
---------------------------------------------------------------------
　川幡 太一 (KAWABATA, Taichi)　　　E-mail: kawabata.taichi@gmail.com
]


 To report this email as spam click
 https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg== .


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: Fw: ZSH history file VS. UTF-8 data

[Mikael Magnusson]

> Dear sir,
>
> I'm sorry to disturb you.
>
> I've been a user of ZSH  for a long time and I really
> appreciate for the developer of this great software.
>
> Recently, I've got one trouble in ZSH history file, and as I tried to
> solve it by
> surveying Webs and Sources, with no help.  As of it, knowing that this
> disturbs you,
> I'm trying to ask for any help.
>
> The trouble is as follows:
>
> When I try to use UTF-8 file name in shell command, ZHS history file
> seems to save
> it with "meta code".
>
> For example, executing
>  $ ls \346\226\207\345\255\227  (octal expression of "ls 文字”）
> results in histfile
>  $ ls \346\203\266\203\247\2\345\255\203\267
>
> That is, when 0x80-0x9F characters are used, then always 0x83 Meta
> character is inserted and following character is bit shifted, resulting
> garbage history.
>
> Any way to avoid this situation?
>
> Any help is really appreciated,

This isn't a bug, the history file is saved in metafied format. If you
want to print it outside zsh you can use this simple program.

#define Meta ((char) 0x83)

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>

/* from zsh utils.c */
char *unmetafy(char *s, int *len)
{
  char *p, *t;

  for (p = s; *p && *p != Meta; p++);
  for (t = p; (*t = *p++);)
    if (*t++ == Meta)
      t[-1] = *p++ ^ 32;
  if (len)
    *len = t - s;
  return s;
}

int main(int argc, char *argv[]) {
  char *line = NULL;
  size_t size;

  while (getline(&line, &size, stdin) != -1) {
    unmetafy(line, NULL);
    printf("%s", line);
  }

  if (line) free(line);
  return EXIT_SUCCESS;
}


-- 
Mikael Magnusson

Re: Fw: ZSH history file VS. UTF-8 data

[lilydjwg]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On Wed, Mar 02, 2011 at 09:43:36AM +0000, Peter Stephenson wrote:
> I got this.  I haven't investigated.  Replies will need to be copied
> directly.
> 
> 
> Begin forwarded message:
> 
> Date: Wed, 2 Mar 2011 11:49:29 +0900
> From: ıɥɔıɐʇ ɐʇɐqɐʍɐʞ <kawabata.taichi@gmail.com>
> To: <pws@csr.com>
> Subject: ZSH history file VS. UTF-8 data
> 
> 
> Dear sir,
> 
> I'm sorry to disturb you.
> 
> I've been a user of ZSH  for a long time and I really
> appreciate for the developer of this great software.
> 
> Recently, I've got one trouble in ZSH history file, and as I tried to
> solve it by
> surveying Webs and Sources, with no help.  As of it, knowing that this
> disturbs you,
> I'm trying to ask for any help.
> 
> The trouble is as follows:
> 
> When I try to use UTF-8 file name in shell command, ZHS history file
> seems to save
> it with "meta code".
> 
> For example, executing
>  $ ls \346\226\207\345\255\227  (octal expression of "ls 文字”）
> results in histfile
>  $ ls \346\203\266\203\247\2\345\255\203\267
> 
> That is, when 0x80-0x9F characters are used, then always 0x83 Meta
> character is inserted and following character is bit shifted, resulting
> garbage history.
> 
> Any way to avoid this situation?
> 
> Any help is really appreciated,
> 

I met the same problem, and had a look at the zsh source code, attached
is the C program I use to deal with this. However, I have not found an
option to turn this feature off or why the history file is encoded in
this way.

-- 
Best regards,
lilydjwg

Linux Vim Python 我的博客
http://lilydjwg.is-programmer.com/

[-- Attachment #2: zhist.c --]
[-- Type: text/x-csrc, Size: 1584 bytes --]

//=====================================================================
// 让 zsh 的历史记录可读
//=====================================================================
#include<unistd.h>
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
//---------------------------------------------------------------------
void readhist();
void writehist();
void usage();
static int ismeta(unsigned char ch);
//---------------------------------------------------------------------
int main(int argc, char** argv){
  if(argc == 1)
    readhist();
  else if(argc == 2){
    if(!strcmp(argv[1], "read"))
      readhist();
    else if(!strcmp(argv[1], "write"))
      writehist();
    else
      usage();
  }else
    usage();
  return 0;
}
void readhist(){
  unsigned char c, d;
  int change = 0;
  while(read(0, &c, 1)){
    if(c != 0x83){
      if(change){
	d = c ^ 32;
      }else
	d = c;
      write(1, &d, 1);
      change = 0;
    }else
      change = 1;
  }
}
void writehist(){
  unsigned char c, d;
  while(read(0, &c, 1)){
    if(ismeta(c)){
      d = 0x83;
      write(1, &d, 1);
      d = c ^ 32;
      write(1, &d, 1);
    }else
      write(1, &c, 1);
  }
}
void usage(){
  fprintf(stderr, "Usage: zhist [read|write]\n"
      "\tzhist reads from stdin and outputs to stdout, "
      "with the content changes\nto be read by human being "
      "or by the zsh program.\n");
  exit(1);
}
static int ismeta(unsigned char ch){
  return (ch > 0x83 && ch < 0x9e)
    || ch == 0xa0 || ch == 0x83 || ch == 0;
}
//=====================================================================