[bug] history corrupt on utf8

[bug] history corrupt on utf8

[Alexander Voikov]

% echo "АБВГДабвгд"
АБВГДабвгд

% tail -n 2 ~/.zsh_history
: 1371926671:0;echo "Ѓ�Ѓ�Ѓ�Ѓ�Ѓ�абвгд"

% zsh --version
zsh 5.0.2 (x86_64-pc-linux-gnu)

Debian Testing

Re: [bug] history corrupt on utf8

[Mikael Magnusson]

The history file isn't saved verbatim, since the command line can
contain non-utf8, nuls, etc.

On 22 June 2013 20:45, Alexander Voikov <alex-voikov@yandex.ru> wrote:
> % echo "АБВГДабвгд"
> АБВГДабвгд
>
> % tail -n 2 ~/.zsh_history
> : 1371926671:0;echo "Ѓ�Ѓ�Ѓ�Ѓ�Ѓ�абвгд"
>
> % zsh --version
> zsh 5.0.2 (x86_64-pc-linux-gnu)
>
> Debian Testing
>
>
>
>



-- 
Mikael Magnusson

Re: [bug] history corrupt on utf8

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Jun 22, 2013 7:07 PM, "Mikael Magnusson" <mikachu@gmail.com> wrote:
>
> The history file isn't saved verbatim, since the command line can
> contain non-utf8, nuls, etc.

To be more specific, the history file is saved in the same format used to
represent history lines internally, except for prefixing newlines with
backslash.  The file is only "corrupted" if zsh itself can't re-read it.
We gave up on sharing history files with other shells a long time ago.

Re: [bug] history corrupt on utf8

[Mikael Magnusson]

On 23 June 2013 20:33, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jun 22, 2013 7:07 PM, "Mikael Magnusson" <mikachu@gmail.com> wrote:
>>
>> The history file isn't saved verbatim, since the command line can
>> contain non-utf8, nuls, etc.
>
> To be more specific, the history file is saved in the same format used to
> represent history lines internally, except for prefixing newlines with
> backslash.  The file is only "corrupted" if zsh itself can't re-read it.
> We gave up on sharing history files with other shells a long time ago.

If you want, you can use this small program to "uncorrupt" it
(obviously don't overwrite the history file with the result),

#define Meta ((char) 0x83)

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>

/* from zsh utils.c */
char *unmetafy(char *s, int *len)
{
  char *p, *t;

  for (p = s; *p && *p != Meta; p++);
  for (t = p; (*t = *p++);)
    if (*t++ == Meta)
      t[-1] = *p++ ^ 32;
  if (len)
    *len = t - s;
  return s;
}

int main(int argc, char *argv[]) {
  char *line = NULL;
  size_t size;

  while (getline(&line, &size, stdin) != -1) {
    unmetafy(line, NULL);
    printf("%s", line);
  }

  if (line) free(line);
  return EXIT_SUCCESS;
}

--
Mikael Magnusson