zsh history gets destroyed when running out of disk space

zsh history gets destroyed when running out of disk space

[Michael Prokop]

[-- Attachment #1.1: Type: text/plain, Size: 777 bytes --]

Hi,

Problem:

Zsh truncates the zsh history file if you are running out of disk
space (AKA ENOSPC). If you don't have any space left in your $HOME
and exit zsh you'll find an empty $HISTFILE left.

Reason:

The savehistfile() function in Src/hist.c does not handle write
errors very smart.

Fix:

See attached patch (against version 4.3.4). The fix might need some
more work though. For example there should be much more and better
error handling; Zsh shouldn't exit with a return value of 0 when
such serious errors happen; the user should be informed about the
reason why it failed.

regards,
-mika-
-- 
 ,'"`.         http://www.michael-prokop.at/
(  grml.org -» Linux Live-CD for texttool-users and sysadmins
 `._,'         http://www.grml.org/

[-- Attachment #1.2: zsh_fix_history_enospc.patch --]
[-- Type: text/x-diff, Size: 864 bytes --]

--- Src/hist.c.orig	2007-06-11 21:02:42.000000000 +0200
+++ Src/hist.c	2007-06-11 21:58:37.000000000 +0200
@@ -2158,6 +2158,11 @@
     Histent he;
     zlong xcurhist = curhist - !!(histactive & HA_ACTIVE);
     int extended_history = isset(EXTENDEDHISTORY);
+    int write_file = 1; // by default assume we can write the file
+
+    // make sure we don't write anything in case of an error
+    if (errno == ENOSPC)
+      write_file = 0;
 
     if (!interact || savehistsiz <= 0 || !hist_ring
      || (!fn && !(fn = getsparam("HISTFILE"))))
@@ -2222,7 +2227,7 @@
 #endif
 	}
     }
-    if (out) {
+    if (out && write_file) {
 	for (; he && he->histnum <= xcurhist; he = down_histent(he)) {
 	    if ((writeflags & HFILE_SKIPDUPS && he->node.flags & HIST_DUP)
 	     || (writeflags & HFILE_SKIPFOREIGN && he->node.flags & HIST_FOREIGN)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: zsh history gets destroyed when running out of disk space

[Frank Terbeck]

Michael Prokop <news@michael-prokop.at>:
> Problem:
> 
> Zsh truncates the zsh history file if you are running out of disk
> space (AKA ENOSPC). If you don't have any space left in your $HOME
> and exit zsh you'll find an empty $HISTFILE left.
[...]

Hey, I experienced just the same a few days back. So, I also think
this should be addressed. Maybe a new option could be introduced, that
works similar to the 'check_jobs' option that warns if jobs would be
killed. Similarly a 'hist_check_space' option could (if set) warn, if
a history file can not be written to disk.

In any case, I agree with Michael, that a history file should not be
truncated to zero-size if the needed space is not available. In that
case it's better, IMHO, to keep the old history file.

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: zsh history gets destroyed when running out of disk space

[Peter Stephenson]

On Thu, 21 Jun 2007 10:04:54 +0200
Frank Terbeck <ft@bewatermyfriend.org> wrote:
> Michael Prokop <news@michael-prokop.at>:
> > Problem:
> > 
> > Zsh truncates the zsh history file if you are running out of disk
> > space (AKA ENOSPC). If you don't have any space left in your $HOME
> > and exit zsh you'll find an empty $HISTFILE left.
> 
> Hey, I experienced just the same a few days back. So, I also think
> this should be addressed.

The following detects errors during writing better.  If you're writing a
new file it'll leave it wherever it gets to; if you're replacing an old
file it will leave the old file alone.  That's about the best we can do.
I haven't tested this on a failure, but it should be better than what
we've got.

Index: Src/hist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/hist.c,v
retrieving revision 1.66
diff -u -r1.66 hist.c
--- Src/hist.c	1 Nov 2006 12:25:22 -0000	1.66
+++ Src/hist.c	22 Jun 2007 20:19:50 -0000
@@ -2158,6 +2158,7 @@
     Histent he;
     zlong xcurhist = curhist - !!(histactive & HA_ACTIVE);
     int extended_history = isset(EXTENDEDHISTORY);
+    int ret;
 
     if (!interact || savehistsiz <= 0 || !hist_ring
      || (!fn && !(fn = getsparam("HISTFILE"))))
@@ -2242,59 +2243,69 @@
 	    }
 	    t = start = he->node.nam;
 	    if (extended_history) {
-		fprintf(out, ": %ld:%ld;", (long)he->stim,
-			he->ftim? (long)(he->ftim - he->stim) : 0L);
+		ret = fprintf(out, ": %ld:%ld;", (long)he->stim,
+			      he->ftim? (long)(he->ftim - he->stim) : 0L);
 	    } else if (*t == ':')
-		fputc('\\', out);
+		ret = fputc('\\', out);
 
-	    for (; *t; t++) {
+	    for (; ret >= 0 && *t; t++) {
 		if (*t == '\n')
-		    fputc('\\', out);
-		fputc(*t, out);
+		    if ((ret = fputc('\\', out)) < 0)
+			break;
+		if ((ret = fputc(*t, out)) < 0)
+		    break;
 	    }
-	    fputc('\n', out);
+	    if (ret < 0 || (ret = fputc('\n', out)) < 0)
+		break;
 	}
-	if (start && writeflags & HFILE_USE_OPTIONS) {
+	if (ret >= 0 && start && writeflags & HFILE_USE_OPTIONS) {
 	    struct stat sb;
-	    fflush(out);
-	    if (fstat(fileno(out), &sb) == 0) {
-		lasthist.fsiz = sb.st_size;
-		lasthist.mtim = sb.st_mtime;
+	    if ((ret = fflush(out)) >= 0) {
+		if (fstat(fileno(out), &sb) == 0) {
+		    lasthist.fsiz = sb.st_size;
+		    lasthist.mtim = sb.st_mtime;
+		}
+		zsfree(lasthist.text);
+		lasthist.text = ztrdup(start);
 	    }
-	    zsfree(lasthist.text);
-	    lasthist.text = ztrdup(start);
-	}
-	fclose(out);
-	if (tmpfile) {
-	    if (rename(tmpfile, unmeta(fn)) < 0)
-		zerr("can't rename %s.new to $HISTFILE", fn);
-	    free(tmpfile);
 	}
+	if (fclose(out) < 0 && ret >= 0)
+	    ret = -1;
+	if (ret >= 0) {
+	    if (tmpfile) {
+		if (rename(tmpfile, unmeta(fn)) < 0)
+		    zerr("can't rename %s.new to $HISTFILE", fn);
+		free(tmpfile);
+	    }
 
-	if (writeflags & HFILE_SKIPOLD
-	 && !(writeflags & (HFILE_FAST | HFILE_NO_REWRITE))) {
-	    int remember_histactive = histactive;
-
-	    /* Zeroing histactive avoids unnecessary munging of curline. */
-	    histactive = 0;
-	    /* The NULL leaves HISTFILE alone, preserving fn's value. */
-	    pushhiststack(NULL, savehistsiz, savehistsiz, -1);
-
-	    hist_ignore_all_dups |= isset(HISTSAVENODUPS);
-	    readhistfile(fn, err, 0);
-	    hist_ignore_all_dups = isset(HISTIGNOREALLDUPS);
-	    if (histlinect)
-		savehistfile(fn, err, 0);
+	    if (writeflags & HFILE_SKIPOLD
+		&& !(writeflags & (HFILE_FAST | HFILE_NO_REWRITE))) {
+		int remember_histactive = histactive;
+
+		/* Zeroing histactive avoids unnecessary munging of curline. */
+		histactive = 0;
+		/* The NULL leaves HISTFILE alone, preserving fn's value. */
+		pushhiststack(NULL, savehistsiz, savehistsiz, -1);
+
+		hist_ignore_all_dups |= isset(HISTSAVENODUPS);
+		readhistfile(fn, err, 0);
+		hist_ignore_all_dups = isset(HISTIGNOREALLDUPS);
+		if (histlinect)
+		    savehistfile(fn, err, 0);
 
-	    pophiststack();
-	    histactive = remember_histactive;
+		pophiststack();
+		histactive = remember_histactive;
+	    }
 	}
-    } else if (err) {
+    } else
+	ret = -1;
+
+    if (ret < 0 && err) {
 	if (tmpfile) {
-	    zerr("can't write history file %s.new", fn);
+	    zerr("can't write history file %s.new: %e", fn);
 	    free(tmpfile);
 	} else
-	    zerr("can't write history file %s", fn);
+	    zerr("can't write history file %s: %e", fn);
     }
 
     unlockhistfile(fn);

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh history gets destroyed when running out of disk space

[Michael Prokop]

[-- Attachment #1: Type: text/plain, Size: 1458 bytes --]

* Peter Stephenson <p.w.stephenson@ntlworld.com> [20070622 22:31]:
> On Thu, 21 Jun 2007 10:04:54 +0200 Frank Terbeck <ft@bewatermyfriend.org> wrote:
> > Michael Prokop <news@michael-prokop.at>:

> > > Problem:
> > > Zsh truncates the zsh history file if you are running out of disk
> > > space (AKA ENOSPC). If you don't have any space left in your $HOME
> > > and exit zsh you'll find an empty $HISTFILE left.

> > Hey, I experienced just the same a few days back. So, I also think
> > this should be addressed.

> The following detects errors during writing better.  If you're writing a
> new file it'll leave it wherever it gets to; if you're replacing an old
> file it will leave the old file alone.  That's about the best we can do.
> I haven't tested this on a failure, but it should be better than what
> we've got.

Verified, works.

Exiting shell quits with:

| zsh: failed to write history file /home/grml/.zsh_history.new: unknown error 1182546552

and an empty /home/grml/.zsh_history.new file is left. My original
/home/grml/.zsh_history is left untouched (no truncating anymore).
During a shell session the history file is getting longer only as
long as there a free bits left (as you intented to). The error code
might be improved though. ;)

regards,
-mika-
-- 
 ,'"`.         http://www.michael-prokop.at/
(  grml.org -» Linux Live-CD for texttool-users and sysadmins
 `._,'         http://www.grml.org/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: zsh history gets destroyed when running out of disk space

[Peter Stephenson]

On Sat, 23 Jun 2007 00:40:09 +0200
Michael Prokop <news@michael-prokop.at> wrote:
> | zsh: failed to write history file /home/grml/.zsh_history.new: unknown error 1182546552
> 
> The error code might be improved though. ;)

Thanks for trying it.

I forgot that %e doesn't use errno directly, it expects an argument
(that's because sometimes we save the number and do something else
before reporting the error).

I've also reset errno to 0 beforehand, on the off chance that one of the
possible failures doesn't set errno.  I don't think that should happen,
but I don't want a stray ENOENT from a previous error confusing the
matter.

Index: Src/hist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/hist.c,v
retrieving revision 1.67
diff -u -r1.67 hist.c
--- Src/hist.c	22 Jun 2007 20:32:20 -0000	1.67
+++ Src/hist.c	23 Jun 2007 18:05:01 -0000
@@ -2190,6 +2190,7 @@
 	if (isset(SHAREHISTORY))
 	    extended_history = 1;
     }
+    errno = 0;
     if (writeflags & HFILE_APPEND) {
 	tmpfile = NULL;
 	out = fdopen(open(unmeta(fn),
@@ -2302,10 +2303,10 @@
 
     if (ret < 0 && err) {
 	if (tmpfile) {
-	    zerr("failed to write history file %s.new: %e", fn);
+	    zerr("failed to write history file %s.new: %e", fn, errno);
 	    free(tmpfile);
 	} else
-	    zerr("failed to write history file %s: %e", fn);
+	    zerr("failed to write history file %s: %e", fn, errno);
     }
 
     unlockhistfile(fn);

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/