Re: history file being truncated (konsole with multiple shells context)

Re: history file being truncated (konsole with multiple shells context)

[Samuel Krempp]

I'm really glad this subject is finally discussed, whatever the result
will be I'm very grateful to all zsh workers who take some time to think
over this issue.
I was beginning to feel like I were the only one concerned about that..

On Thu, 2004-09-30 at 06:39, Bart Schaefer wrote:
> We could do something sneaky like fork a child with most signals blocked 
> that does nothing but write the history file and exit, while the parent 
> exits immediately to fake out _its_ parent.  That's not a very resource- 
> friendly idea, though, because (among other things) it might fail due to 
> inability to fork.  However, it's probably the only one that covers all 
> cases short of catastrophe.

In the cases of computer crash / brutal reset, the user should know bad
things can happen anyway. but still, maybe zsh can even cover those
safely.

I think the history update can not be completely safe without some kind
of "atomic transaction" technique, to avoid any problem when zsh gets
stopped at any time.

> > Updating a .new file and then moving it into place would be a safer way 
> > to ensure that the history file never gets truncated.
> 
> On the other hand, in that case the changes get lost entirely, which might
> be just as confusing.

well, it's already better than the risk of losing the entire history.
And in fact, I think it's possible to design a scheme where you can not
lose more than the few last history items.
For instance :
1. every second save in a file the latest commands, in the simplest
append way, without any processing. Its only purpose is crash-recovery.
Let's call this the temp files
(I guess we need one temp file per zsh instance in order to do plain,
safe appends)
2.a every minute (and on exit), do the whole history processing and
create a ${HISTFILE}.new file, 
  2.b then erase the tempfile
  2.c then move ${HISTFILE}.new to ${HISTFILE}

This way, it should be possible to recover all hanging modifications
even after a crash, whatever zsh was doing at the time :
at zsh startup, 
(0 : parse the history file)
1. check for temp files. If there are any, ignore the .new file if it
exists (it wasn't completely written, else the temp files would be
cleared), process the temp files into memory, and start over at step 2.a
 Else, if a .new file exists start at 2.c 
Else, it means zsh had time to finish its history update on last exit,
and just start at step 1.

Such a scheme should be safe whatever happens, its only vulnerable parts
are its pseudo-atomic transactions, 2.c (i.e. the move was not completed
on the filesystem, and as a result the old histfile is unlinked and the
new one is not yet relinked. maybe that's not even possible with modern
filesystems though) and 2.b (i.e. not all files were erased - though by
using a dummy temp file to be erased first, partial erase of the temp
files is detected. this step might in fact be made safe).  
A filesystem expert can probably make even those steps safe. (and in any
case they're so fast that their being interrupted is unprobable)

Now, I agree it's not the easiest solution, and easier ones might very
well be satisfactory enough.


Hmm, I'm realizing this kind of scheme is not really specific to shell
history updates anyway, there might already be such a library/tool
somewhere that does everything the right way.
Or, at least, it's available with any database.
If zsh can afford dependancies on a database library, storing the
history in a database could be a solution  :-)

> > ... the INC_APPEND_HISTORY and SHARE_HISTORY add new lines onto the end
> > of the history file via a normal appending write.  When the file gets to
> > be a certain percent larger than its configured size, it gets rewritten
> > to bring its size back down.
> 
> So a signal in the middle of that rewrite would still be an issue, even if
> we change the exit-time behavior.

ah yes. This would make the issue less probable, but not solve it
completely.
-- 
Samuel