From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 9407 invoked from network); 30 Sep 2004 08:44:02 -0000 Received: from news.dotsrc.org (HELO a.mx.sunsite.dk) (130.225.247.88) by ns1.primenet.com.au with SMTP; 30 Sep 2004 08:44:02 -0000 Received: (qmail 98534 invoked from network); 30 Sep 2004 08:43:57 -0000 Received: from sunsite.dk (130.225.247.90) by a.mx.sunsite.dk with SMTP; 30 Sep 2004 08:43:57 -0000 Received: (qmail 224 invoked by alias); 30 Sep 2004 08:43:29 -0000 Mailing-List: contact zsh-workers-help@sunsite.dk; run by ezmlm Precedence: bulk X-No-Archive: yes X-Seq: 20422 Received: (qmail 26428 invoked from network); 30 Sep 2004 08:36:49 -0000 Received: from unknown (HELO a.mx.sunsite.dk) (130.225.247.88) by sunsite.dk with SMTP; 30 Sep 2004 08:36:49 -0000 Received: (qmail 89821 invoked from network); 30 Sep 2004 08:35:49 -0000 Received: from m50.net81-65-198.noos.fr (HELO charonne.dyndns.org) (81.65.198.50) by a.mx.sunsite.dk with SMTP; 30 Sep 2004 08:35:47 -0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by charonne.dyndns.org (Postfix) with ESMTP id 4092A3CC03D; Thu, 30 Sep 2004 10:35:42 +0200 (CEST) Subject: Re: history file being truncated (konsole with multiple shells context) From: Samuel Krempp To: zsh-workers@sunsite.dk Cc: Wayne Davison , Samuel Krempp , zsh-users@sunsite.dk In-Reply-To: References: <1088937610.4949.41.camel@marvin.localdomain> <20040929230327.GC7866@blorf.net> Content-Type: text/plain Message-Id: <1096533341.15338.79.camel@marvin.localdomain> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Thu, 30 Sep 2004 10:35:41 +0200 Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 2.63 on a.mx.sunsite.dk X-Spam-Level: **** X-Spam-Status: No, hits=4.2 required=6.0 tests=RCVD_IN_DYNABLOCK,RCVD_IN_RFCI, RCVD_IN_SORBS autolearn=no version=2.63 X-Spam-Hits: 4.2 I'm really glad this subject is finally discussed, whatever the result will be I'm very grateful to all zsh workers who take some time to think over this issue. I was beginning to feel like I were the only one concerned about that.. On Thu, 2004-09-30 at 06:39, Bart Schaefer wrote: > We could do something sneaky like fork a child with most signals blocked > that does nothing but write the history file and exit, while the parent > exits immediately to fake out _its_ parent. That's not a very resource- > friendly idea, though, because (among other things) it might fail due to > inability to fork. However, it's probably the only one that covers all > cases short of catastrophe. In the cases of computer crash / brutal reset, the user should know bad things can happen anyway. but still, maybe zsh can even cover those safely. I think the history update can not be completely safe without some kind of "atomic transaction" technique, to avoid any problem when zsh gets stopped at any time. > > Updating a .new file and then moving it into place would be a safer way > > to ensure that the history file never gets truncated. > > On the other hand, in that case the changes get lost entirely, which might > be just as confusing. well, it's already better than the risk of losing the entire history. And in fact, I think it's possible to design a scheme where you can not lose more than the few last history items. For instance : 1. every second save in a file the latest commands, in the simplest append way, without any processing. Its only purpose is crash-recovery. Let's call this the temp files (I guess we need one temp file per zsh instance in order to do plain, safe appends) 2.a every minute (and on exit), do the whole history processing and create a ${HISTFILE}.new file, 2.b then erase the tempfile 2.c then move ${HISTFILE}.new to ${HISTFILE} This way, it should be possible to recover all hanging modifications even after a crash, whatever zsh was doing at the time : at zsh startup, (0 : parse the history file) 1. check for temp files. If there are any, ignore the .new file if it exists (it wasn't completely written, else the temp files would be cleared), process the temp files into memory, and start over at step 2.a Else, if a .new file exists start at 2.c Else, it means zsh had time to finish its history update on last exit, and just start at step 1. Such a scheme should be safe whatever happens, its only vulnerable parts are its pseudo-atomic transactions, 2.c (i.e. the move was not completed on the filesystem, and as a result the old histfile is unlinked and the new one is not yet relinked. maybe that's not even possible with modern filesystems though) and 2.b (i.e. not all files were erased - though by using a dummy temp file to be erased first, partial erase of the temp files is detected. this step might in fact be made safe). A filesystem expert can probably make even those steps safe. (and in any case they're so fast that their being interrupted is unprobable) Now, I agree it's not the easiest solution, and easier ones might very well be satisfactory enough. Hmm, I'm realizing this kind of scheme is not really specific to shell history updates anyway, there might already be such a library/tool somewhere that does everything the right way. Or, at least, it's available with any database. If zsh can afford dependancies on a database library, storing the history in a database could be a solution :-) > > ... the INC_APPEND_HISTORY and SHARE_HISTORY add new lines onto the end > > of the history file via a normal appending write. When the file gets to > > be a certain percent larger than its configured size, it gets rewritten > > to bring its size back down. > > So a signal in the middle of that rewrite would still be an issue, even if > we change the exit-time behavior. ah yes. This would make the issue less probable, but not solve it completely. -- Samuel