(Fwd) Re: [PATCH] db/gdbm rewrite

(Fwd) Re: [PATCH] db/gdbm rewrite

[Bart Schaefer]

So ... I wrote the following a few days ago and didn't remember to use my
own gmail-block workaround, so it never made it to the list.  Still, it
seems to remain relevant to the latest patches.


On Feb 14,  4:20am, Sebastian Gniazdowski wrote:
}
} -- hashtable is filled normally, initially with ~PM_UPTODATE
} -- Param "dereference" detects ~PM_UPTODATE, uses database, sets u.str,
} sets PM_UPTODATE

This is not going to provide equivlent behavior, is it? If the database
is being read by zsh and updated by some other process, how does zsh know
that it needs to re-fetch what has now become a cached value?

This PM_UPTODATE flag also points out something that I've sort of had
percolating for a while -- we need a flag bit or two reserved for use
by modules.  (A counter-argument is that we're all out of flag bits.)

Re: (Fwd) Re: [PATCH] db/gdbm rewrite

[Sebastian Gniazdowski]

On Sat, Feb 18, 2017, at 04:43 PM, Bart Schaefer wrote: 
> On Feb 14,  4:20am, Sebastian Gniazdowski wrote:
> }
> } -- hashtable is filled normally, initially with ~PM_UPTODATE
> } -- Param "dereference" detects ~PM_UPTODATE, uses database, sets u.str,
> } sets PM_UPTODATE
> 
> This is not going to provide equivlent behavior, is it? If the database
> is being read by zsh and updated by some other process, how does zsh know
> that it needs to re-fetch what has now become a cached value?

As the other thread pointed out, GDBM_SYNC flag means no change to
database can be done, even when opening in read-only mode. This is
certain, and allows PM_UPTODATE to be used. The possible concurrent
change in Zsh forked process is a hole. It can be turned into an
feature, though. A new builtin "zgdbmfetch paramname key" could be
provided to re-fetch element when someone is going to try to do
concurrent access with inherited FD and GDBM_SYNC lock, say with use of
zsystem flock on other FD.

-- 
Sebastian Gniazdowski
psprint2@fastmail.com

Re: (Fwd) Re: [PATCH] db/gdbm rewrite

[Sebastian Gniazdowski]

On Sun, Feb 19, 2017, at 12:46 AM, Sebastian Gniazdowski wrote:
> change in Zsh forked process is a hole. It can be turned into an
> feature, though. A new builtin "zgdbmfetch paramname key" could be
> provided to re-fetch element when someone is going to try to do
> concurrent access with inherited FD and GDBM_SYNC lock, say with use of
> zsystem flock on other FD.

PS. Doing "fetch" might seem not nice in "hash-as-database" frontend,
but it would just clear PM_UPTODATE flag, rest is automatic. So maybe
"zgdbmclear param key".  In my gdbm update I allocate regular Params
instead of heap-arena-ghosts, access database only when needed
leveraging PM_UPTODATE and GDBM_SYNC, it might be even possible to open
database without GDBM_SYNC and do locking with a "gdbmlock" builtin.

-- 
  Sebastian Gniazdowski
  psprint2@fastmail.com

Re: [PATCH] db/gdbm rewrite

[Bart Schaefer]

On Feb 19, 12:46am, Sebastian Gniazdowski wrote:
}
} As the other thread pointed out, GDBM_SYNC flag means no change to
} database can be done, even when opening in read-only mode.

?? That's not what it means at all:

   GDBM_READER reader
   GDBM_WRITER writer
   GDBM_WRCREAT writer - if database does not exist create new one
   GDBM_NEWDB writer - create new database regardless if one exists
   For  the  last  three  (writers  of the database) the following may be
   added added to read_write by bitwise or: GDBM_SYNC, which  causes  all
   database  operations  to be synchronized to the disk, and GDBM_NOLOCK,
   which prevents the library from performing any locking on the database
   file.

GDBM_SYNC is meaningless for read-only mode; in any write mode it means
that changes made to by the program are immediately flushed out to the
file, so that other readers can immediately see the change.  Locking is
a separate op.  zsh/db/gdbm does use locking by default (does not pass
teh NOLOCK flag); I have never deeply investigated how gdbm handles that
underneath.

Man page also says:

   It is important that every file opened is also closed.  This is needed
   to update the reader/writer count on the file.

Why would there be a reader/writer count if there is no concurrent change
allowed?

Re: [PATCH] db/gdbm rewrite

[Sebastian Gniazdowski]

On Sun, Feb 19, 2017, at 10:19 AM, Bart Schaefer wrote: 
> GDBM_SYNC is meaningless for read-only mode; in any write mode it means
> that changes made to by the program are immediately flushed out to the

Ah, I had a matrix on my eyes and read GDBM_SYNC as GDBM_LOCK.

> Man page also says:
> 
>    It is important that every file opened is also closed.  This is needed
>    to update the reader/writer count on the file.
> 
> Why would there be a reader/writer count if there is no concurrent change
> allowed?

It must be for multiple threads. Sorry for not reading the docs doing
update, but by testing gdbmtool behavior I have got correct image:
- database access is guarded by locks
- writing to single file by 2 threads without locking corrupts database
- gdbm isn't a server and doesn't order transactions with a level of
concurrency

We could update db_gdbm to use GDBM_NOLOCK and seamlessly ~PM_UPTODATE,
providing a zgdbmlock builtin. I did a good thing with PM_UPTODATE, all
doors are open.

-- 
Sebastian Gniazdowski