Re: Gdbm module

Re: Gdbm module

[Bart Schaefer]

On Feb 21, 12:21pm, Sebastian Gniazdowski wrote:
}
} Do you think commiting "zgdbmclear" I've sent doesn't have sense?

I would prefer a bit more thought be put into the design.  There's no
particular problem with the implementation underneath, but:

The whole point of "ztie -d ..." and having the modules named with the
/db/ piece in the middle is so that a generic interface to multiple
database backends can be implemented.  "ztie" and "zuntie" happen to
be in db_gdbm.c for now because there is no other such module, but
the intention was that there would be a base "zsh/db" module of which
zsh/db/gdbm was one implementation.

In keeping with that, there shouldn't be commands named "zgdbm..." and
there should be no "zgdbm_tied" parameter -- there should instead be a
single array or hash in the (so far nonexistent) base module to return
all the ztie'd names, and if there are other ops needed than z(un)tie
then something like "zdb path ...", "zdb clear ...", etc. to give a
consistent front to whatever backend has been passed to "ztie -d".
(Or option names/letters instead of subcommands, to be more consistent
with most builtins, but you get the idea.)

This may mean that there are some ops that either we don't provide for
any backend because they're too specific to one, or that fail for some
backends.  E.g. "zdb path" doesn't make sense for a network-connected
database; maybe it returns a URL in that case, or we should consider
that the -f FILENAME argument of ztie is not sufficiently general, and
"zdb path" would just return back that argument of ztie.

None of this means we need to rush off to implement zsh/db/sqlite or
something, but we should be thinking about it when extending db/gdbm.

Re: Gdbm module

[Sebastian Gniazdowski]

On Tue, Feb 21, 2017, at 09:25 AM, Bart Schaefer wrote:
> On Feb 21, 12:21pm, Sebastian Gniazdowski wrote:
> }
> } Do you think commiting "zgdbmclear" I've sent doesn't have sense?
> 
> I would prefer a bit more thought be put into the design.  There's no
> particular problem with the implementation underneath, but:
> 
> The whole point of "ztie -d ..." and having the modules named with the
> /db/ piece in the middle is so that a generic interface to multiple
> database backends can be implemented.  "ztie" and "zuntie" happen to
> be in db_gdbm.c for now because there is no other such module, but
> the intention was that there would be a base "zsh/db" module of which
> zsh/db/gdbm was one implementation.

The dream of general ztie.. I saw how difficult it is when writing. Now
it seems totally possible, but when writing, it's other thing. The need
to move "bin_ztie" function, for example. Who would accept my patch that
would do that? (grumbling mode ended). And implementation difficulties,
first of all. Two modules, one has to detect the other one, route call.
Hm, this seems now quite easy, but when writing I saw the difficulties.
And what arises from that is mkfs-like design. Hands up who called
mkfs.ext2 or .ext4 instead of mkfs wrapper. And, who would accept Oracle
DBM module into Zsh? It's a nice DBM, free, unix-like (chances it nicely
fits just like GDBM, has its gdbm_fdesc() call). Then, this database has
it's own customs, for example access methods: btree, heap. Chances for
new special parameters and stuff.

> In keeping with that, there shouldn't be commands named "zgdbm..." and
> there should be no "zgdbm_tied" parameter -- there should instead be a
> single array or hash in the (so far nonexistent) base module to return
> all the ztie'd names, and if there are other ops needed than z(un)tie
> then something like "zdb path ...", "zdb clear ...", etc. to give a
> consistent front to whatever backend has been passed to "ztie -d".
> (Or option names/letters instead of subcommands, to be more consistent
> with most builtins, but you get the idea.)

Abstraction can go together with specific parts. A general ztie can be
maybe accomplished, but having "zgdbm_tied" is like a functionality:
"Someone passed me (P flag) a parameter, I can check if it's gdbm
frontend via zgdbm_tied array". Then we would have "zodbm_tied" for
Oracle DBM. Specific things at first level of use cases, then abstract
ones for possible many use cases.

> This may mean that there are some ops that either we don't provide for
> any backend because they're too specific to one, or that fail for some
> backends.  E.g. "zdb path" doesn't make sense for a network-connected
> database; maybe it returns a URL in that case, or we should consider
> that the -f FILENAME argument of ztie is not sufficiently general, and
> "zdb path" would just return back that argument of ztie.

Ok, such strive for abstraction is a good thing. One has to be in good
position in Zsh world to work on that comfortably, think freely about
Oracle DBM and its traits, then he can have success in discovering
symmetrical API.

> None of this means we need to rush off to implement zsh/db/sqlite or
> something, but we should be thinking about it when extending db/gdbm.

I examined SQLite. There's a no-go – it doesn't return file descriptors
for database file and journal file. This is handy example now –
maintainers said: "file descriptors work on Unix, it would be not good
abstraction to return them".  Let's not do it, because it's not
abstract. Zsh is purely Unix world, file descriptor would be "fully
abstract", but no..  Here is full discussion:

http://sqlite.1065341.n5.nabble.com/how-to-get-file-handle-from-sqlite3-object-td28466.html

Then, as FDT_INTERNAL nor FDT_MODULE doesn't matter, SQLite might be
still possible. I looked at it, to do:

result=sqlarray["SELECT telephone FROM contacts WHERE id=1"]

And require single column to be selected, or concatenate columns, then
also append after \n when "...id=1 OR id=2". This doesn't match DBM
usage, though.

-- 
  Sebastian Gniazdowski
  psprint2@fastmail.com

Re: Gdbm module

[Bart Schaefer]

On Feb 21, 11:52am, Sebastian Gniazdowski wrote:
}
} [...] who would accept Oracle DBM module into Zsh? 

Depends entirely on the licensing Oracle specifies.

} I looked at [sqlite], to do:
} 
} result=sqlarray["SELECT telephone FROM contacts WHERE id=1"]
} 
} And require single column to be selected, or concatenate columns

I would not suggest attempting to make a full SQL interface from the
tied hash to the database.  GDBM is effectively one table per file
with a single unique key column, so the only thing you can do is
"SELECT * FROM ...".  For a SQL-type database you'd probably want
to do something like

    ztie -d db/sqlite -Q "SELECT ..." ... queryresult

and define queryresult in a manner similar to the way Perl DBI does
with fetchall_hashref.

This theoretical -Q option could be extended to GDBM as a way to
filter the set of keys fetched so the hash does not have to load the
entire dbmfile.  However, at first we might prefer to restrict things
to flatter single-table access.

Also note I'm only trying to offer some guidelines here; I've seen no
evidence that this is needed or more than academically interesting.

Re: Gdbm module

[Sebastian Gniazdowski]

On Tue, Feb 21, 2017, at 08:41 PM, Bart Schaefer wrote: 
> Depends entirely on the licensing Oracle specifies.

It's GNU Affero General Public License. Apparently it's like GPL, but
aimed at resolving issues with server software "distributed" (that's the
point) over network.

Just built it, configure runs fast, compilation is long (2:35) but code
is C.

>     ztie -d db/sqlite -Q "SELECT ..." ... queryresult
> 
> and define queryresult in a manner similar to the way Perl DBI does
> with fetchall_hashref.

Interesting, it assumes single row to be selected in Zsh conditions, as
far I understand. It is towards "tie", close or at conforming ztie.

> This theoretical -Q option could be extended to GDBM as a way to
> filter the set of keys fetched so the hash does not have to load the
> entire dbmfile.  However, at first we might prefer to restrict things
> to flatter single-table access.

If a way of specifying keys would have been developed (in above it's for
filtering keys) it could be also used to stick ~PM_UPTODATE to that
keys. This might somehow translate to SQL. Interesting. Not that I want
to patch my up-to-date solution, it allocates via zalloc() and hashtable
is always in good shape, previous code was doing hcallloc() and
returning internal GDBM string pointers.

> Also note I'm only trying to offer some guidelines here; I've seen no
> evidence that this is needed or more than academically interesting.

I've found use case. Turns out managing plugins via zshrc is rather a
pain. If I want to disable single plugin for a while, then `vim
~/.zshrc`, search, comment – it's much tiring. I will provide TUI
configurator for Zplugin. Each plugin will have .gdbm file. For TUI I
will use my 70% finished Zcurses UI library:

https://asciinema.org/a/aner2l63zm6bhmie3nxy3jkc7

It's based on (z) flag :) Having a text-box will mean "zcurses string
win ${(q)text}", i.e. quoting, but that's shellish.

Bottom line is – it might be possible to map some of Zshrc to gdbm file,
to be able to avoid whole `vim ~/.zshrc`, search, etc. cycle. Provided
that the cycle is really tiring.

So developing DBM modules would be boosted. I would go for writing new
module now, to hold ztie, zuntie, ... and route to db_gdbm, but I'm busy
with the ZUI lib. Maybe there's time before next release. Good that gdbm
is a gem, but Oracle dbm is also very interesting, then there's the
"non-existent" Berkley DB (it's probably installed on FreeBSD, a ghost
software if not available via tarball with configure, I couldn't find
it).

-- 
  Sebastian Gniazdowski
  psprint2@fastmail.com

Re: Gdbm module

[Bart Schaefer]

On Feb 22,  1:10am, Sebastian Gniazdowski wrote:
}
} On Tue, Feb 21, 2017, at 08:41 PM, Bart Schaefer wrote: 
} >     ztie -d db/sqlite -Q "SELECT ..." ... queryresult
} > 
} > and define queryresult in a manner similar to the way Perl DBI does
} > with fetchall_hashref.
} 
} Interesting, it assumes single row to be selected in Zsh conditions, as
} far I understand.

Either that, or the keys would be the unique key for each row and the
values would be either a single column, or encoded somehow so that
they could be split into fields.  We are somewhat limited by lacking
a mechanism for nested hashrefs, but consider the way "typeset -T" can
specify the format of array <-> scalar joining.

Re: Gdbm module

[Sebastian Gniazdowski]

On Wed, Feb 22, 2017, at 07:44 AM, Bart Schaefer wrote:
> We are somewhat limited by lacking
> a mechanism for nested hashrefs, but consider the way "typeset -T" can
> specify the format of array <-> scalar joining.

I plan to write a module that will just blindly put PM_HASH instead of
PM_HASHELEM into PM_HASH. See what will happen. The code handling value
retrieval is sophisticated, maybe even blind to structure, just follows
its PM_HASH etc. detection, I'm counting on that.

BTW. Retrieving from PM_SCALAR means double getter call. Saw that when
working on db_gdbm/zgdbm. I'm sure there's no error on my side, I've
even rolled out complete debugger walk. This could be an optimization,
to change/fix that.

-- 
  Sebastian Gniazdowski
  psprint2@fastmail.com