[9fans] mkindex of dict(7)

[9fans] mkindex of dict(7)

[Akshat Kumar]

In its current state, /sys/src/cmd/dict/mkindex suicides if
/lib/dict/oed2 is not present and '-d' option is not specified (along
with the dict name) -- fix:
	move /sys/src/cmd/dict/mkindex:57 to
	after /sys/src/cmd/dict/mkindex:62
(that is, place the Bseek after the conditional on Bopen/bdict)

Is no one is using it these days?
ak

Re: [9fans] mkindex of dict(7)

[Akshat Kumar]

/sys/src/cmd/dict/dict.c states:
/*
 * Assumed index file structure: lines of form
 * 	[^\t]+\t[0-9]+
 * First field is key, second is byte offset into dictionary.
 * Should be sorted with args -u -t'	' +0f -1 +0 -1 +1n -2
 */

whereas, /sys/src/cmd/dict/mkindex outputs:
<byte offset>	<key>
i.e.,
0	ヽ [くりかえし]
(custom dictionary from EDICT)
or
158928	Ab*sorb`a*bil"i*ty
(pgw)

thus, the resulting index from mkindex seems to not be usable with
dict(7)


Perhaps, before I dive into thinking I'm fixing things,
someone would be kind enough to look into the above?
ak

Re: [9fans] mkindex of dict(7)

[erik quanstrom]

> /sys/src/cmd/dict/dict.c states:
> /*
>  * Assumed index file structure: lines of form
>  * 	[^\t]+\t[0-9]+
>  * First field is key, second is byte offset into dictionary.
>  * Should be sorted with args -u -t'	' +0f -1 +0 -1 +1n -2
>  */
>
> whereas, /sys/src/cmd/dict/mkindex outputs:
> <byte offset>	<key>
> i.e.,
> 0	ヽ [くりかえし]
>
> Perhaps, before I dive into thinking I'm fixing things,
> someone would be kind enough to look into the above?
> ak

clearly there was some post processing.
why do you need to regenerate the index?

- erik

Re: [9fans] mkindex of dict(7)

[erik quanstrom]

> learly there was some post processing.
> why do you need to regenerate the index?

garr.  reply to myself.

there are a number of awk and rc scripts in
the directory.  it would be a good quick
project to put the pieces together in the
mkfile, but i think some of the pieces are
missing.  the wouldn't be hard to recreate.

running pgwindexraw through
	awk -F^'	' -f canonind.awk
seems pretty reasonable, but there was different
processing applied before the sort to remove
the syllable markers and the leading ||.

- erik

Re: [9fans] mkindex of dict(7)

[Akshat Kumar]

Regarding the dict index files, what I understand is that dict(7)
receives a pattern (may also be a byte offset or whatever, but suppose
pattern), looks it up in the first fields of the lines in the dict
index, and uses the corresponding byte offset in the index to find the
full line in the dict file.  Well, I've been trying to make the EDICT
dictionary[1] usable with dict(7), using just the "simple" dict scheme
as described in /sys/src/cmd/dict/simple.c, and have made (for now) a
	<kanji>	<byte offset>
index file from the output of mkindex (piping through to sed and
switching the order of the kanji and byte offset).  I've tried quite a
few ways of making that index file, but have yet not succeeded in
getting dict(7) to actually find a corresponding line in the dict file
(`pattern not found'), given any kanji in the first fields in the
index file as a pattern.

I cannot attach the index file nor the dictionary file with this
E-Mail, since both are too big -- though I've put them online[2] --
but the dictionary file made available at [1] is in a slightly
different format (inserted tab after each kanji/kana) and charset
(EUC-JP/JIS X 0208 → UTF-8) than I have converted at [2].  If anyone
is willing to help figure this out, I'd be very grateful.


[1] http://www.csse.monash.edu.au/~jwb/edict_doc.html
	(see FORMAT for default formatting, and CURRENT VERSION & DOWNLOAD
	 to grab edict.gz)
[2] http://sounine.nanosouffle.net/magic/webls?dir=/comp/dict


Please alert me if the information here is insufficient --
I also don't mind if you go ahead and make the dict files yourself...
just let me in on it --
ak

Re: [9fans] mkindex of dict(7)

[Fazlul Shahriar]

Hi,

You need to sort your index file. Looks like dict(7) is doing binary
search on it. After sorted, it works fine.

fhs

On Tue, Jan 6, 2009 at 12:34 AM, Akshat Kumar
<akumar@sounine.nanosouffle.net> wrote:
> Regarding the dict index files, what I understand is that dict(7)
> receives a pattern (may also be a byte offset or whatever, but suppose
> pattern), looks it up in the first fields of the lines in the dict
> index, and uses the corresponding byte offset in the index to find the
> full line in the dict file.  Well, I've been trying to make the EDICT
> dictionary[1] usable with dict(7), using just the "simple" dict scheme
> as described in /sys/src/cmd/dict/simple.c, and have made (for now) a
>        <kanji> <byte offset>
> index file from the output of mkindex (piping through to sed and
> switching the order of the kanji and byte offset).  I've tried quite a
> few ways of making that index file, but have yet not succeeded in
> getting dict(7) to actually find a corresponding line in the dict file
> (`pattern not found'), given any kanji in the first fields in the
> index file as a pattern.
>
> I cannot attach the index file nor the dictionary file with this
> E-Mail, since both are too big -- though I've put them online[2] --
> but the dictionary file made available at [1] is in a slightly
> different format (inserted tab after each kanji/kana) and charset
> (EUC-JP/JIS X 0208 → UTF-8) than I have converted at [2].  If anyone
> is willing to help figure this out, I'd be very grateful.
>
>
> [1] http://www.csse.monash.edu.au/~jwb/edict_doc.html
>        (see FORMAT for default formatting, and CURRENT VERSION & DOWNLOAD
>         to grab edict.gz)
> [2] http://sounine.nanosouffle.net/magic/webls?dir=/comp/dict
>
>
> Please alert me if the information here is insufficient --
> I also don't mind if you go ahead and make the dict files yourself...
> just let me in on it --
> ak

Re: [9fans] mkindex of dict(7)

[Akshat Kumar]

...
> You need to sort your index file.

Aha! This was exactly the problem!

> Looks like dict(7) is doing binary
> search on it. After sorted, it works fine.
>

Indeed.


> fhs
>
Thanks,
ak

Re: [9fans] mkindex of dict(7)

[Akshat Kumar]

/sys/src/cmd/dict/dict.c states:
/*
 * Assumed index file structure: lines of form
 * 	[^\t]+\t[0-9]+
 * First field is key, second is byte offset into dictionary.
 * Should be sorted with args -u -t'	' +0f -1 +0 -1 +1n -2
 */

whereas, /sys/src/cmd/dict/mkindex outputs:
<byte offset>	<key>
i.e.,
0	ヽ [くりかえし]
(custom dictionary  from EDICT)
or
158928	Ab*sorb`a*bil"i*ty
(pgw)

thus, the resulting index from mkindex seems to not be usable with
dict(7)


Perhaps, before I dive into thinking I'm fixing things,
someone would be kind enough to look into the above?
ak