[1/2 OT] Indexing (and mergeable Index-algorithms)

[Oliver Bandel <oliver@first.in-berlin.de>]

Hello,


I'm looking for indexing algorithms and especially - if
such a thing exists - mergeable/extendable indexing algorithms.

So, say we have 10^6 texts that we want ot have an index for,
to retrieve the text according to some parts of the text
(keywords, substrings,...).
We want to make an index from these texts.

After a while we get 10^5 new texts and want to extend
the exisiting index, so that the whole index not necessarily
must be created again, with the indexer-tool running on
all files (^10^6 + 10^5) again, but only have to index the new files,
but the big index can be extended with additional smaller indizes.

Is there something like that already existing?
Or must the new index be created on all files again,
or must there be a workaround with the big and a small index-file,
where handling of both would be a solution we must provide by ourselfes?

It's mainly a question on datastructures/algorithms, so this mailing list
may be the wrong, but the reason to aske here is: Are functional
datastructures in some way good for implementing such tools?


BTW: Let's mention that the application I intended to write is
     performance critical.... so, if functional datastructures
     are quite good for such extendable indexes, but are too slow,
     then thsi would also be a problem.


Any hints here?
(Maybe using OCaml, but the imperative features of it would help,
 if the functional features would be too slow?)

Any hint on algorithms/datastructures for this would be fine...


Thanks In Advance,
           Oliver