Re: [TUHS] Steve Bellovin recounts the history of USENET

["Justin R. Andrusk" <jra@andrusk.com>]

On Fri, Nov 22, 2019 at 06:21:49PM -0500, Arthur Krewat wrote:
> 
> On 11/22/2019 3:18 PM, Justin R. Andrusk wrote:
> > I'm half tempted to take the archive.org Usenet files and throw them
> > into Elasticsearch and create a web front end for searching. Storage
> > would be expensive, but search would rock!
> 
> Can we run multiple nodes of Elastic, and replicate between each other?
> 
> I just recently started playing with it, it's quite impressive. Except
> for that one logstash file "read" mode that by default deletes the file
> once it's done with it (a 4-year-long access.log that I wanted to read in).
> 
> anyway.
> 
> art k.

Yes, that's how the clustering works with Elasticsearch. You setup
multiple nodes that are part of a cluster and data is replicated across
all of them. If one goes down, you don't lose any data as the others
will reconstitute the data. 

Going to look at adding the Usenet data to a Graylog instance as that
uses Elasticsearch as a backend and the front end UI is already there to
give you a GUI for searching and doing analytics on what you send to it.

Justin