[magic sv logging]

[magic sv logging]

[Sergiusz Pawlowicz]

Hi,
I am trying to write a python script which can be put in place of
svlogd/multilog, but I am stuck just at the beginning.

The test script is pretty simple, it just forwards stdin to stdout:

def main():

    while True:
        try:
            line = raw_input()
            print 'got', line
        except EOFError:
            print 'done'
            pass

  if __name__ == '__main__':
    main()

It works, but not as desired, as the script is being restarted by runsv
constantly, it seems after it finishes reading stdin. Could you
please help me how can run that continuously listening to stdin?

cheers,
Serge
-- 
https://pawlowicz.name/

Re: [magic sv logging]

[Mike Buland]

[-- Attachment #1: Type: text/plain, Size: 1796 bytes --]

I took a different approach and monitored the log output of svlogd, since I
wanted the logs anyway.  This is some code that well help you with that,
anyway:

import os
import time
import stat

fIn = open("current", "r")
sid = os.stat("current")[stat.ST_INO]

while True:
        try:
                sidNew = os.stat("current")[stat.ST_INO]
                if sidNew != sid:
                        fIn.close()
                        fIn = open("current", "r")
                        sid = sidNew
        except:
                time.sleep( 1 )
                continue

        while True:
                l = fIn.readline()
                if l is None or l == '':
                        break
                # do something interesting with your line here
                time.sleep( 1 )

This will read everything from current, and when current moves, it will
read what's left, open the new current, and read all of it.  I use it to
monitor apache log output, and it hasn't failed me yet.

--Mike
On Thu, Sep 20, 2012 at 5:35 PM, Sergiusz Pawlowicz <
sergiusz.pawlowicz@gmail.com> wrote:

> Hi,
> I am trying to write a python script which can be put in place of
> svlogd/multilog, but I am stuck just at the beginning.
>
> The test script is pretty simple, it just forwards stdin to stdout:
>
> def main():
>
>     while True:
>         try:
>             line = raw_input()
>             print 'got', line
>         except EOFError:
>             print 'done'
>             pass
>
>   if __name__ == '__main__':
>     main()
>
> It works, but not as desired, as the script is being restarted by runsv
> constantly, it seems after it finishes reading stdin. Could you
> please help me how can run that continuously listening to stdin?
>
> cheers,
> Serge
> --
> https://pawlowicz.name/
>

Re: [magic sv logging]

[Lorenzo Beretta]

On 21/09/2012 04:25, Mike Buland wrote:
> I took a different approach and monitored the log output of svlogd, since I
> wanted the logs anyway.  This is some code that well help you with that,
> anyway:
>
> import os
> import time
> import stat
>
> fIn = open("current", "r")
> sid = os.stat("current")[stat.ST_INO]
>
> while True:
>          try:
>                  sidNew = os.stat("current")[stat.ST_INO]
>                  if sidNew != sid:
>                          fIn.close()
>                          fIn = open("current", "r")
>                          sid = sidNew
>          except:
>                  time.sleep( 1 )
>                  continue
>
>          while True:
>                  l = fIn.readline()
>                  if l is None or l == '':
>                          break
>                  # do something interesting with your line here
>                  time.sleep( 1 )
>
> This will read everything from current, and when current moves, it will
> read what's left, open the new current, and read all of it.  I use it to
> monitor apache log output, and it hasn't failed me yet.
>
> --Mike

Unless I missed something, it looks as if you're reimplementing the '-F' 
flag in tail :)

Re: [magic sv logging]

[Mike Buland]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

You haven't missed anything, well, it's more like "tail --follow=name
<file>' but instead of piping the output of tail into another program for
processing, this just does all the work in python.  Not only was it fun to
write, but I like the fact that I don't need to chain more programs
together.

--Mike

On Fri, Sep 21, 2012 at 9:29 AM, Lorenzo Beretta <lory.fulgi@infinito.it>wrote:

> On 21/09/2012 04:25, Mike Buland wrote:
>
>> I took a different approach and monitored the log output of svlogd, since
>> I
>> wanted the logs anyway.  This is some code that well help you with that,
>> anyway:
>>
>> import os
>> import time
>> import stat
>>
>> fIn = open("current", "r")
>> sid = os.stat("current")[stat.ST_**INO]
>>
>> while True:
>>          try:
>>                  sidNew = os.stat("current")[stat.ST_**INO]
>>                  if sidNew != sid:
>>                          fIn.close()
>>                          fIn = open("current", "r")
>>                          sid = sidNew
>>          except:
>>                  time.sleep( 1 )
>>                  continue
>>
>>          while True:
>>                  l = fIn.readline()
>>                  if l is None or l == '':
>>                          break
>>                  # do something interesting with your line here
>>                  time.sleep( 1 )
>>
>> This will read everything from current, and when current moves, it will
>> read what's left, open the new current, and read all of it.  I use it to
>> monitor apache log output, and it hasn't failed me yet.
>>
>> --Mike
>>
>
> Unless I missed something, it looks as if you're reimplementing the '-F'
> flag in tail :)
>
>
>

Re: [magic sv logging]

[Jason Dusek]

2012/9/21 Mike Buland <xagafinelle@gmail.com>:
> You haven't missed anything, well, it's more like "tail --follow=name
> <file>' but instead of piping the output of tail into another program for
> processing, this just does all the work in python.  Not only was it fun to
> write, but I like the fact that I don't need to chain more programs
> together.

Surely you could start `tail' with the subprocess module and
process its output line-by-line?

--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B

Re: [magic sv logging]

[Mike Buland]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

yeah, I could have done that too, but...I'm happy, and so is Sergiusz.  I
guess I don't see the problem.

I was mistaken, this is very much like tail -F, I just didn't read that
very carefully.  The biggest difference is that this reads the entire file
initially instead of starting near the end. (yes, you could easily do that
with tail too).

On Fri, Sep 21, 2012 at 9:35 AM, Jason Dusek <jason.dusek@gmail.com> wrote:

> 2012/9/21 Mike Buland <xagafinelle@gmail.com>:
> > You haven't missed anything, well, it's more like "tail --follow=name
> > <file>' but instead of piping the output of tail into another program for
> > processing, this just does all the work in python.  Not only was it fun
> to
> > write, but I like the fact that I don't need to chain more programs
> > together.
>
> Surely you could start `tail' with the subprocess module and
> process its output line-by-line?
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
>

Re: [magic sv logging]

[Laurent Bercot]

> You haven't missed anything, well, it's more like "tail --follow=name
> <file>' but instead of piping the output of tail into another program for
> processing, this just does all the work in python.  Not only was it fun to
> write, but I like the fact that I don't need to chain more programs
> together.

 So, instead of a relatively small processor, you fire up a big, powerful
and oversized-for-your-needs Python interpreter. ;)

-- 
 Laurent

Re: [magic sv logging]

[Jason Dusek]

2012/9/22 Laurent Bercot <ska-supervision@skarnet.org>:
>> You haven't missed anything, well, it's more like "tail --follow=name
>> <file>' but instead of piping the output of tail into another program for
>> processing, this just does all the work in python.  Not only was it fun to
>> write, but I like the fact that I don't need to chain more programs
>> together.
>
>  So, instead of a relatively small processor, you fire up a big, powerful
> and oversized-for-your-needs Python interpreter. ;)

In this case, since further processing needs to be done with Python,
there is actually less being loaded in to memory overall.

The argument for tail is more aesthetic and maintenance oriented
(at least for me). Since I'm "doing a tail" might as well just use tail.
I happen to personally enjoy, also, programming with processes and
pipes as a way of breaking down my problems.

--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B