Cool bug in URL parsing

Cool bug in URL parsing

[Karl Kleinpaste]

If you haven't stopped your setup from doing highlighting of URLs
embedded in text, here's an entertaining glitch to see.

>From the end of this line containing the sequence to start a "<URL:"
everything will be highlighted as a supposed URL until, for example,
some quoted text shows up to provide the terminator.

> Such as on this line here.

Methinks there's a regexp that gets a /little/ too aggressive...

--karl

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Karl Kleinpaste <karl@jprc.com> writes:

> If you haven't stopped your setup from doing highlighting of URLs
> embedded in text, here's an entertaining glitch to see.
> 
> >From the end of this line containing the sequence to start a "<URL:"
> everything will be highlighted as a supposed URL until, for example,
> some quoted text shows up to provide the terminator.
> 
> > Such as on this line here.
> 
> Methinks there's a regexp that gets a /little/ too aggressive...

What makes you think this is a bug?  According to rfc1738:

APPENDIX: Recommendations for URLs in Context

   URIs, including URLs, are intended to be transmitted through
   protocols which provide a context for their interpretation.

   In some cases, it will be necessary to distinguish URLs from other
   possible data structures in a syntactic structure. In this case, is
   recommended that URLs be preceeded with a prefix consisting of the
   characters "URL:". For example, this prefix may be used to
   distinguish URLs from other kinds of URIs.

   In addition, there are many occasions when URLs are included in other
   kinds of text; examples include electronic mail, USENET news
   messages, or printed on paper. In such cases, it is convenient to
   have a separate syntactic wrapper that delimits the URL and separates
   it from the rest of the text, and in particular from punctuation
   marks that might be mistaken for part of the URL. For this purpose,
   is recommended that angle brackets ("<" and ">"), along with the
   prefix "URL:", be used to delimit the boundaries of the URL.  This
   wrapper does not form part of the URL and should not be used in
   contexts in which delimiters are already specified.

   In the case where a fragment/anchor identifier is associated with a
   URL (following a "#"), the identifier would be placed within the
   brackets as well.

   In some cases, extra whitespace (spaces, linebreaks, tabs, etc.) may
   need to be added to break long URLs across lines.  The whitespace
   should be ignored when extracting the URL.

   No whitespace should be introduced after a hyphen ("-") character.
   Because some typesetters and printers may (erroneously) introduce a
   hyphen at the end of line when breaking a line, the interpreter of a
   URL containing a line break immediately after a hyphen should ignore
   all unencoded whitespace around the line break, and should be aware
   that the hyphen may or may not actually be part of the URL.

   Examples:

      Yes, Jim, I found it under <URL:ftp://info.cern.ch/pub/www/doc;
      type=d> but you can probably pick it up from <URL:ftp://ds.in
      ternic.net/rfc>.  Note the warning in <URL:http://ds.internic.
      net/instructions/overview.html#WARNING>.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
I'm a Lisp variable -- bind me!

Re: Cool bug in URL parsing

[Per Abrahamsen]

Karl Kleinpaste <karl@jprc.com> writes:

> If you haven't stopped your setup from doing highlighting of URLs
> embedded in text, here's an entertaining glitch to see.

Oh, you are on usenet-format too :-)

As Hrvoje explains, it is not a bug, but a feature which has taken
some work to implement.

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

> As Hrvoje explains, it is not a bug, but a feature which has taken
> some work to implement.

Note that I do think we should be more careful about the regexps.
Large articles take very very long to show in Gnus, and my profiling
shows that 94% of the time is lost in re-search-forward.

Constructive proposal: turn more regexp-driven code into specialized
loops.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Sauron is alive in Argentina!

Re: Cool bug in URL parsing

[Russ Allbery]

Per Abrahamsen <abraham@dina.kvl.dk> writes:
> Karl Kleinpaste <karl@jprc.com> writes:

>> If you haven't stopped your setup from doing highlighting of URLs
>> embedded in text, here's an entertaining glitch to see.

> Oh, you are on usenet-format too :-)

*laugh*

This is why I always cite the format as <URL:...>.

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>

Re: Cool bug in URL parsing

[Lars Magne Ingebrigtsen]

Hrvoje Niksic <hniksic@srce.hr> writes:

> Note that I do think we should be more careful about the regexps.
> Large articles take very very long to show in Gnus, and my profiling
> shows that 94% of the time is lost in re-search-forward.
> 
> Constructive proposal: turn more regexp-driven code into specialized
> loops.

The nice thing about regular expressions is that it is a languange
many people understand, and are willing to fiddle with.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The nice thing about regular expressions is that it is a languange
> many people understand, and are willing to fiddle with.

I don't think many people are fiddling with things like
`gnus-button-alist', or even `gnus-emphasis-alist', but I might well
be wrong.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
If anything goes wrong, blame the random-number generator!

Re: Cool bug in URL parsing

[Lars Magne Ingebrigtsen]

Hrvoje Niksic <hniksic@srce.hr> writes:

> > The nice thing about regular expressions is that it is a languange
> > many people understand, and are willing to fiddle with.
> 
> I don't think many people are fiddling with things like
> `gnus-button-alist', or even `gnus-emphasis-alist'

You're probably right there.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Hrvoje Niksic <hniksic@srce.hr> writes:
> 
> > > The nice thing about regular expressions is that it is a languange
> > > many people understand, and are willing to fiddle with.
> > 
> > I don't think many people are fiddling with things like
> > `gnus-button-alist', or even `gnus-emphasis-alist'
> 
> You're probably right there.

Would you accept faster code that buttonizes articles according to
builtin rules *other* than regexps?  What I have in mind is something
like:

(defvar gnus-emphasis-plist
        '((_ word (:repeat separator word) _) gnus-underline-face
          (* word (:repeat separator word) *) gnus-bold-face
          ...))

Code that parses forms like this into a matcher faster than hairy
regexps should not be too hard to write.

    "You have a Lisp interpreter, so use it!"   -- Jamie Zawinski

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Sauron is alive in Argentina!

Re: Cool bug in URL parsing

[Lars Magne Ingebrigtsen]

Hrvoje Niksic <hniksic@srce.hr> writes:

> Would you accept faster code that buttonizes articles according to
> builtin rules *other* than regexps?  What I have in mind is something
> like:
> 
> (defvar gnus-emphasis-plist
>         '((_ word (:repeat separator word) _) gnus-underline-face
>           (* word (:repeat separator word) *) gnus-bold-face
>           ...))

Well, `gnus-emphasis-alist' could be extended to accept non-regexp
forms.  If you write the code and documentation for that, I'll apply
the patch.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Hrvoje Niksic <hniksic@srce.hr> writes:
> 
> > Would you accept faster code that buttonizes articles according to
> > builtin rules *other* than regexps?  What I have in mind is something
> > like:
> > 
> > (defvar gnus-emphasis-plist
> >         '((_ word (:repeat separator word) _) gnus-underline-face
> >           (* word (:repeat separator word) *) gnus-bold-face
> >           ...))
> 
> Well, `gnus-emphasis-alist' could be extended to accept non-regexp
> forms.  If you write the code and documentation for that, I'll apply
> the patch.  :-)

I might give it a try, but:

* I'd like to make the mechanism a bit more general, so it covers both 
  emphasis and buttonization.  It remains to be seen if the end result 
  will actually be faster than the regexp code.

* I'm not sure I want to support the old regexp special cases of
  emphasis and buttonization.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Manic depression is cool...  your body can make its own drugs.

Re: Cool bug in URL parsing

[Dave Love]

Is it really necessary to do (more-or-less slow) buttonizing
etc. automatically for the whole of a big message rather than
restricting it to sensible-sized regions at the top and bottom where
the relevant stuff is likely to be?
.

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Dave Love <d.love@dl.ac.uk> writes:

> Is it really necessary to do (more-or-less slow) buttonizing
> etc. automatically for the whole of a big message rather than
> restricting it to sensible-sized regions at the top and bottom where
> the relevant stuff is likely to be?

I believe it is necessary to scan the whole message.  The whole point
is to have, for instance, clickable URLs anywhere in the message, even 
if it is a long FAQ.

However, the citation functions have a limit to the message length
after which they no longer attempt to do anything.  Maybe the same
variable should be extended to buttonization and emphasis functions?

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
HOW YOU CAN TELL THAT IT'S GOING TO BE A ROTTEN DAY:
        #15 Your pet rock snaps at you.

Re: Cool bug in URL parsing

[Jan Vroonhof]

Hrvoje Niksic <hniksic@srce.hr> writes:

> I believe it is necessary to scan the whole message.  The whole point
> is to have, for instance, clickable URLs anywhere in the message, even 
> if it is a long FAQ.

The solutions is of course to do it lazily, f.e. using the same technique
lazy-shot uses. I guess do need to find the time to fix up the
interface and add some general 'do-lazy' patches.

Jan

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Jan Vroonhof <vroonhof@math.ethz.ch> writes:

> Hrvoje Niksic <hniksic@srce.hr> writes:
> 
> > I believe it is necessary to scan the whole message.  The whole point
> > is to have, for instance, clickable URLs anywhere in the message, even 
> > if it is a long FAQ.
> 
> The solutions is of course to do it lazily, f.e. using the same technique
> lazy-shot uses.

Oboy.  Oboy.  Oboy.

Please, no!

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
You have an unusual magnetic personality.  Don't walk too close to
metal objects which are not fastened down.

Re: Cool bug in URL parsing

[Jari Aalto+list.ding]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

| 98-06-08 Jan Vroonhof <vroonhof@math.ethz.ch> list.ding
| Hrvoje Niksic <hniksic@srce.hr> writes:
| 
| > I believe it is necessary to scan the whole message.  The whole point
| > is to have, for instance, clickable URLs anywhere in the message, even 
| > if it is a long FAQ.
| 
| The solutions is of course to do it lazily, f.e. using the same technique
| lazy-shot uses. I guess do need to find the time to fix up the
| interface and add some general 'do-lazy' patches.

...I use TinyUrl.el which scans line for any urls when Emacs goes idle
for 2 seconds. If I know there is url in current buffer, I just let cursor
sit on the line for a while and start clicking the buttons. Buffer
length is not important with this approach and the minor mode uses
Mouse-2 as "paste/launch url" depending if the line is buttonised 
or not.

I have turned off all other package based URL handlings 
(TM/SEMI/VM/Gnus) to speed up reading.

jari

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

Re: Cool bug in URL parsing

[Jan Vroonhof]

Hrvoje Niksic <hniksic@srce.hr> writes:

> > The solutions is of course to do it lazily, f.e. using the same technique
> > lazy-shot uses.
> 
> Please, no!

? Why not? You have something that takes to long to precompute so you
do it on demand? I fail to say why buttonizing is any different from
syntax-highlighting in this regard. Or do you object to the "same
technique" part? Whats wrong with that (I did say 'after cleaning up',
which I haven't done yet since it needs more than a little work).

Jan

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Jan Vroonhof <vroonhof@math.ethz.ch> writes:

> Hrvoje Niksic <hniksic@srce.hr> writes:
> 
> > > The solutions is of course to do it lazily, f.e. using the same
> > > technique lazy-shot uses.
> > 
> > Please, no!
> 
> ? Why not?  You have something that takes to long to precompute so
> you do it on demand? I fail to say why buttonizing is any different
> from syntax-highlighting in this regard.

It is not.  I don't like on-demand fontification either, both
conceptually (it makes scrolling slower), and implementationally (none
of the current implementations are really good).  In fact, if
half-cooked on-demand buttonization were added, it would make matters
worse, because if someone argued that buttonization was slow, the
standard argument would be: "use on-demand buttonization".

To conclude, the issues of buttonization speed and on-demand
buttonization are orthogonal, and should be treated as such.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Idle RAM is the Devil's playground.

Re: Cool bug in URL parsing

[Dave Love]

>>>>> "Hrvoje" == Hrvoje Niksic <hniksic@srce.hr> writes:

 Hrvoje> Dave Love <d.love@dl.ac.uk> writes:
 >> Is it really necessary to do (more-or-less slow) buttonizing
 >> etc. automatically for the whole of a big message rather than
 >> restricting it to sensible-sized regions at the top and bottom where
 >> the relevant stuff is likely to be?

 Hrvoje> I believe it is necessary to scan the whole message.  

I wasn't proposing either preventing you doing so explicitly or
setting the limits arbitrarily large.

 Hrvoje> The whole point is to have, for instance, clickable URLs
 Hrvoje> anywhere in the message, even if it is a long FAQ.

Not for me.

I too am happy generally with, in my case, Emacs' lazy-lock but doubt
such complication is merited here.
.

Re: Cool bug in URL parsing

[Hrvoje Niksic]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "Hrvoje" == Hrvoje Niksic <hniksic@srce.hr> writes:
> 
>  Hrvoje> Dave Love <d.love@dl.ac.uk> writes:
>  >> Is it really necessary to do (more-or-less slow) buttonizing
>  >> etc. automatically for the whole of a big message rather than
>  >> restricting it to sensible-sized regions at the top and bottom where
>  >> the relevant stuff is likely to be?
> 
>  Hrvoje> I believe it is necessary to scan the whole message.  
> 
> I wasn't proposing either preventing you doing so explicitly or
> setting the limits arbitrarily large.

Of course not, but I argued two things:

1) The default should be to scan the whole message, not only the parts 
   at top and bottom;

2) The code should be optimized so that it is done fairly fast.

However, if a maximum-message-size variable should be implemented,
then yes, I agree with you that the parts at the top and the bottom
should still be buttonized.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
"Memory is like an orgasm. It's a lot better if you don't have to
fake it."  -- Seymour Cray, on virtual memory