Fwd: Re: RESEND: syntax-highlighting.py

Fwd: Re: RESEND: syntax-highlighting.py

[stefan]

Am 08.01.2014 16:19, schrieb Jason A. Donenfeld:
> Okay reading this closer, it seems what the one in-tree could benefit from is:
> 
> - Expanded list of filename mappings, made more generic than what you
> have in your script, but basically the same idea. -- { "pkgbuild":
> "bashlexer", "cmakelists.txt", "cmakelexer" }. Is there a way to do
> this that ties directly into pygment's guess_lexer_for_filename? If
> not, could you submit a patch upstream?

I have looked in pygment's sourcecode and found this [0]. I just have to
add the filenames in the lexer definitions. I will try to create an
upstream pull request. The only problem is that the latest release of
pygments is almost 1 year old. It think it will take a while until the
changes get merged...

As a workaround we could create such a dict as you suggested and check
if the filenames match.

> - We should make use of guess_lexer as you do, but perhaps not only if
> the shebang is present

At first I run guess_lexer_for_filename (line 26). It checks the content
of the file and also looks at the filename. After that I check if there
is a shebang because the script has already looked at the content with
guess_lexer_for_filename. I have tested this without the shebang
detection and especially with plaintext files it often returns crap...

> Would love for a patch against the in-tree version that gets it up to
> date like this.

:-)
It will take a few days because currently I have exams... Should I work
on the master branch or something else?

What do you want to de with the CSS definitions? Just put them into the
default cgit CSS file or maybe include it seperately?

Stefan

Links:
[0]:
https://bitbucket.org/birkenfeld/pygments-main/src/9246d7a47a42e7920c0ecee4d4946b982e9e2430/pygments/lexers/_mapping.py?at=default





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140108/d242b74e/attachment.asc>

RESEND: syntax-highlighting.py

[Jason]

On Wed, Jan 8, 2014 at 5:22 PM, Stefan Tatschner <stefan at sevenbyte.org> wrote:
> I have looked in pygment's sourcecode and found this [0]. I just have to
> add the filenames in the lexer definitions. I will try to create an
> upstream pull request. The only problem is that the latest release of
> pygments is almost 1 year old. It think it will take a while until the
> changes get merged...
>
> As a workaround we could create such a dict as you suggested and check
> if the filenames match.

Another workaround might be just to monkey patch _mapping.LEXERS. Any
of these possibilities is probably fine.


> At first I run guess_lexer_for_filename (line 26). It checks the content
> of the file and also looks at the filename. After that I check if there
> is a shebang because the script has already looked at the content with
> guess_lexer_for_filename. I have tested this without the shebang
> detection and especially with plaintext files it often returns crap...

Okay fair enough. Shebang guessing only.


> It will take a few days because currently I have exams... Should I work
> on the master branch or something else?

Yes the master branch.

>
> What do you want to de with the CSS definitions? Just put them into the
> default cgit CSS file or maybe include it seperately?

We already handle that on line 34 of [1], so no need to do anything
differently. Please base your work off [1] generally.


Thanks a bunch! Looking forward to seeing the results.

Jason

[1] http://git.zx2c4.com/cgit/tree/filters/syntax-highlighting.py

RESEND: syntax-highlighting.py

[stefan]

Am 08.01.2014 17:28, schrieb Jason A. Donenfeld:
>> What do you want to de with the CSS definitions? Just put them into the
>> default cgit CSS file or maybe include it seperately?
> 
> We already handle that on line 34 of [1], so no need to do anything
> differently. Please base your work off [1] generally.
>
> [1] http://git.zx2c4.com/cgit/tree/filters/syntax-highlighting.py

Actually I do not like this implementation of CSS output in the script.
In the final HTML output there is something like this:

<td class='lines'><pre><code><style>.highlight .hll { background-color:
#ffffcc }
.highlight  { background: #ffffff; }
.highlight .c { color: #888888 } /* Comment */
.highlight .err { color: #a61717; background-color: #e3d2d2 }

[...]

</style><div class="highlight"><pre>

You see the CSS definitions for the synthax highlighting thing come
within a table within a pre within a code tag. I do not think this is
the right place for CSS definitions.

What do you think?

Stefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140109/f76b1af2/attachment.asc>

RESEND: syntax-highlighting.py

[Jason]

On Fri, Jan 10, 2014 at 2:07 PM, Stefan Tatschner <stefan at sevenbyte.org> wrote:
> I have created an upstream pull request and it got merged after 30
> seconds [1]. Moreover it seems like the cmakelists.txt thing was also
> fixed [2].

Nice! Great!

>
> Should I remove the filename mapping thing in my patch according to
> upstream or should I keep this until the next pygments release?

Yea, go ahead and remove that, and resubmit. When pygments releases,
it'll automatically have that improved functionality.

RESEND: syntax-highlighting.py

[stefan]

Am 08.01.2014 16:19, schrieb Jason A. Donenfeld:
> Okay reading this closer, it seems what the one in-tree could benefit from is:
> 
> - Expanded list of filename mappings, made more generic than what you
> have in your script, but basically the same idea. -- { "pkgbuild":
> "bashlexer", "cmakelists.txt", "cmakelexer" }. Is there a way to do
> this that ties directly into pygment's guess_lexer_for_filename? If
> not, could you submit a patch upstream?

I have created an upstream pull request and it got merged after 30
seconds [1]. Moreover it seems like the cmakelists.txt thing was also
fixed [2].

Should I remove the filename mapping thing in my patch according to
upstream or should I keep this until the next pygments release?


[1]
https://bitbucket.org/birkenfeld/pygments-main/commits/7cc1e6d0143445302445360fb8b963a7644f1912
[2]
https://bitbucket.org/birkenfeld/pygments-main/src/8fda165f5da04bdf5040fc7b8d3e1589d8fc1b4f/pygments/lexers/text.py?at=default#cl-1609

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140110/eb45187a/attachment.asc>

RESEND: syntax-highlighting.py

[Jason]

Hi Stefan,

We ship with this script already:
http://git.zx2c4.com/cgit/tree/filters/syntax-highlighting.py

It's pretty similar. Yours might have some fixes or bells and whistles
that mine does not? If so, would you mind submitting a patch that
improves the one we already have?

Thanks,
Jason

RESEND: syntax-highlighting.py

[stefan]

Am 04.11.2013 21:46, schrieb Stefan Tatschner:
Hi there,
I created a python script for the syntax-highlighting thing in cgit
which solves a few problems with the shipped script in cgit [1]. I use
it in production since half a year.

- It uses python2 because there are still a lot of problems with
  python3 and pygments out there. [2], [3]
- I used guess_lexer_for_filename for a better detecting of the used
  languages (e.g. mixed HTML and PHP, HTML (Django)...).
  guess_lexer_for_filename does not work with python3 yet.
- The script looks whether there is a shebang line if it cannot
  determine the language.
- It maps cmakelists.txt and pkgbuild (arch linux package system) files
  to the correct lexer. Any other filenames can be added easily.
- The CSS has to be defined in the CSS file to avoid inline CSS.
- `sys.stdin.read().decode(encoding='utf-8', errors='ignore')` is used
  to ensure the functionality even for corrupted file encodings.

If you like my implementation I would be happy to create a patchfile. :)

Stefan

Links:
[1]:
http://cgit.sevenbyte.org/cgit_stuff/tree/cgit_pygments.py?id=4f847ddcd5aa7de0948bdb6b200a966b6389d94a
[2]:
https://bitbucket.org/birkenfeld/pygments-main/issue/901/problems-with-python3
[3]:
https://bitbucket.org/birkenfeld/pygments-main/issue/847/test-failures-with-python-33


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140108/8f6e0a2b/attachment.asc>