[COFF] Re: converting lousy scans of pdfs into something more useable

[COFF] Re: converting lousy scans of pdfs into something more useable

[Bakul Shah]

On Feb 3, 2023, at 8:26 AM, Will Senn <will.senn@gmail.com> wrote:
> 
> I can't seem to get away from having to highlight and mark up the stuff I read. I love pdf's searchability of words, but not for quickly locating a section, or just browsing and studying them. I can flip pages much faster with paper than an ebook it seems :).

You can annotate, highlight and markup pdfs. There are apps for that though
I’m not very familiar with them as I don’t markup even paper copies. On an
iPad you can easily annotate pdfs with an apple pencil. 

[COFF] Re: converting lousy scans of pdfs into something more useable

[Ralph Corderoy]

> https://decuser.github.io/pdfs/2023/02/01/pdf-cleanup-workflow.html

units(1) can do the sum for you.

    $ units -1 475point in
	    * 6.5972222

Be aware there are several kinds of ‘point’; search
/usr/share/units/definitions.units for /^point to get to the relevant
area.

It will also do your simple ‘echo ... | bc -l’ sums.

    $ units 2560/96
            Definition: 26.666667

GraphicsMagick, gm(1), may be more consistent and future-proof than
ImageMagick.

> I'd really like to get away from needing Photoscape X, though.
> Then I could entirely automate the workflow in bash...

gmic(1) is a very powerful and little known image-processing program.
It may help dump PhotoScape X.

-- 
Cheers, Ralph.

[COFF] Re: converting lousy scans of pdfs into something more useable

[Will Senn]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

On 2/3/23 10:01 AM, Bakul Shah wrote:
>
> https://github.com/ocrmypdf/OCRmyPDF
>
> It's a python script that runs most any unix and uses
> tesseract. Its author's motivation seems similar to yours:
>
> I searched the web for a free command line tool to OCR PDF files: I found many, but none of them were really satisfying:
>      • Either they produced PDF files with misplaced text under the image (making copy/paste impossible)
>      • Or they did not handle accents and multilingual characters
>      • Or they changed the resolution of the embedded images
>      • Or they generated ridiculously large PDF files
>      • Or they crashed when trying to OCR
>      • Or they did not produce valid PDF files
>      • On top of that none of them produced PDF/A files (format dedicated for long time storage)
> ...so I decided to develop my own tool.

Nice. Off to checking out OCRmyPDF!

> I rarely print PDFs any more.

I can't seem to get away from having to highlight and mark up the stuff 
I read. I love pdf's searchability of words, but not for quickly 
locating a section, or just browsing and studying them. I can flip pages 
much faster with paper than an ebook it seems :).

-will

[-- Attachment #2: Type: text/html, Size: 1916 bytes --]

[COFF] Re: converting lousy scans of pdfs into something more, useable

[Will Senn]

> From: Dennis Boone <drb@msu.edu>
>
> * Don't use JPEG 2000 and similar compression algorithms that try to
>    re-use blocks of pixels from elsewhere in the document -- too many
>    errors, and they're errors of the sort that can be critical.  Even if
>    the replacements use the correct code point, they're distracting as
>    hell in a different font, size, etc.
I wondered about why certain images were the way they were, this 
probably explains a lot.

> * OCR-under is good.  I use `ocrmypdf`, which uses the Tesseract engine.
Thanks for the tips.
> * Bookmarks for pages / table of contents entries / etc are mandatory.
>    Very few things make a scanned-doc PDF less useful than not being able
>    to skip directly to a document indicated page.
I wish. This is a tough one. I generally sacrifice ditching the 
bookmarks to make a better pdf. I need to look into extracting bookmarks 
and if they can be re-added without getting all wonky.

> * I like to see at least 300 dpi.
Yes, me too, but I've found that this often results in too big (when 
fixing existing), if I'm creating, they're fine.

> * Don't scan in color mode if the source material isn't color.  Grey
>    scale or even "line art" works fine in most cases.  Using one pixel
>    means you can use G4 compression for colorless pages.

Amen :).
>
> * Do reduce the color depth of pages that do contain color if you can.
>    The resulting PDF can contain a mix of image types.  I've worked with
>    documents that did use color where four or eight colors were enough,
>    and the whole document could be mapped to them.  With care, you _can_
>    force the scans down to two or three bits per pixel.
> * Do insert sensible metadata.
>
> * Do try to square up the inevitably crooked scans, clean up major
>    floobydust and whatever crud around the edges isn't part of the paper,
>    etc.  Besides making the result more readable, it'll help the OCR.  I
>    never have any luck with automated page orientation tooling for some
>    reason, so end up just doing this with Gimp.
Great points. Thanks.

-will

[COFF] Re: converting lousy scans of pdfs into something more useable

[Bakul Shah]

On Feb 3, 2023, at 7:27 AM, Will Senn <will.senn@gmail.com> wrote:
> 
> what's your experience with using sad pdfs? Do you just live with them as they are, or do you try to fix them and how, or do you use a workflow and get good results?

Usually I just live with them but I may use "ocrmypdf" if search
or copy-paste is unsatisfactory.

https://github.com/ocrmypdf/OCRmyPDF

It's a python script that runs most any unix and uses
tesseract. Its author's motivation seems similar to yours:

I searched the web for a free command line tool to OCR PDF files: I found many, but none of them were really satisfying:
    • Either they produced PDF files with misplaced text under the image (making copy/paste impossible)
    • Or they did not handle accents and multilingual characters
    • Or they changed the resolution of the embedded images
    • Or they generated ridiculously large PDF files
    • Or they crashed when trying to OCR
    • Or they did not produce valid PDF files
    • On top of that none of them produced PDF/A files (format dedicated for long time storage)
...so I decided to develop my own tool.

I rarely print PDFs any more.

[COFF] Re: converting lousy scans of pdfs into something more useable

[Dennis Boone]

 > I read a tremendous number of documents from the web, or at least
 > read parts of them - to the tune of maybe 50 or so a week. It is
 > appalling to me in this era that we can't get better at scanning. Be
 > that as it may, the needle doesn't seem to have moved appreciably in
 > the last decade or so and it's a little sad. Sure, if folks print to
 > pdf, it's great. But, if they scan a doc, not so great, even today.

I see a fair number of frustrating scanned-doc PDFs too.  My thoughts on
what constitutes a decent scan:

* Assume people will print at least a few pages occasionally.  It's
  often easier to print that one table or diagram and take it to the
  bench than to try to use a tablet or run back and forth to a PC.  That
  affects how you think about creating the PDF.

* Don't use JPEG 2000 and similar compression algorithms that try to
  re-use blocks of pixels from elsewhere in the document -- too many
  errors, and they're errors of the sort that can be critical.  Even if
  the replacements use the correct code point, they're distracting as
  hell in a different font, size, etc.

* OCR-under is good.  I use `ocrmypdf`, which uses the Tesseract engine.

* I do get angry when I see people trying to reconstruct the document
  via OCR and omitting the actual scan -- too many errors.

* Bookmarks for pages / table of contents entries / etc are mandatory.
  Very few things make a scanned-doc PDF less useful than not being able
  to skip directly to a document indicated page.

* I like to see at least 300 dpi.

* Don't scan in color mode if the source material isn't color.  Grey
  scale or even "line art" works fine in most cases.  Using one pixel
  means you can use G4 compression for colorless pages.

* Do reduce the color depth of pages that do contain color if you can.
  The resulting PDF can contain a mix of image types.  I've worked with
  documents that did use color where four or eight colors were enough,
  and the whole document could be mapped to them.  With care, you _can_
  force the scans down to two or three bits per pixel.

* Do insert sensible metadata.

* Do try to square up the inevitably crooked scans, clean up major
  floobydust and whatever crud around the edges isn't part of the paper,
  etc.  Besides making the result more readable, it'll help the OCR.  I
  never have any luck with automated page orientation tooling for some
  reason, so end up just doing this with Gimp.

Tuppence.

De