Transcribing text from image needs work

RHvdG

Transcribing text from an image, like a scan from a book without (usable) OCR, is hit and miss. The plain text is usually parsed fine, but recognition of italicised parts tends to fail. In my case, the output has to be LaTeX; the same goes, of course, when Markdown is wanted. I used Perplexity for this before, and it was a bit better at it.

Ideally, you'd paste in a fresh screenshot into the query field followed by 'latex this' or 'markdown this' as a prompt, and the bot would do the rest.

Vlad

RHvdG Steps to reproduce the problem?

mullinggroove

When you upload an image to Kagi Assistant, GPT-4 Vision API is used to analyze it. It is not an OCR model and thus worse than legacy OCR models like Google Cloud Vision OCR. From an article on OCR via GPT-4V:

In general, despite its versatility in handling diverse OCR tasks, GPT-4V does not outperform existing state-of-the-art OCR models.

Vlad

mullinggroove We'd love steps to reproduce the problem you are describing, including example images.

RHvdG

OK. Take this screenshot.

I upload it in Research Assistant (Expert) (for some reason after pasting it into the field I have to click Upload twice). The AI gives me an English summary that I have no need for. I have to wait for it to finish before I can enter my prompt, fine-tuned over time, like this: "Convert the text to LaTeX, NEVER using code like \"a, but ONLY UTF-8, and NEVER changing, adding, removing any words and NEVER changing any spelling, NEVER substituting 'ss' for 'ß', ALWAYS transcribing italics as \emph{}, but taking care NOT to interpret non-italic text parts as italic, and removing hyphenation, footnote markers, line breaks, page numbers, and incomplete sentences at the beginning or end of the text."

The result:

None of the parts in italics is parsed as such. If I then ask to try again, picking up the italics, some, but not nearly all italic words are recognised. The leaving out of incomplete sentences fails, too.

Repeatedly asking to do better with the italics usually leads to more words being coded as such, but some will still be missing, and false positives will appear as well.

I hope this helps. I really want to use Kagi for this, too.

Vlad

RHvdG Do you get different results when using gpt-4o directly with openai?

RHvdG

Let me add that sometimes, for no apparent reason, the parsing fails altogether.

RHvdG

Vlad I have no account there. I do get better results with Perplexity.

Luis

Hey @RHvdG we've made several changes since this was first reported. Are you still experiencing any issues or noticing any problems?