Long texts are not fully fed into Research assistant

httpjames

I uploaded a long text file with about 80k words but only 2.3k of them were read and passed to the LLM.

Upload a long text file to Research assistant
After it processes the file, look at the word count
Observe it only being around 2k, even though the underlying model supports 200k tokens (which should be about 150k

I expected the entire file to passed into Research assistant, not just a small excerpt. If this is a cost saving measure to prevent expensive prompts, then it should be clearly labeled to the user instead of an automatic truncation.

Vlad

httpjames Word count should not matter ultimately, but whether the assistant performed the task or not?

ak42

httpjames
even if you provided it with the full text, it would not work. As of today, LLMs have that "Lost int the middle" limitation. (and also, they are not intelligent 😅)
RAGs and vector databases are there to help.
As of today, no LLM will be able to accurately reply to "what themes are presented in this text?" for a huge chunk of text. You need to rephrase your question and help the RAG extract the relevant data for you.

https://arxiv.org/pdf/2307.03172.pdf

httpjames

Vlad It didn't. It only got the limited context from the beginning 2k words, so it couldn't analyze the full text to extract detailed information. For example, it couldn't accurately answer "what themes are presented in this text?" or come up with wider implications using the contents.

Vlad

httpjames I think you should rephrase your bug report as such as any technique used ultiamtely doe snot matter - what does is whether task was performed or not.

httpjames

Vlad But the LLM must have full context of the attachment for many non-RAG tasks. As far as I understand it currently, the text is being truncated, so the majority of the text isn't being provided in the first place. The current mechanism for discussing long files isn't transparent enough.

Vlad

httpjames Again I do not think the process needs to be transparent as log as the result is there (we would like to produce answers even for 1M token document) so I thin this should be phrased as a bug with a concrete example. not discuss any method we use.

httpjames

ak42 YouChat provides the full text to its LLMs and they're all able to answer my query. The lost in the middle problem is well known, but it not so bad to the point where it cripples high level questions. Anthropic has done a lot of work and research to mitigate the issue in their models, especially in the Claude 3 family.

Vlad

httpjames Still waiting for a reproducible example

httpjames

Vlad This isn't the same text as I tried when making this bug report, but I got a similar result.

I fed in Bill C-27's full text, which is an extremely long document, but still processable by Claude's context window (https://www.parl.ca/Content/Bills/441/Government/C-27/C-27_1/C-27_1.PDF). I asked about something at the very end to test whether it was given this context, but it provided an unrelated answer.

The real passage: