Use groq LLM provider

Browsing6853

The llama3-70b should have a quality similar or slightly supperior to claude3-haiku but is cheaper and a lot faster when using the API from groq. (+300t/s there)

https://wow.groq.com/

Groq doesn't support the new Gemma2-27b model yet, which should be at a similar level to llama-70b (or maybe better) and be even faster.

It's possible to try for free at groq.com, I think it's quite insane to have this speed at the instant answers.

Vlad

Browsing6853 We really wanted this to work, but in our testing llama3-70b is worse than claude 3 haiku, and does not follow instructions for summarization tasks we need.

ntt

llama3-70b is better for English but claude3-haiku is better for other languages.

adisibio

I also find LLama 70B worse, not to mention the limited context length.

Browsing6853

llama 3.1-70b should definitely be better now. Maybe you should even consider the 400b model depending on it's speed (I bet the cost will be pretty good)

dune

++ to the ask of adding support for llama 3.1-70b 🙏