We only get it from one source (out of about 5), one of our upstream indexes, and it is only sometimes present, even for the same search terms. I don't know why, it is probably a bug on their end or some other limitation. Nothing to do with our code, its plainly visible just testing their API directly. So yes, something is definitely "wrong", but for now we're at the limit of what we can do about it.
Also why not have IMDB as one more niche source ? I'm pretty sure they do have an API.
I get this type of question often enough that I'm going to take the time to go into some detail. I am talking about IMDB specifically here, but you can replace it with any other API you could think of us adding, the conversation is the same.
The "why not" with these types of things usually has to do with latency. In the majority of cases, we don't understand your query enough to know whether it will want IMDB results or not ahead of time. So we can perform a search, then if there were IMDB results, we make IMDB API requests to "fill in" the information.
However, if getting results took one second, and IMDB API then took another second (due to IMDB itself having a slowdown or anything in between our two infrastructures), then we've doubled your latency for a low value datapoint. Of course if we were doing this for real, we'd set upper limits so it doesn't get that bad, but as a rule of thumb you're probably at least talking about adding ~200ms anytime you stack another API call on the end, minimum, no matter who its from. Consider that we serve users from every continent, so it has to work just as well from the UK to Austrailia.
This is all fine, and we do this all the time for other similar workflows, but it has to make sense in this cost/benefit tradeoff of value, latency, complexity. Of course if the API is paid, it's now eating into our margins, so we best make sure we call it only when we need to and it was worth the user's time waiting longer for it.
The other way is if by any means we had access to IMDB's data and could host it ourselves in some form, or could otherwise build it ourselves from some back channel. Well, that will solve the latency issues (or at least put them into our own control), but that infrastructure will have some kind of operating cost, and a lot of engineering hours to set up, to import and get everything into a pipeline where our app can use it, etc.; and it will need to be maintained and kept up to date.
We only recently formed an internal search index team of 3 people (Search team is 6 people), who are all working on building a lot of this infrastructure already for broader purposes. But needless to say, getting IMDB ratings for results so we can put a few little stars on the page is not the most important thing for them to be doing at this time.
In summary: Whether we add an API, build our own index, or work with our upstream providers, it all has to fit into the cost-benefit analysis of cost/latency/complexity/value-of-the-data. And at least in this specific case, the argument is very weak for implementing IMDB's API, especially if we can just ask our upstream provider to fix it - and hopefully they can 🙂
And hopefully later this year we'll be in a much better position to start enriching search results with more of our own data at a much larger scale.
