I didn't actively test anything but I noticed it in my normal daily usage. I was getting a call from an unknown number and tried to search for it. As it is from a phone number published on the internet on my city's website (which I later found out), Kagi should have given me results to that end. However, searching for the digits didn't lead to any search results at all. I tried it again after the switch back to both providers and now it's better again.
I also tried some other queries with alphanumeric strings that have a certain meaning but aren't natural language, and I felt I got overall worse results during the test. Hard to say anything really conclusively without testing them next to each other, but the phone number thing was very real.
24h is not enough to make a definitive statement about how much this bothers me. In case of bigger changes at service I usually keep with them for a while to see if they ultimately fit into with my real world needs or not. But tentatively I would put this change at a 5 to 6. That being said, if cost is an issue, and costs become too much so that Kagi cannot exist any longer, this would bother me even more, so with that trade-off in mind maybe it could go down to a 3-4.