Can Gandhiji lie and can Pinocchio speak the truth?
Let’s put jev to the test.
Here is a question for jev:
Alan told Bob that he had lied about Charlie being ill because he had seen him playing football that morning. Who is the liar?
Read that again carefully.
It must be obvious to you as a human that Bob is the liar.
Is jev able to figure it out?
We can run this through jev and check the probabilities and it does a pretty good job.
But what if I now replace Alan with Pinocchio (a name that LLMs have learnt to associate with lying) and Bob with Mahatma Gandhi.
The question now reads: Pinocchio told Mahatma Gandhi that he had lied about Charlie being ill because he had seen him playing football that morning. Who is the liar?
Surprise, surprise this confuses jev who immediately calls out Pinocchio as the liar and deems Mahatma Gandhi more truthful than Pinocchio.
Raises all kinds of questions about the latent bias of LLMs and neutralizing that.
In any case jev is not ready for production. At least when it comes to Winograd Schema Challenges.