Hint: In stable equilibrium, the forces acting on the body try to bring the body in the original position when the body is slightly displaced.
Step 1: Find the net force on the object.
Let the mass and radius of each identical heavy sphere be M and R respectively. An object of mass m is placed at the mid-point P of the line joining their centres.
Force acting on the object placed at the mid-point,
F1=F2=GMm(5R)2
The direction of forces are opposite, therefore the net force acting on the object is zero.
Step 2: Find the net force on the object again when the object is displaced.
To check the stability of the equilibrium, we displace the object through a small distance x towards sphere A.
Now, the force acting towards sphere A, F'1=GMm(5R−x)2
Force acting towards sphere B, F2'=GMm(5R+x)2
As F1'>F2', therefore a resultant force (F1'−F2') acts on the object towards sphere A, therefore the object starts to move towards sphere A and hence equilibrium is unstable.