Wonder no more.
I'm going to break this up for the explanation.
1. FINE - With rare exceptions, the finer a thing is the better; this is true even in material objects such as flour or cloth. So if someone asks you if such-and-such is OK and you reply it's fine (or fine by me), you're saying that it's good.
2. Frog's Hair - Have you ever seen frog's hair? NO? It's because frog's hair is so fine you can't "see" it. In other words it's so fine as to be considered non-existent. Super Fine.
So the phrase "finer than frog's hair" means something that's extraordinary.