Yep, a refresh does not mean that you have to start analyzing files all over, only newly arrived/changed or removed files are affected.
It would of course be great if such changes could be picked up automatically, but that's not necessarily a trivial thing to implement.