What you want to leave your reader feeling is a cross between (a) wanting to read more and (b) being glad you haven't because any extra will have killed off the moment that this really was the end of the story.
As for characters knowing when to let go, this will involve them working out if they have achieved their goal (or as close to it as possible), whether there is anything to be gained by continuing their "quest" or whether they have had enough of it and decide to give up. The latter can be a goal in itself in a way because realising you are never going to achieve what you thought can make you re-evaluate what you want to achieve. Sometimes that is the point of the story!
A lot, of course, depends on just how important the characters' "wants" are. If in a life or death situation, they are unlikely to want to let go at all until any danger has passed. Also just how determined are your characters or are they easily swayed?