You can use one Gem to fully recover all of your AP if you do not have enough to embark on a Quest.