If you fail a Quest or give up before completing it, you will forfeit all Gold, items, and Proficiency you obtained in the Quest.
However, you will retain any EXP you earned from battles up to that point. There are two ways to leave a Quest in progress:

[Failing a Quest]
If you are defeated before completing a Quest, the Quest will end in failure.
You can spend Gems to continue from where you left off.
Each Quest has a time limit.
Taking more than the allotted time to clear the Quest will also result in a Quest failure.
If the Quest's time limit runs out, you will not have the option to continue.

[Retiring From a Quest]
You can quit a Quest at any time by using the [Home] button from the Menu accessible from the bottom-left corner of the Quest screen.