Yes, if you start an exam but do not want to submit it, you can discard it. A discarded exam does not affect the scores on your skill profile. The system does not save your progress if you want to start the exam again later.
Within the exam, you can click the Discard Exam button that appears on the right. A window appears where you confirm that you want to discard the exam.