Adultery is defined as sexual relations between a married person and someone who's not his/her spouse. As Webster 1913 points out, when neither party is married to anyone, the deed is instead called "fornication".
This is rarely an important distiction, except when arguing about the Bible and the morality of premarital sex. And even then you'd better make sure you have the most accurate translation available.