The opposite of 'fewer' is 'more', and the opposite of 'less' is also 'more'. This is part of the reason behind the confusion between the two words.
One way to test which word is appropriate is that something you can have 'fewer' of is something you can have 'a few' of (something quantum), something you can have 'less' of is something you can have 'a little' of (something continuous). For example, 'a little oranges' just sounds wrong - you say, 'a few oranges', so the opposite of 'more oranges' is 'fewer oranges'. Likewise, 'a few orange juice' is clearly wrong, it's 'a little orange juice', so the opposite of 'more orange juice' is 'less orange juice'.