Non-military contracted labor built the Death Star

The Empire was in fact notorious for using slave labor. Most commonly the slaves would be conscripted from the 'lesser' species of nonhumans. Raids would be made against the homeworlds of some of these species, especially Kashyyyk, home planet of the Wookiees. Sometimes, though, humans, former citizens of the Empire, would be imprisoned for their political views and forced into slavery as well.

Actually, the construction of the Death Star Mk 1 is mentioned in one of the Jedi Academy trilogy books by Kevin J. Anderson, which is only "extended universe" canon, but there is no contradicting movie authority to suggest it is incorrect. In fact, the Death Star was constructed in orbit of the penal world Despayre, using the prisoners as slave labourers. When the construction was completed, the planet's atmosphere was so horribly damaged by the industry involved in the construction effort, that it was no longer of any use. As such, Emperor Palpatine ordered the use of Despayre as a test of the Death Star's superlaser, with spectacular results.

Admittedly this doesn't prove that the same source of labour was used in the Death Star Mk 2, but I'd think that slave labour was being used again. Since the imperial fleet are clearly in control of construction (the admiral on the death star is held responsible by Darth Vader for the slow progress of construction), it presumably isn't being constructed by Seinar or any of the other major contractors, who typically managed construction work themselves. Also, if you were building a superweapon, being built as a successor to one destroyed largely due to espionage, would you bring in unknown civilians to do the job?

Therefore, sure, the work crews were "innocent" and working against their will, but most of them are criminals anyway, so they have it coming....