Would you, then, advocate a world where no one is accountable for his or her actions?
The simple fact is that the universe itself has no concept of justice. If humans want justice, they cannot simply wait for the universe to dispense it; things do not work that way. A corrupt politician can commit myriad unspeakable acts, and so long as he is careful not to get caught (or silence any who do catch him) he will attain great power. A murder or rapist who doesn't get caught will inflict horrific pain on others with absolutely no consequences for himself, save perhaps the possibility of being injured in the attempt.
Are those who dispense justice really the scarier of the two? I don't think so. Far scarier is the person who believes they ought to be able to cause suffering to others with no consequence to himself, for whatever reason. Even judges or police officers face potential consequences for the "violence" they dispense: if they are found using it wrongly (be it through ignorance or malice) they can find themselves on the other side of the bench very quickly. Indeed, sometimes this happens even without any wrongdoing on the part of the judge or officer.
Self-defense is another case where the ideas you espouse break down. Are you stating that a person who is being attacked should simply bend over and take it up the ass (figuratively in most situations, of course, but my wording is intentional) simply because the other person wants something out of them? Of course not. While it is true that one person has no more right to be safe than any other, that same person also has no less right to be safe, and an aggressor forfeits their own right by infringing on another's.
It is true that, in an ideal world, there would be no need for violence of any kind. But this is hardly an ideal world. In the case of choice of food, it's a simple fact that humans must eat, and humans in a natural state (unassisted by any technology, including shipping food in from other places) must eat meat to survive. In the case of punishment versus "reasoning" with a person (which I can only assume you're advocating, though you provide absolutely no alternative for the current system in your writeup), there's the fact that many people, ranging from your average two-year-old to the psycho in the mental ward, simply cannot be reasoned with. A two-year-old hasn't yet developed the mental faculties to be effectively reasoned with, and an insane person (for whatever reason) has lost that capability. In cases such as these, an alternative must be found. And even in other cases, a person must be held accountable for his or her actions. This is what justice is meant to do. It has nothing to do with violence, except that the two are often at odds.