There are two widely differing kinds of degenerate gas: Fermi and Bose.
The Fermi degenerate gas is a highly compressed quantum mechanical state. It is a gas, though it is not an ideal gas which can be thought of as tiny particles whizzing around.
What makes it a gas, then? If it were a solid, the particles would be localized. They would have local physical relations to each other which minimize their energy. A prime example is the formation of crystals. Some places have stuff, others don't. That's the lattice.
In a Fermi degenerate gas, however, compression overwhelms all of this and the stuff is squished together so tightly that it can't form any sort of structure at all. The only relevant consideration is cramming everything in. This makes these particles' states essentially independent, simply through the expedient that all states are occupied so there is no room for structure. It is this independence which characterises gases.
This is is not a stable state in the absence of pressure - some force must push the particles to take up as little space as possible. It can be approximated as an impenetrable barrier, which turns this into an elementary problem in quantum mechanics, the Infinite Square Well. The solution is that each particle spreads out over all of the space available to the entire collection. In particular, in each dimension, the wavefunctions of the energy eigenstates are sin(nπx/L), with n in the whole numbers; the energies vary as the square of n and as the inverse square of L. Each particle needs to pick a different set of three n (one in each dimension) in order to be consistent with the Pauli Exclusion Principle.
What provides the pressure here? To decrease the volume, you need to decrease the size of the box, L. But all of the particles have an energy which varies as the inverse square of L. To make the box smaller, you need to give all those particles some energy.
It may turn out that there is some size at which the system reaches equilibrium. Near this point, the energy cost to decrease L gets very high very quickly, so this gas is nearly incompressible. Note however that there is no upper limit on the density in this simple treatment. Gravity does provide an upper limit, when the collection of particles becomes a black hole.
This achieves much better packing than one can otherwise achieve otherwise (many times denser than the densest solid material), but at the cost of making each element of the gas not fit into a lower-energy structure with its neighbors. Thus, the degerate gas only occurs when the compressive force is strong enough to break down all internal structure (on the scale of the degenerate gas, that is. It is permissible for there to be smaller-scale structure like atoms existing).
Interestingly enough, it turns out that increasing the temperature in a degenerate gas does not significantly affect the pressure. The temperature is determined by the occupation of the various states, and in particular the energy at which you begin to find unoccupied states. If the gas is indeed degenerate, that means by definition that you basically don't find unoccupied states until you're really close to the chemical potential. That means that nearly nothing changes with temperature -- if temperature were a significant factor in behavior, the gas would not be fully degenerate. Yes, this is not a case of the No True Scotsman Fallacy: it must be noted that there is such a thing as 'partially degenerate'. This is of course more complicated to work with, and Fully Degenerate is just a limiting case.
The most common kind of Fermi degenerate gas is the gas of valence electrons in ordinary metals. Otherwise unbound atoms can form these gases in the cores of large gas giants and brown dwarves, and through all but the very surface of white dwarves. In neutron stars, individual nucleons are the components of the gas. In all of these, there is a gradual transition from ideal gas to strongly interacting fluid (it's beyond the critical point, so question of gas or liquid is moot) to partially degenerate to fully degenerate.
The Bose Degenerate gas is also known as a Bose-Einstein Condensate or BEC. It is completely different in every possible way (cold, low density, low pressure, bosons instead of fermions).
Basically, if a collection of bosons is made cold enough, a macroscopic number of them fall into the ground state (n =1). This newfound coherence has some odd consequences. See Bose-Einstein Condensate and superfluid.