What is a greenhouse gas?

It is any gas with 3 atoms or more in the molecule. For example CO2, water vapour, methane.

Why is it warm in a greenhouse, as opposed to a dwelling house?

Mainly because there is no wind; but there is also the effect that shorter wavelength, blueish, light sees the glass as transparent and so comes straight in; while, once absorbed and re-emitted longer wavelength reddish light cannot, as easily get back out through the glass. It is this later effect that is being referred to in the famous 'greenhouse effect'.

What happens when the concentration of greenhouse gases increases in the atmosphere?

Heat from the sun comes straight in, relatively unhindered, it is blueish. It is absorbed at the Earth's surface and re-emitted relatively reddish. It flies up into the air but hits a greenhouse gas molecule and is absorbed. The excited molecule re-emits it in random direction – so much of it goes down, not out to space. The surface of the earth warms up.

But, just as insulating a pipe with hot water in it, less heat gets to the upper atmosphere which cools down.

So close to the surface hotter, higher up colder. Recall the system is out of equilibrium for the time being.