At least for the first few years
of life. It's the natural
state of the human being
to not believe in any religion
However, for most children, their parents soon indoctrinate them with their belief system, resulting in the children learning it as a fact before they are old enough to actually think about it and question it.
I must be clear here that being atheist simply means not believing in any gods. Surely a newborn baby doesn't believe in any.