What scares the immortal would be, I suppose, not the erosion of loved ones nor the ever-changing and impossible to keep up with world, but rather a night sky where the stars slowly shift positions and the knowledge that all the positions can be seen until those options have run out, the stars have burned out, and the void has closed in.
Immortality...
Ultimately, the worst part of immortality wouldn't be the inevitable loss of humanity or the grinding crush of reality or even near constant thoughts of unattainable suicide, it would be loneliness. Loneliness through all of life, never able to make a connection and then at the end staring into endless darkness and being truly alone knowing there isn't a soul to talk to or even see.
To have no other view points and no one to hug.