Tolstoy's narrator says, Ilych begins to feel that he is living "all alone on the brink of an abyss, with no one who understood or pitied him."...

Remember, Ivan Ilych is an "Everyman."  His name is like "John Smith" in English.  He stands for all mankind universal.  Later in the novella, when he's closer to death, he will have these two epiphanies:

A.   That all of his human relationships have occurred simply as institutional categories, rather than as genuine human connections.

B.    That he himself has been nothing more than an institutional category and is going to die without ever having lived in a meaningful way.

This sentence in your question comes at the end of chapter 4 (of which there are 12).  Tolstoy focuses on Ivan's health, although the narrative avoids medical discussion: "The doctor said that so-and-so indicated that there was so-and-so inside the patient..." (one of the great sentences by Tolstoy, who otherwise famed for his realism).  The chapter ends thusly:

With this consciousness, and with physical pain besides the terror, he must go to bed, often to lie awake the greater part of the night. Next morning he had to get up again, dress, go to the law courts, speak, and write; or if he did not go out, spend at home those twenty-four hours a day each of which was a torture. And he had to live thus all alone on the brink of an abyss, with no one who understood or pitied him.

This paragraph describes not only a physically sick man but a spiritually sick one.  Both mind ("consciousness") and body ("physical pain") are afflicted.  Ivan feels tortured by the absurdity of pointless labor which, ironically, he's been doing for years.

His trip the the doctor's office is no consolation.  The doctor is so matter-of-fact about Ivan's maladies that Ivan feels like a litigant in his own courtroom being callously dealt with by a judge.  To the doctor Ivan is an abstraction, patient X, a disease, nobody, a cog in a machine.  Ivan is beginning to realize this about all of the illegitimate society, and it sickens him.

It is also ironic that Ivan's symptoms--the bad taste in the mouth and fits of anger--coincide with Praskovya's pregnancy.  So, we have a death-birth connection here: neither the end of life nor the beginning of life is free from this spiritual malaise.

So, the "brink of the abyss" is not only death but the aforementioned realizations about one's life (A & B above).  The whole novella is a brink of Ivan's abyss, the last moments of a dying man.  Before death, like the thief on the cross, he will be realize that a Buddhist brand of Christianity (a secular belief in Jesus' teachings, focusing on poverty and servitude) should have given meaning to his otherwise pointless life.