How do we know when an infinitive is an adverb or a noun?For example, He lives to swim and water-ski. I am going to the pond to fish. Chen has...

In the first and the third examples you give above, the infinitive is functioning as a noun (and as the object of the sentence).  In the middle example, it is functioning as an adverb.

It's not always easy to tell.  But here's how you can try.

If used as a noun, it tells what the object of the verb is.  As in this example:

I wanted to buy the house. (What received my wanting?  To buy the house.  So it's a noun and an object.)

Or in your example, what received Chen's learning?

If it's used as an adverb, it tells you why or how the verb is happening.  As in this example:

We were impatient to start the game.  (Why were we impatient?  To start the game.)

Or in your example, why am I going?