What are the rules in "Rules of the Game" by Amy Tan and how do they relate to Waverly and her mum?

You have identified one of the many master touches about this short story. The title itself operates on many different levels and is clearly relevant to the struggle that is ongoing between Waverly and her mother. You need to remember that this short story in essence is about the desire for independence and the right to live your own life. Waverly, as someone who is born and bred in the United States, has very different values and beliefs from her mother, who was born and bred in China and emigrated later on in her life. What we see in this story is the "Rules of the Game" that Waverly is playing in her effort to achieve independence.

However, there are other rules as well. You may wish to consider the rules of Santa Claus about being a good girl, the rules of chess, the American rules and procedures and the rules of chess etiquette. For me, one of the most striking set of rules are those that Waverly and her family have had to learn very quickly in order to operate:

"This American rules," she concluded at last. "Every time people come out from foreign country, must know rules. You not know, judge say, Too bad, go back. They not telling you why so you can use their way go forward. They say, Don't know why, you find out yourself. But they knowing all the time. Better you take it, find out why yourself." She tossed her head back with a satisfied smile.

This in a sense captures both the immense struggle it must be for immigrants to settle in the United States but also the rules of the game that operate in Waverly's family. She is forced to discover the rules by trial and error in just the same way that her mother has had to learn the rules of immigration and American life. The ending of the story clearly displays that Waverly has to learn to live within the restrictions of her mother's rules - her dream at the end of the story where she imagines playing a game of chess against her mother, who is personified as "two angry black slits", clearly shows that she has "lost" this game of achieving independence for now, but the last line clearly states that the overall game is not over yet:

I closed my eyes and pondered my next move.

The cool and calculating Waverly that we have seen playing chess is not going to be defeated by a setback, however major, and we can see her determination to continue playing the game and fighting in her attempt to break free from her mother.