To what measures do Hrothgar and his nobles resort to rid themselves of Grendel?

They essentially try everything possible in their cultural context. They are used to dealing with other men, of course, and so their initial responses to Grendel follow the protocol of situations of war and battle. Yet Grendel does not respond as a man or another nation would.

He would brook no parley with any earls of the Daneland, would make no pact of peace, nor come to agreement on the blood gold—nor did any councilman expect fitting payment for the feud from his fiendish hands.

So it's clear they tried to make some kind of deal with him, perhaps reach a kind of peace accord. They also attempt to pay "blood gold": a common practice in the culture. If someone was murdered, the family would pay money in order to forestall further killing. The payment was usually related to the victim's social status. Hrothgar also tries rituals to rid the land of Grendel.

Many times did the realm gather in council, seeking out how best the stout-hearted men could try their hand against the horrific menace. Betimes at heathen shrines they made sacrifice, asking with rites that the slayer of souls would afford them relief against their people's great pain.

So they planned attacks of their own, & they sacrificed at shrines. All of these were methods tried before Beowulf comes and tears Grendel apart with his bare hands.