Iran has nothing to gain by their stance unless they believe they are right about the boat's position.
I'm not sure that anyone can say it's "certain" that they were outside Iranian waters, That would put them in Iraqi waters, where the captors almost certainly would have been fired upon by any American forces or British forces in the area. Naturally Britain would say they weren't, to save face. If you assume that the Iranians believe they were in their waters, then the sanctity of its borders hangs in the balance. If they cave, and just release the sailors, they can expect a little more encroachment next time, and the time after that, and so on, until they finally respond with force, and then they will be at war.
Israel sent two of its soldiers into Lebanon last year, knowing that they probably would be captured, and used that as an excuse to carpet bomb Beirut.
British and U.S. forces were running bombing raids through the Iraqi no-fly zone 6 months before 9/11, and then cried"Look, look, he's shooting at us!"
So there is a recent history of provocative actions by the U.S. and its allies in that region.
It's not like they blew the British boat out of the water, so it should be handled as a border dispute: send envoys to negotiate in private so no one loses face; and get those sailors home safely, no matter what you intend to do after that.
I know we don't have any diplomats, but maybe Britain still has 1 or 2....