Your customers are only entitled to the source if you accepted the terms of the GPL and technically you can claim that you didn't accept them and that you simply distributed the derived work illegally.

If your customers wanted the source, they would have to ask the original authors to sue you and have the court force you to accept the GPL post-factum. It's not clear how this would turn out because afaik such line of "defense" has never been tried by violators so far. A few such lawsuits settled out of court, usually with source code being part of the deal.

A ruling in this kind of case, where the violator stubbornly refused to comply and maybe tried to escape with just paying damages or some other trickery, that would be interesting.

A court won't force people to comply with the terms of a license. If they don't comply, they just don't have a license and can be liable for copyright infringement. A court can then enjoin them from continuing to distribute the product, which may be incentive for them to reach an agreement on a commercial license or it might induce them to modify the product to not use the infringed code. And a court can force them to pay damages, which will be the cost of what a commercial license would have been. However a court won't force them to release their code under the terms of the GPL. Even if the GPL were a contract, which it's not, courts just don't order specific performance for breach of contract.

As you mention, though, some potential defendants will determine it's just cheaper to just disclose the modified code and walk away.

I agree, a real, thorough test of this in court would be a great precedent.

And yes, if you refuse source to your customers that's where you are - software copyright violation. Just the same as if you did this with proprietary stuff. Someone with standing would have to start the ball rolling.