The popular telling of the story is that the team behind Mono asked for too much money to allow the Unity3D team to have a commercial license exception to newer versions of Mono.
(This comes from the Unity team's explanation to the Unity community about why their runtime remains out of date. It's since become a meme to paint the Mono folks as a bunch of jerks over this. This endured for a few years. In the meantime, however, Microsoft has released .NET under permissive terms that would allow Unity to use Microsoft's own runtime royalty-free—no need for a commercial license exception. Despite the opportunity it presented, an upgraded runtime failed to materialize from Unity. Maybe the transition from Mono to .NET Core was too big of a leap? Fair point. Thing is, Mono itself is now also relicensed under MIT, by way of Microsoft's acquisition of Xamarin, so Unity is free to make commercial use of newer Mono versions on proprietary platforms, just like Unity always wanted. Still, no upgrade to the Unity runtime.)
Xamarin and Unity never did a renewed licensing deal. That doesn't make the lovely xamarin people nor us jerks :)
Licensing issues aside, for an ecosystem like Unity to upgrade its runtime, c# language, and base class libraries is an enormous undertaking. While we are not perfect, we take backward compatibility very seriously.
All of mono and coreclr has been available under MIT licensing for a very long time now. In Unity2017.1 we now ship an experimental option to use modern .net. this uses a new mono (not using sgen), new c# compiler (still mcs for now, so c#6, not c#7, and .net4.6 class libraries).
In a future unity version (won't burn my fingers on exactly which one) this new .net experience will become default. (depends on how many problems you find! :)
the xamarin (and microsoft .net) team are wonderful folks, and it's great to work closely with them to make .net in unity, and .net in general solid.
next up on the list is improving the garbage collection story, which is in the works, but we're first focussing on shipping a new runtime+c#+.net base class libraries.
While there are some very valid areas to point out that make c# challenging to be "the language of an entire game", we believe that c# is a great controlling language for Unity. We're working on some really interesting (imho) compilation pipeline for a restricted compute subset of c# that I think will completely turn around people's expectations and opinions of c# in gaming. stay tuned, this is one of the most interesting projects happnening inside of Unity today.
(This comes from the Unity team's explanation to the Unity community about why their runtime remains out of date. It's since become a meme to paint the Mono folks as a bunch of jerks over this. This endured for a few years. In the meantime, however, Microsoft has released .NET under permissive terms that would allow Unity to use Microsoft's own runtime royalty-free—no need for a commercial license exception. Despite the opportunity it presented, an upgraded runtime failed to materialize from Unity. Maybe the transition from Mono to .NET Core was too big of a leap? Fair point. Thing is, Mono itself is now also relicensed under MIT, by way of Microsoft's acquisition of Xamarin, so Unity is free to make commercial use of newer Mono versions on proprietary platforms, just like Unity always wanted. Still, no upgrade to the Unity runtime.)