I was looking this up just now. It seems 29 million died in India alone under British rule due to famine. [1] However there was widespread crop failure in the late 19th century. [2] I don't know how much control they had over what and how every set of deaths occurred, so it's not yet clear to me you can just directly compare number of deaths between empires without taking into account the surrounding context.
While it wouldn't be wise to attribute all those deaths to the British Empire, it's important to note that British agricultural practices led to widespread famines and crop-failures. Diversion of food crop land to grow Opium and Indigo led to famines[1]. And the risk of crop-failures was often beared by the cultivator[2].
The Bengal Famine is a well-studied case where, AIUI, the current consensus is that imperial policies contributed significantly to the suffering and death toll:
>so it's not yet clear to me you can just directly compare number of deaths between empires without taking into account the surrounding context.
This is true, but perhaps in a way you weren't anticipating; this point was made decades ago by Herbert Marcuse in response to people attempting to compare, for instance, Nazi death tolls with Soviet ones. This mode of argumentation reduces concrete matters of policy and motivation (quality) into mere numbers to be thrown about (quantity). But authoritarianism is not a quantitative matter - a nation with harsh laws under which only few are convicted is still an authoritarian nation.
[1] https://www.quora.com/How-many-people-did-the-British-empire...
[2] https://en.wikipedia.org/wiki/British_Empire#East_India_Comp...