Big Bang Nucleosynthesis (BBN) says that the amount of baryonic (i.e. usual) matter cannot be much higher than what we directly observe. However, it accounts for only ~16% of the mass necessary to account for galactic rotation curves, gravitational lensing by galaxies etc. So the leftover must be non-baryonic dark matter.

One can of course say that BBN is wrong but that's very unlikely as it has been tested to high precision from CMB study.