The panic is in part because of how contagious it is. And for argument sake, let's say the death rate is exactly like the flu. That would still be a problem to let it go unchecked with no restrictions/social distancing because the raw numbers would just get out of control.
I'm itching for sports to return, especially baseball. But I think it would have to be done in empty stadiums or at least greatly reduced capacity in the stands. Packing in a ton of people next to each other is just asking for trouble.