Mfcttrf

Let us consider a unitary or antiunitary operator $hatU$, that associates with each quantum state $| psi rangle$ another state $hatU | psi rangle$. I have read that to $hatU$ be a symmetry transformation it has to keep the Hamiltonian $hatH$ invariant. It means that $hatU^dagger hatH hatU = hatH Rightarrow [hatH,hatU] = 0$. But what does it mean physically?

I believe that a symmetry is a transformation that doesn't change the physics of the system, that is, this doesn't change neither the expection values of the physical observables nor the probabilities, right? So, how is this related to the invariance of the Hamiltonian?

Sometimes this is claimed without much explanation.

The time evolution operator is given by exponentiating the Hamiltonian:
$$
U(t) = exp(-i that H / hbar ).
$$
For concreteness, when we think about a symmetry operation (what you called $U$) let's think about rotations around the $z$-axis. A rotation by $theta$ degrees is given by
$$
R(theta) = exp(-itheta hat J_z/hbar)
$$
where $hat J_z$ is the angular momentum operator in the $z$-direction.

If our symmetry commutes with time translations, we have

$$
[U(t), R(theta)] = 0 implies U(t) R(theta) = R(theta) U(t).
$$

This means that, for any $|psi rangle$,

$$
U(t) R(theta) |psi rangle = R(theta) U(t) |psi rangle.
$$

In other words, if you rotate the state by $theta$ degrees and then wait $t$ seconds, you will end up with the same state as if you first waited $t$ seconds before rotating $theta$ degrees.

The "commutativity" of these operations is often what physicists mean when they say they have a symmetry.

By differentiating the equation $[U(t), R(theta)] = 0$ by $t$, $theta$, or both, we can see that this statement is actually equivalent to four closely related statements

1. 
   $[e^-i t hat H/hbar, e^- i theta hat J/hbar] = 0$: Rotating and then time evolving a state is the same as time evolving and then rotating. (We have a symmetry.)


2. 
   $[e^-i t hat H/hbar,hat J] = 0$: The angular momentum of a state does not change after time evolution. (Angular momentum is conserved.)


3. 
   $[hat H, e^- i theta hat J/hbar] = 0$: The energy of a state does not change if the state is rotated.


4. 
   $[hat H, hat J] = 0$: If you measure the angular momentum of a state, the probability that the state will have any particular energy afterwards will not change. The reverse is also true. ($hat H$ and $hat J$ can be simultaneously diagonalized.)

If an operator doesn't commute with a Hamiltonian, then the eigenstates of that operator are not also eigenstates of the Hamiltonian. In that case, we say that the transformation defined by the operator is not a symmetry of the system.

Here's an example from classical physics. The law that the magnitude and direction of the angular momentum a vector are constant is a consequence of Noether's theorem, where the transformation of interest is changes of orientation in space. Angular momentum is conserved because space doesn't have any preferred direction. But, here on Earth's surface, space does have a preferred direction: it's "down." And so, if you have an isolated object rotating on Earth's surface, its angular momentum is not generally a constant. Instead, the orientation of the rotating object precesses.

If you have some operator which does not commute with the Hamiltonian, you would say that the transformation embodied by that operator is not a symmetry of your system.

Requiring that an unitary operator $U$ doesn't change the transition rates is an empty statement, because it's always true that$$
langle psi | chirangle = langle U psi | Uchirangle = langlepsi | U^dagger U |chirangle,.
$$
On the other hand requiring that it leaves expectations values unchanged is too strong of a constraint. Take for instance a rotationally invariant system. If you rotate with respect to any axis that is not $vecz$, the expectation value
$$
langle psi | hatJ_z | psirangle,,
$$
will change. In particular it flips sign if you rotate by $pi$ around, say, $vecx$.

The short answer to your question is: by definition. But I'll try to explain the motivation.

Symmetries in physics are deeply connected to constants of motion. Every time you have a symmetry in classical dynamics (rotation, translation, $U(1)$, ...) you get a constant of motion (angular momentum, momentum, charge,...). We want to import the same concept to quantum mechanics. And it turns out that the operators play both roles at the same time. They act as generators of a symmetry if you use them on the state and they act as constants of motion if you take their expectation value.

Now let's see why an operator with an expectation value that is constant in time must commute with the Hamiltonian. Call $J$ the generator of the symmetry and $U(theta) = exp(itheta J)$ its associated unitary operator. Our expectation value is
$$
E_psi(t) equiv langle psi | e^i H t / hbar,J, e^-i H t / hbar| psirangle,.
$$
We require the derivative of this to be zero
$$
-ihbarfracmathrmdE_psimathrmdt = langle psi | ,[H, J],|psirangle = 0;;forall;psi in mathscrH,(mboxHilbert space),.
$$
Obviously if you take $psi = chi + phi$ you can prove that $langlechi|[H,J]|phirangle= 0$ so the commutator is zero as an operator. Finally, if $H$ commutes with $J$, then it commutes with any power series in $J^n$ and thus $U(theta)$ as well.