Totally agree. Since I started refereeing basketball last year, I have made it the point to let the officials do their job and only comment to myself. Yelling at them will only make them either miss more calls, or make the wrong calls.
MY concern as well is around the safety of the players. Because he let a few things go the other night it was very chippy. With about 3 minutes left in the first half a player for the other team was called for a dangerous play in the box on a corner kick. He tried to heel kick the ball with his foot near player's head. At the time, the call was right and we were awarded an indirect kick at the 6 yard line. A few seconds later, the ref realized that our player was bleeding from the head because of the contact. Because of this new information, he could have given the player a card and change the result of the foul to a direct (penalty) kick, but didn't. The player couldn't play the second half because the bleeding never stopped, but could have been avoided if the referee had better control of the game.