What counts in pitch count?

MLB teams count the pitches thrown during a game for each individual pitcher so that the pitcher does not throw too many. This is to avoid injury. But what about other throws that day including: - pregame warm ups - between inning warm ups - throws to first to hold runners - fielding throws? Eight warm up pitches are allowed between innings. If a starting pitcher begins the sixth inning that is 5 * 8 = 40. Forty would be a lot relative to the normal pitch count, let's say 80 through five innings. If a pitcher takes fewer warm ups, shouldn't that be a factor in when he is removed? Pregame warm ups may vary a lot from pitcher to pitcher and even for an individual from start to start. Shouldn't that be a factor in when he is removed?