If you look at the example load calculations in Annex D of the NEC, specifically Example D1(a) you will see that for a similar dwelling the calculated load is 78 amps.
Load calculations are complicated and are based on the actual watts of the devices that will be present in the home and how they will be used not by the amps of the breakers. Breakers and their amp ratings are for protection against worse case scenarios. If you went to any home and added all the breakers together non would be OK according to your approach.
As others have said Service Size cannot be determined by adding all the breakers sizes.
The process involves a series of calculations which home inspectors are not required to do. Having said that, it sometimes is possible from experience that an inspector may suspect the Service Size not to be adequate based on what he/she is seeing. In your particular case though, 100A maybe adequate.