Make sure that you’re only using the Jetpack modules you want.
Most of Jetpack is designed to speed up your site by taking load off of your server, like our site accelerator https://jetpack.com/support/site-accelerator/ or stats https://jetpack.com/support/wordpress-com-stats/ subscriptions https://jetpack.com/support/subscriptions/ and related posts https://jetpack.com/support/related-posts/ which are all processed on our end (rather than hammering your database as these features traditionally do).
With that said though, the more Jetpack modules you add, the more your site has to initially process, just like it would if you added a separate plugin for the features, so always make sure you’re only using the features you want.
Regarding overall speed and load, here’s an excellent independent report: https://mattreport.com/jetpack-speed/
