I can see exactly the same problem as the one you are describing. If I do not access the site for a couple of hours, the initial access is about 20 seconds. All following accesses is below 2 seconds.
I suspect either database accesses or (security) plugins which makes expensive calls to external web sites. To address the problem, I have initially switched security plugin (which i believe decrease the delay to load the page no matter what). I also installed a cache plugin. In my case, I have a static front page. So I don’t expect any heavy database access for the initial page. I also have some doubts about a statistics plugin I am using, but I keep it for now.
When it comes to the server side caching, my host runs the Varnish Cache on the HTTP server. I believe it will work well together with the WordPress cache plugin.
I’ll wait and see if my updates improves the access time.
