When I used LHTH with my second son, he was already reading short vowel words and Bob books, yet he was really only 4 turning 5. We still did all of LHTH, even though he knew most of what was in the program.
As we used LHTH, there were so many times I was thankful for the Biblical discussions and connections that my son was making that would not have been possible without LHTH. Today, my son has just completed "Bigger..." and is heading into 4th grade! Yet, we still both remember our time spent with LHTH so fondly.
Academic-wise my second son is so strong and does well overall, so doing LHTH (even though he knew most of it) in no way held back his learning.
Hold onto the joy of those first years, as the heavier academic years come quickly enough!