Hi there,
As you noticed, you cannot make a reservation for a bus from Shinjuku to Yamanakako online unless you make a phone call to the ticket office at +81-3-5376-2222 for reservation. However, the final destination of most buses is actually Yamanakako. So, what you can do is to make a reservation from Shinjuku to either Kawaguchiko or Fujikyu Highland online. At the end of your reservation process, the confirmation page will be displayed for you to print out. If you see the page, please check the final destination of the bus under the item described as "Boarding bus" (sorry, this description is a bit confusing...). If it says something like 15:15 For Yamanakako, then the bus goes to Yamanakako, so when you pay for ticket at Shinjuku station, please tell the clerk that you want to go to Yamanakako instead.
I hope this is clear enough for you. If you have any further questions regarding this matter, please get back to us. Thank you!