Been to Japan in December, August, and May. Plan to go this June. It depends where you are going, but you really should go to Kyoto and the best time is in the spring for Kyoto, April or May before it gets too hot and the flowers are in full bloom. September or October work as well. In May there tends to be a lot of school trips to Kyoto, so you might see large groups of Japanese junior high kids in uniform. We were approached by a few groups so they could practice English.
When we went in December, we went south to Nagasaki and Okinawa for warmer weather. Kyoto was not freezing, but the gardens were bare.