If you can get to Himeji for when it opens at 9 (recommended anyway in terms of crowds) you really only need 2-3 hours and actually you can save time by only going into the garden, the inside of the castle isn't a must see. You'll then have plenty of time to travel on.
The inside of the castle is worthy if you are interested in how they construct the castle and the various defensive functions. The interior is sparse though. I recommend that you research and read up a bit before entering. Just going to the outside to snap pictures is kind of pointless. Within the grounds, a separate Western Corridor Building tells a nice story of Princess Sen. Next door, head to Koko-En (combination ticket available). It's a pretty decent garden.
If you aren't going on weekends or sakura/foliage season, there shouldn't be much of a line heading into the castle. It does pay to be early so you can head to Kobe afterwards. Allow 3 hours or so as Natalie suggested for the castle + Koko-en.
Your best pass to get on balance is the JR Kansai Area Pass 1-day version:
I agree with the opinion that if you're going to Himeji Castle, see Himeji Castle, all of it. Two or there hours is plenty there, and if you have time go to the gardens next door (there's a combo ticket with the castle and gardens). My typical suggestion is to walk around the castle's exterior if you can; there's a nice park behind and many places where you can get a good view; in particular is Otokoyama, a small hill just west of the castle area. My profile pic was taken from there; it's a very nice place, though quite a few steps.
The easiest way to get from Himeji to Kobe is JR; but the Sanyo line also goes there; it is cheaper, though it's a bit slower. If you already have a non-JR pass (Kansai Thru Pass, for example), that will work with Sanyo.
As to whether you have time for Himeji and all you want to do in Kobe: it depends. Get to the castle when it opens and you should be able to leave Himeji by 1:00 at the latest. If you're going in summer, the view from the gardens in the evening is very nice, and you could do China Town first and then head up to Nunobiki. But in winter that isn't an option so you should go to the ropeway first and then China Town.
A couple of good links:
Thank You NatalieP, SammyF1oyd, Nishin91 and worthj1970
we will be there in early april . so we have to be in himeji castle by 9am from osaka so that we can enjoy the kokoen garden? then head off to Kobe for Lunch in china town or in sanomiya station for Kobe beef than nunobiki rope way in late afternoon?
I think it's good to plan to arrive by 9:00. Personally I would go to the castle first, buying the combo ticket for Kokoen, and then go there after the castle. If you have time after Kokoen, you could then do the walk around the castle park for a more full experience; but most people seem to just do those two attractions and then return. As Nishin noted, most likely you'll be leaving Himeji after noon. Remember that to get to China Town, you need to change trains at JR Kobe to the local line; it's only one station to JR Motomachi, and a couple of blocks to China Town. I usually walk from China Town to Sannomiya, or you could take a taxi up to the ropeway in Shin Kobe.
In regards to the Himeji castle, can I get into the gardens area without buying a ticket to see the inside of the castle or do I still have to pay the 1040 yen entrance fee? Or are there areas to photograph the castle without having to purchase the entrance fee?
There is a front courtyard that is free, and it is free to go through the parks around the castle. Nearby is Otokoyama, a small hill with the best castle view in town.
But the best parts of the castle require the admission fee.