By default, OpenSolaris doesn’t broadcast the hostname on the local network, just the IP address. To rectify this, find out what network interface you have (often nge0) by running ifconfig -a. It’ll be the one with the IP address given by the router (i.e., not 127.0.0.1).
Then, as root, edit the file /etc/default/dhcpagent. Find the line # CLIENT_ID=, delete the # to uncomment the line, and append your hostname. Find the line # REQUEST_HOSTNAME=no and change to REQUEST_HOSTNAME=yes. As documented in that file, you also need to edit your /etc/hostname.<if> file, where <if> is your network interface (often nge0), adding the line inet hostname.
You can then either reboot the machine, or renew the DHCP lease on a console on the machine (since it will disconnect you from any SSH sessions). As root, first execute ifconfig <if> dhcp release to discard the current lease, then ifconfig <if> dhcp start to start DHCP on the interface again. Complete documentation can be found through the man ifconfig and man dhcpagent pages.
Result: I can see the hostname through the DHCP server now, which I couldn’t before, but I still can’t see it from the Windows box. So that’s another piece of the puzzle to track down.
Mind you, I need to give the Solaris box a static IP address for serving websites, so the DHCP naming thing is moot. It would be nice to be able to ssh <hostname> from other machines on the network rather than using the IP address though.