Relaying to a non-standard SmartHost SMTP Port
In your /etc/mail/sendmail.mc, try:
define(`SMART_HOST',`relay:your.relay.system.com')dnl
define(`RELAY_MAILER',`esmtp')dnl
define(`RELAY_MAILER_ARGS', `TPC $h 587')dnl
This ensures that you use the relay mailer when talking to your smarthost (which for some reason isn't the default with Fedora Core 3), that you use esmtp when talking to your smarthost (possibly unneccessary), and overrides the 'RELAY' mailer defaults to connect on port 587.
Update 20 Dec 2004: Fixed typo IPC to TCP
Update 23 Feb 2005: Added the SMART_HOST and RELAY_MAILER lines. It works now for real.