At a customer recently I was switching a cross platform agent from a 2003 machine to a new 2008 R2 machine. However, when I tried to do this, I got an agent heartbeat failure reporting “wsman returned a 501 error”. This error occurred even when running winrm from the command line.
With the help of Steve Veitch I turned on the logging with the following command
Scxadmin –log-set all verbose
This writes a trace file “cimserver.trc” which contains the winrm calls and their responses. This allowed me to test calling winrm from both machines and compare the results. One message that jumped out at me as being different was the following:
332939007s-853849us: Http [30817:47632577662800:HTTPConnection.cpp:2012]: This Request has non-valid CIM-HTTP Method: 4F 53 54 20 2F 77
Looking for this error led me to http://social.technet.microsoft.com/Forums/en-US/operationsmanagerunixandlinux/thread/ef611c9e-b36a-4d23-a680-de7e1b550fd6 and then on to http://operatingquadrant.com/2012/01/12/opsmgr-unixlinux-heartbeat-failures-after-applying-kb2585542/
This last blog post confirms that MS12-006 patch (KB2585542) can stop connections to Unix/Linux agents. It provides a work around, but cautions that this makes you vulnerable again.
I have then applied the registry setting and we started to get good connections. If you experience this problem you should evaluate the security threat that this patch protects against and investigate whether you need to apply it in these circumstances. If you do, then you should contact Microsoft to work through the issue.