FIM SSPR error 3000

One day I ran into this error on a previously working FIM 2010R2 self service password reset portal configuration instance.  All users were getting error 3000:

An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)

On the usual rounds of google, I found a few articles for testing wmi permissions and guides to ensure permissions.  The Microsoft provided install guide also covers some of this as well.  The one thing though is WMI permissions that are discussed are always root\cimv2 permissions.  When looking at my configuration, everything was already set up as it should be (according to all of these documents).  I enabled debug logging on the portal to get more details.  This showed me this message among the other related events:

System.Management: System.Management.ManagementException: Access denied
   at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)
   at System.Management.ManagementScope.InitializeGuts(Object o)
   at System.Management.ManagementScope.Initialize()
   at System.Management.ManagementObjectSearcher.Initialize()
   at System.Management.ManagementObjectSearcher.Get()
   at Microsoft.ResourceManagement.PasswordReset.ResetPassword.ResetPasswordHelper(String domainName, String userName, String newPasswordText)

So obviously I still had a permissions problem somewhere.  After digging around and getting nowhere, I fell back on my favorite troubleshooting rule, "when in doubt, Netmon".

 

As you can see in the last frame, the access denied error message is present on a WMI call.  So this quickly narrows it down to a WMI issue.  Go up a few frames to the blacked out server name, and we see it accessing the root\MicrosoftIdentityIntegrationServer.  Oddly, you won't find this mentioned anywhere in the documentation.  Once I granted the service account "enable account" and "remote enable" rights on this namespace on the sync server, everything started working fine.  The root caused ended up being removal of admin rights from the fim service account on the sync server.  As usual, in bad documentation and bad troubleshooting, granting admin rights solves a lot of problems.  If you want to go with a more restricted environment, add this to your build steps for SSPR.

Just for search purposes, here are some of the other event log texts that may be seen when this issue occurs:

WorkflowInstance '3eb56c33-cc6f-4c4f-90cf-30086a5d1fbd' [Description: ] recorded the following event for activity authenticationGateActivity1.FailureBranch (type:System.Workflow.Activities.IfElseBranchActivity): Executing at 2/27/2015 6:39:02 AM.
-----------
The error page was displayed to the user.
Details:
Title: Error
Message: An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)
Source:
Attributes:
Details: System.InvalidProgramException: Error while performing the password reset operation: PWUnrecoverableError
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Reset.AttemptToResetPassword()
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
CorrelationId:
RequestId:
ErrorCode: 3000
CaughtTime: 02/27/2015 01:39:02
Web Portal: FIM Password Reset Portal
Session Id: mfa4tg550e52v4be0bhhxu45


-----------------

Microsoft.IdentityManagement.CredentialManagement.Portal: System.Web.HttpUnhandledException: ScriptManager_AsyncPostBackError ---> System.InvalidProgramException: Error while performing the password reset operation: PWUnrecoverableError
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Reset.AttemptToResetPassword()
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Site.ScriptManager_AsyncPostBackError(Object sender, AsyncPostBackErrorEventArgs eventArgs)
   at System.Web.UI.ScriptManager.OnAsyncPostBackError(AsyncPostBackErrorEventArgs e)
   at System.Web.UI.PageRequestManager.OnPageError(Object sender, EventArgs e)
   at System.Web.UI.TemplateControl.OnError(EventArgs e)
   at System.Web.UI.Page.HandleError(Exception e)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP.default_aspx.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

---------

A user's attempt to reset password ended with the following result.
The result details:
Error while performing the password reset operation: PWUnrecoverableError
Web Portal: FIM Password Reset Portal
Session Id: mfa4tg550e52v4be0bhhxu45

---------
Service fault of type DataRequiredFault was received.
----------

The web portal received a fault error from the FIM service.
Details:
Microsoft.ResourceManagement.WebServices.Faults.ServiceFaultException: DataRequiredFaultReason
   at Microsoft.ResourceManagement.WebServices.ResourceFactoryClient.Create(Message request)
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Common.ResetProxy.InteractWithPasswordResetActivity(SecureString newPassword, String activityEndpoint, String workflowInstanceId, ContextualSecurityToken sessionSecurityToken)
Web Portal: FIM Password Reset Portal
Session Id: mfa4tg550e52v4be0bhhxu45