FIM SSPR error 3000

One day I ran into this error on a previously working FIM 2010R2 self service password reset portal configuration instance.  All users were getting error 3000:

An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)

On the usual rounds of google, I found a few articles for testing wmi permissions and guides to ensure permissions.  The Microsoft provided install guide also covers some of this as well.  The one thing though is WMI permissions that are discussed are always root\cimv2 permissions.  When looking at my configuration, everything was already set up as it should be (according to all of these documents).  I enabled debug logging on the portal to get more details.  This showed me this message among the other related events:

System.Management: System.Management.ManagementException: Access denied
   at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)
   at System.Management.ManagementScope.InitializeGuts(Object o)
   at System.Management.ManagementScope.Initialize()
   at System.Management.ManagementObjectSearcher.Initialize()
   at System.Management.ManagementObjectSearcher.Get()
   at Microsoft.ResourceManagement.PasswordReset.ResetPassword.ResetPasswordHelper(String domainName, String userName, String newPasswordText)

So obviously I still had a permissions problem somewhere.  After digging around and getting nowhere, I fell back on my favorite troubleshooting rule, "when in doubt, Netmon".

 

As you can see in the last frame, the access denied error message is present on a WMI call.  So this quickly narrows it down to a WMI issue.  Go up a few frames to the blacked out server name, and we see it accessing the root\MicrosoftIdentityIntegrationServer.  Oddly, you won't find this mentioned anywhere in the documentation.  Once I granted the service account "enable account" and "remote enable" rights on this namespace on the sync server, everything started working fine.  The root caused ended up being removal of admin rights from the fim service account on the sync server.  As usual, in bad documentation and bad troubleshooting, granting admin rights solves a lot of problems.  If you want to go with a more restricted environment, add this to your build steps for SSPR.

Just for search purposes, here are some of the other event log texts that may be seen when this issue occurs:

WorkflowInstance '3eb56c33-cc6f-4c4f-90cf-30086a5d1fbd' [Description: ] recorded the following event for activity authenticationGateActivity1.FailureBranch (type:System.Workflow.Activities.IfElseBranchActivity): Executing at 2/27/2015 6:39:02 AM.
-----------
The error page was displayed to the user.
Details:
Title: Error
Message: An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)
Source:
Attributes:
Details: System.InvalidProgramException: Error while performing the password reset operation: PWUnrecoverableError
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Reset.AttemptToResetPassword()
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
CorrelationId:
RequestId:
ErrorCode: 3000
CaughtTime: 02/27/2015 01:39:02
Web Portal: FIM Password Reset Portal
Session Id: mfa4tg550e52v4be0bhhxu45


-----------------

Microsoft.IdentityManagement.CredentialManagement.Portal: System.Web.HttpUnhandledException: ScriptManager_AsyncPostBackError ---> System.InvalidProgramException: Error while performing the password reset operation: PWUnrecoverableError
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Reset.AttemptToResetPassword()
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Site.ScriptManager_AsyncPostBackError(Object sender, AsyncPostBackErrorEventArgs eventArgs)
   at System.Web.UI.ScriptManager.OnAsyncPostBackError(AsyncPostBackErrorEventArgs e)
   at System.Web.UI.PageRequestManager.OnPageError(Object sender, EventArgs e)
   at System.Web.UI.TemplateControl.OnError(EventArgs e)
   at System.Web.UI.Page.HandleError(Exception e)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP.default_aspx.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

---------

A user's attempt to reset password ended with the following result.
The result details:
Error while performing the password reset operation: PWUnrecoverableError
Web Portal: FIM Password Reset Portal
Session Id: mfa4tg550e52v4be0bhhxu45

---------
Service fault of type DataRequiredFault was received.
----------

The web portal received a fault error from the FIM service.
Details:
Microsoft.ResourceManagement.WebServices.Faults.ServiceFaultException: DataRequiredFaultReason
   at Microsoft.ResourceManagement.WebServices.ResourceFactoryClient.Create(Message request)
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Common.ResetProxy.InteractWithPasswordResetActivity(SecureString newPassword, String activityEndpoint, String workflowInstanceId, ContextualSecurityToken sessionSecurityToken)
Web Portal: FIM Password Reset Portal
Session Id: mfa4tg550e52v4be0bhhxu45

Get-wmiobject "User credentials cannot be used for local connections"

In Advanced event 2 of the 2013 scripting games, I was going through scripts and thinking it was very strange that people were creating separate blocks of code to do get-wmiobject calls that involved the local machine. This is somewhat disturbing when you are writing code that hits a long list of machines that may include your own local machine. Having to add all the constant checks to see if you are on the local machine seems like a bit of a pain. I can't think of any great work around for this that isn't going to cause additional problems, but you could just let the call fail and pick it up in a catch block. One other option, which creates seperate powershell processes would be:

start-job -credential $cred -scriptblock {get-wmiobject -class yourclass -computer $args[0]} -argumentlist $computer

Then go back and receive-job on the output to process it.

Delegating WMI security remotely with powershell

A while back, I was working on a script for checking domain controller security event logs which needed to be handed off to a team that was not part of the Domain Admins group.  While they had permissions to access the security event logs through user rights in GPO, trying to read the event log through an MMC remotely is ridiculously slow.  A good solution was to use WMI with a tight filter for event ID's and a brief time window for the specific event.  The problem was, none of this team had WMI access.  So to go about fixing a few hundred domain controllers, I started poking around at WMI permissions.  You can edit this through the MMC->Component Services console.  But doing this via RDP on such a large scale is not an option.  There are some script examples in VB, such as this one.  But, being a powershell guy, I wanted to use some existing code and wrap in the additional lines to update the security.  So, what I did was created the permissions that I wanted on one specific machine (similar to the above article), and use powershell to pull the security descriptor.  The example below is delegating the common root\CimV2 namespace which contains the event log event classes.


#Collect your security descriptor

$sd = gwmi -namespace "root\cimv2" -Class __SystemSecurity -ComputerName $FixedMachine
$sdhelper = new-object system.management.ManagementClass Win32_SecurityDescriptorHelper
$binarySD = @($null)
[void]$sd.PsBase.InvokeMethod("GetSD",$binarySD)

#At this point you can loop through a list of machines and push out the updated permissions.
$sdlocal = gwmi -Namespace "root\cimv2" -Class __SystemSecurity  -computername $remotemachine
$sdhelper2 = new-object system.management.ManagementClass Win32_SecurityDescriptorHelper
[Void]$sdlocal.psbase.invokemethod("SetSD",$binarySD)



For more details on the various types of permissions, you can reference this technet article.  For remote read-only, you can go with "remote enable", and "enable account".

Converting time formats in Powershell

Since I am spending the morning looking at solutions to track down lockout events to source in a more automated fashion, I'm stuck looking at how to dig through event log events. Using the Active Directory values for bad password time helps to locate a domain controller with useful log events, but the format needs some adjustment. When looking at AD, the format can look like this:

lastlogontimestamp {129121380931028667}
badpasswordtime {129115363568077019}

In powershell it is very easy to convert this to a datetime object for further use. If you want to just display the value in a readable format, you can just use this command. If you want to work with it some more, assign it to a variable:

[datetime]::FromFileTime($mydatetime)

ex:

PS C:\> [datetime]::fromfiletime(129121380931028667)

Wednesday, March 03, 2010 7:01:33 PM


Now when looking at event logs with something such as WMI queries, the date time format is different.
Format: YYYYMMDDHHMMSS.000000-UUU
Reference

The System.Management.ManagementDateTimeConverter class can be used to convert between WMI's format and DateTime.

Once the value is in WMI format, into a wmi queries. If you are searching logs in a timewritten range you may want to adjust the seconds of this value +/- a few to help ensure that everything links up. You can also drop the milliseconds as the AD time stamp may not perfectly match up to a system event log time stamp. In any case, having a tight time range and very specific WMI query will help keep remote event log search queries fast and leave you will less results to process in any addition work you plan to do.

Some basic code using these conversions and providing a time range for event log searches:

function wmiTime-toDateTime([string]$wmitime){
#YYYYMMDDHHMMSS.000000-UUU where -UUU is the three-digit offset
return [system.management.managementdatetimeconverter]::todatetime($wmitime)

}

function dt-toWmiTime([system.datetime]$dt) {
#YYYYMMDDHHMMSS.000000-UUU where -UUU is the three-digit offset
$dt = $dt.touniversaltime()
$wmiformat = [System.Management.ManagementDateTimeConverter]::ToDmtfDateTime($dt)
return $wmiformat

}

function adtime-toWMItime([string]$mydatetime) {
$dt = [datetime]::FromFileTime($mydatetime) 
$wmitime =  dt-toWmiTime $dt
return $wmitime
}


function adjust-wmitime ([string]$wmiOrigTime, [double]$secondOffset) {
#WMI time format: YYYYMMDDHHMMSS.000000-UUU where -UUU is the three-digit offset
#This function takes an original time and provides an array of two values, search start time and
#search end time

$mybasetime = wmiTime-toDateTime $wmiOrigTime
$myadjustedDT = $mybasetime.addseconds($secondoffset)
$myResultingWMI = dt-toWmiTime $myadjustedDT
return $myResultingWMI

}

function compare-wmiTimes($startTime,$endTime) {
 $dtS = wmiTime-toDateTime $startTime
 $dtE = wmiTime-toDateTime $endTime
 $timediff = new-timespan -start $dtS -end $dtE
 if ($timediff.TotalMilliseconds -gt 0) {
  #second date is later than the first
  return 1
 } elseif ($timediff.TotalMilliseconds -eq 0) {
  #time is the same
  return 0
 } else {
  #first date is later than the second
  return -1
 }

}

function wmitime-timerange ([string]$wmiOrigTime, $secondOffset) {
$secondoffset = [double]$secondoffset
$mystartWMI = adjust-wmitime $wmiOrigTime -$secondoffset
$myendWmi = adjust-wmitime $wmiOrigTime $secondoffset
$myResult = ($mystartWMI, $myendWMI)
return $myResult
}

Dell system monitoring with WMI

For those who may be interested in checking Dell hardware through scripts there are several ways to do it. I will focus on Windows in this post, and skip over wmic and vbscript, to show some powershell examples of this. For background reference and WMI, there is a useful document at Dell for wmic and a reference guide for Common Information Model administration as well. For some background on Dell's WMI namespace, this has existed for quite some time in Dell OpenManage, starting sometime in version 4.x (I have seen in it 4.4 and higher, though I can't verify earlier versions). The namespace is unique branch on CIMV2, called root\CIMV2\Dell. Some examples of information provided inside this namespace: -Per component details and status -Hardware log retrieval -Component types installed and firmware information -Remote access card component information Using the same WMI capabilities that come with the Dell OpenManage software, we can use Powershell's Get-WMIObject commandlet to pull information from a machine. Using this, you can limit the results displayed similar to the examples of wmic, however the results are objects, so they are much easier to work with for further programming. Here is a basic example

PS C:\> gwmi -namespace root\cimv2\dell -class CIM_PhysicalMemory __GENUS : 2 __CLASS : CIM_PhysicalMemory __SUPERCLASS : CIM_Chip __DYNASTY : CIM_ManagedSystemElement __RELPATH : CIM_PhysicalMemory.CreationClassName="CIM_PhysicalMemory ",Tag="0" __PROPERTY_COUNT : 27 __DERIVATION : {CIM_Chip, CIM_PhysicalComponent, CIM_PhysicalElement, C IM_ManagedSystemElement} __SERVER : myserver __NAMESPACE : root\cimv2\dell __PATH : \\myserver\root\cimv2\dell:CIM_PhysicalMemory.Crea tionClassName="CIM_PhysicalMemory",Tag="0" BankLabel : Capacity : 2147483648 Caption : CreationClassName : CIM_PhysicalMemory DataWidth : 64 Description : FormFactor : 9 HotSwappable : InstallDate : InterleavePosition : Manufacturer : MemoryType : 19 Model : Name : DIMM1_A OtherIdentifyingInfo : PartNumber : PositionInRow : PoweredOn : Removable : Replaceable : SerialNumber : SKU : Speed : 2 Status : OK Tag : 0 TotalWidth : 72 Version : __GENUS : 2 __CLASS : CIM_PhysicalMemory __SUPERCLASS : CIM_Chip __DYNASTY : CIM_ManagedSystemElement __RELPATH : CIM_PhysicalMemory.CreationClassName="CIM_PhysicalMemory ",Tag="1" __PROPERTY_COUNT : 27 __DERIVATION : {CIM_Chip, CIM_PhysicalComponent, CIM_PhysicalElement, C IM_ManagedSystemElement} __SERVER : myserver __NAMESPACE : root\cimv2\dell __PATH : \\myserver\root\cimv2\dell:CIM_PhysicalMemory.Crea tionClassName="CIM_PhysicalMemory",Tag="1" BankLabel : Capacity : 2147483648 Caption : CreationClassName : CIM_PhysicalMemory DataWidth : 64 Description : FormFactor : 9 HotSwappable : InstallDate : InterleavePosition : Manufacturer : MemoryType : 19 Model : Name : DIMM1_B OtherIdentifyingInfo : PartNumber : PositionInRow : PoweredOn : Removable : Replaceable : SerialNumber : SKU : Speed : 2 Status : OK Tag : 1 TotalWidth : 72 Version :

As you can see, this provides a lot of attributes, many of which are not read by the system. You can filter out what you want with Select-Object, such as: PS C:\> gwmi -namespace root\cimv2\dell -class CIM_PhysicalMemory|select-object name,status name status ---- ------ DIMM1_A OK DIMM1_B OK This provides a nice summary of a component. You can easily do follow on processing to look for bad components, such as a foreach loop on the result and list where status does not match "OK". If you are looking for a solution in a mixed hardware environment where not all servers may be up to date for OMSA, and updating is not an easy task, a work around is to set your erroraction to silentlycontinue, read the error message of the wmi connection and direct the user to use the web interface for omsa or a command line tool like omreport to collect the information. For more examples of code, you can use scriptomatic to look through the namespace and test what information is available. I also have a few examples posted on Microsoft's technet script center.