FIM Certificate manager portal "Value does not fall within the expected range"

Recently I ran into an unusual issue with a FIM certificate manager portal installation occasionally throwing the "Value does not fall within the expected range" error when doing searches.  Some searches would work all of the time, while others would fail all of the time.


The error:

1) Exception Information
*********************************************
Exception Type: System.ArgumentException
Message: Value does not fall within the expected range.
ParamName: NULL
Data: System.Collections.ListDictionaryInternal
TargetSite: Int32 SecurityDescriptorToBinarySD(Microsoft.Clm.Security.Structs.VariantIDispatch, IntPtr ByRef, UInt32 ByRef, System.String, System.String, System.String, UInt32)
HelpLink: NULL
Source: Microsoft.Clm.Security.Authorization

StackTrace Information
*********************************************
   at Microsoft.Clm.Security.NativeMethods.SecurityDescriptorToBinarySD(VariantIDispatch vVarSecDes, IntPtr& ppSecurityDescriptor, UInt32& pdwSDLength, String pszServerName, String userName, String passWord, UInt32 dwFlags)
   at Microsoft.Clm.Security.Authorization.SecurityDescriptor.ConvertToByteArray(DirectoryEntry entry)



After digging through various components, checking AD, etc, a pattern seemed to emerge.  Whenever the search result should have returned results of users that were in a specific OU, it would fail.  While results that gave only results in other OU's would work.  On checking the metadata for the OU, the ntSecurityDescriptor had recently changed right around the time that the FIM CM portal started throwing errors.  A large number of property management ACE's had been added, which pushed the size of the ACL too high for the system to deal with.  According to Microsoft, the max size for an ACL is 64k.  My previous post shows how easy it can be to hit that limit when you get too fine grained in your entries.  Removing the added ACE's resolved the issue.