Finding account lockout source (general practice)

A lot of escalations come my way regarding difficult account lockout issues, and repeat lockout events. Over time, I took some of the methodology I used in tracking down the root source of the failed logon and put it into an auto tracker script. This runs quite quickly using WMI's WIN32_NtlogEvent class with a very tight filter for the timegenerated attribute. When providing and upper and lower limit, searching event logs can be done very quickly on remote machines. This is important as in most cases you may traverse through 2 or 3 machines to get to the originating machine.

When starting to track down a lockout source, either manually or through automated practice, you will check the bad password timestamps for all domain controllers [this is an ldap query to all writable domain controllers so it can be slow. Alternatively a much faster approach is to read the metadata of that attribute on the PDC only to get the time and source]. You can do this with ldap queries to each one, or use the microsoft lockoutstatus tool. Ldap queries will allow you to script it. The key point here is that the PDC will always have the most recent bad password time, even if the events are not located on that machine. So for automation purposes, I simplified the process by ignoring the PDC. You could always find the most recent reported event and compare it to see if the PDC is more recent by at least a few seconds to see if the PDC is where you should look. It is also important to note that you are assuming the last bad password time that you are looking at is related to the problem and is not something else like the user mistyping a password. This may require several runs of an automated script to see if you get different sources.

From the bad password time stamp, you can go to the security log for that domain controller, filtering for failure audits and check the event text or InsertionStrings to get the source machine or source IP. Different event ID's have different information. Some failed logons will not show up in the log, or may have a blank source. In this case it is important to have netlogon debugging enabled so you can check the netlogon.log, as it may contain more details for the source. In netlogon, you want to look at the entry in parenthesis that shows via XXXXXX. Ensure the code in the line is a failure, and not success, so you don't chase the wrong entries. For automating, it is helpful to have some code that decodes the various security event hex codes and integer values to plain text for display purposes (my code).

When looking at sources in the event log, you need to carefully check source IP and source workstation if both are present. Sometimes they differ, and the source workstation may be the machine you are already on. Sometimes it is best to use the IP as a preferred value for source. Go to the source machine and look at the security logs there for additional information. When you get down to workstations and member servers, you may get process id's and port information that can help narrow down to a process. Checking logon type codes can be a dead giveaway for scheduled tasks and services with bad cached passwords.  On some occasions you will not see a source machine in the security log.  If this is the case, you will want to enabled netlogon debugging (from command prompt use: nltest /dbflag:0x2080ffff.  Some logon events will show up there, and you can get further source information from the events.  Just search the text files in c:\windows\debug\netlogon* for the user name and look for other machine names in any event that doesn't have a 0x0 status.

You can look at my high level overview for how I wrote my tracking script. For security events, I had to identify some standards between each event ID's I wanted to work with and use insertionstrings to provide a standardized generic PSObject that can be provided by parsing the available information.

Some useful commands to find uses of an account on a local machine (open powershell as administrator, set the target account name without domain)

$targetaccount = "johndoe"

write-host "`n checking services"
gwmi win32_service |select name,startname | where {$_.startname -match $targetaccount}

write-host "`n checking processes"
gwmi win32_process |select name,@{name="ownerdomain"; exp={$_.getowner().domain}},@{name="owneruser" ; expr={$_.getowner().user}}  | where {$_.owneruser -match $targetaccount}

write-host "`n dumping any IIS appPools"
c:\windows\system32\inetsrv\appcmd.exe list apppool /text:* |where {$_ -match "APPPOOL\.NAME|userName"}

write-host "`n checking remote desktop connections"
qwinsta

write-host "`n checking scheduled tasks"
schtasks /query /v -fo CSV | where {$_ -match $targetaccount}

write-host "`n checking mapped network drives for currently logged on user"
dir hkcu:\network