Wednesday, March 29, 2017
Password change failed: Configuration information could not be read from the domain controller, either because the machine is unavailable, or access has been denied
Tuesday, August 13, 2013
Kerberos SPN configuration errors for dummies
First of all, lets understand what kerberos is doing for us. Authentication, is how we identify ourselves. In the example WEB Server->SQL Server, it could be:
1) a service account on the webserver that is logging into the SQL server
2) The end user (at the browser) authenticating to the webserver and the webserver is set to log into the SQL server on the user's behalf (delegation)
Authentication uses protocols to ensure that the various applications and servers are all speaking the same language. Typically this is NTLM, NTLMv2, or Kerberos v5. Here we will focus on kerberos.
The way kerberos works is, you have a "Service" that you want to access. This "Service" has a type and a host machine that it runs on. Example:
1) Web service on machine Server1.mydomain.com. In Kerberos SPN format: HTTP/server1.mydomain.com
2) MSSQL service on machine ServerSQL1.mydomain.com. In Kerberos SPN Format: MSSqlSvc/ServerSQL1.mydomain.com
There are other variations that include port numbers and domain names, but to keep things simple we will stick to standard ports and windows services here.
So what is the SPN used for? Lets look at it in less technical terms first:
John wants to call Amy on the phone. Amy wants to ensure that the people who call her are really who they say they are. To enable John to meet Amy's requirements, he calls Amy through a phone Operator. The phone operator has a list of names (account), phone numbers (service) and passwords (secrets/keys) for everyone that calls through their system, including John and Amy. John tells the operator his password and the number he is calling, the operator looks up the phone number and the operator gives him a temporary code to use for his conversation. John gets through to Amy on the phone and tells her the code. Amy uses a special program that takes her password and decrypts the temporary code that John got from the operator. If she can decrypt the code, she knows that she is talking to John.
And now for the technical terms. When a client connects to the service, they are told that they need to authenticate. The Client connects to a KDC (Kerberos Key Distribution Center) and to request a ticket. In the windows world, the KDC is a domain controller (Active Directory). During a user's logon (or an application starting running under a service account), the user will log into the KDC to get a Ticket Granting Ticket (TGT). When it wants to connect to a service, the user will sent a request to the KDC for a Service Ticket. The KDC will look through its database to see what account holds the SPN for the service that the user wants to connect to. If it can find one, it will issue a ticket that is encrypted to both the Requestor and the Account with the SPN. The user will then take this ticket, send it back to the application that they are connecting to and the application will review the ticket to grant/deny access. (see the previous article for the step by step)
The problem can come in at this point in several ways. If the SPN was set up on the wrong account...then the ticket is encrypted to the wrong person.
Back to the non-technical example:
When John calls the operator, let us assume there was some bad information in the operators list of names and passwords. The Operator then provides a temporary code that works for Susan. When John gives this code to Amy, Amy can not decrypt the code and will have to reject the phone call.
In another form of this problem, if more than one person have the same phone number (duplicate SPN in kerberos), the operator may look up the wrong name.
To solve these problems, it is important to know
- What accounts (users or computer objects) are in use
- What service they run on
- What servers they are configured on
- Do they run services on non standard ports
- Is there delegation from one service to connect to another service (double hop)
- How does authentication from from end to end (have a diagram or documentation as many of the support people you end up working with do not know anything about your application)
Thursday, December 27, 2012
"System detected a possible attempt to compromise security" Enter network password popup comes up on 2008R2
1) Dns registration failed (secure DDNS)
2) Group policy processing failed: Event 1053 "Could not resolve the user name"
3) LSASRV 40960 events with authentication errors to various kerberos services such as domain controllers LDAP/ SPNS, and cifs/ for the DFS namespace.
4) TerminalServices error 1067, Cannot registery TERMSRV Service Principal Name
5) When joining the domain and changing primary dns suffix "Changing the primary domain DNS name of this computer failed." "A Directory service error has occurred"
6) Klist shows no kerberos tickets
When I looked at this in netmon, all of the Kerberos transactions for TGT requests would receive a preauthentication required error from the KDC (domain controller) and it wasn't following up on that. After trying to dig around for information related to that, and any possible Kerberos settings that might impact this, I could find nothing. I looked in the registry for LSA settings and found LMCompatibilityLevel at 1. After changing this to a 2, everything started to work fine. From the Microsoft description of NT Compatibility levels, I don't see how this would impact Kerberos transactions, but apparently there may be some correlation. This fix worked for a short while, however everything broke again soon after. Later investigation found the Kerberos encryption types had been restricted to AES only, which was not compatible with the domain. After enabling RC4-HMAC, the problems went away. I have seen related issues on other machines where neither of these two should have been the problem. So perhaps there are many causes.
Thursday, October 11, 2012
DES encryption and 2008R2 (does it really work?)
Java.exe "-Djava.security.krb5.kdc=MyDomainController1" "-Djava.security.krb5.conf=c:\windows\krb5.ini" "-Djava.security.krb5.realm=CONTOSO.COM" "-Djava.security.krb5.internal.crypto=DesCbcCRCEType" "sun.security.krb5.internal.tools.Kinit" someuseraccount@CONTOSO.COM UserPassword
If you look at the example above (the quotes are in there for use within powershell), the first -D option points to your KDC, which is the domain controller you want to test. The second points to a krb5.ini file (might not be required, you can search around to find the format of that if needed). The third option: Realm, is the dns name of your domain. Our fourth option is the encryption type you want to test. You can do DesCbcCRCEtype or DesCbcMd5Etype (among others), to test different types of DES encryption ticket requests. The last part of the java class paths is the Kinit class utility, and the two arguments after that are the username and password. These can be any account in the domain, and doesn't require the DES checkbox on the user account to be checked. Run this command and check output. If there are errors, it will give you further details. If DES is not supported on a domain controller, you will get an error 14 talking about encryption type no supported. Otherwise, you will get a kerberos ticket that will be stored in a file.
If you want verbose debugging details for the whole connection, you can add these two options prior to the kinit:
-Dsun.security.krb5.debug=true
-Dsun.security.krb5.krb5debug=all
If you want to test a large number of domain controllers, you can wrap this into a powershell function similar to this:
function check-dessupport {
param(
[string]$server,
[string]$realm,
[string]$user,
[string]$password
)
echo $server >>c:\temp\kerbout.txt
$res = new-object PSObject
add-member -inputobject $res NoteProperty Server $server
Java.exe "-Djava.security.krb5.kdc=$($server)" "-Djava.security.krb5.conf=c:\windows\krb5.ini" "-Djava.security.krb5.realm=$($realm)" "-Djava.security.krb5.internal.crypto=DesCbcCRCEType" "sun.security.krb5.internal.tools.Kinit" $user $password 2>>c:\temp\kerbout.txt
add-member -inputobject $res NoteProperty DesCbcCrc $?
Java.exe "-Djava.security.krb5.kdc=$($server)" "-Djava.security.krb5.conf=c:\windows\krb5.ini" "-Djava.security.krb5.realm=$($realm)" "-Djava.security.krb5.internal.crypto=DesCbcMd5EType" "sun.security.krb5.internal.tools.Kinit" $user $password 2>>c:\temp\kerbout.txt
add-member -inputobject $res NoteProperty DesCbcMd5 $?
return $res
}
This will return the server name and true/false for each time. Just note that the failure of a test can just be the server is not available or is slow to respond. This example logs the full details into c:\temp\kerbout.txt
To check the allowed encryption types that are set by policy you can look at the registry key
function check-enctypeRegEntry {
param(
[string]$server
)
$key = "Software\Microsoft\Windows\CurrentVersion\Policies\System\kerberos\parameters"
$type = [Microsoft.Win32.RegistryHive]::LocalMachine
$regkey = [Microsoft.win32.registrykey]::OpenRemoteBaseKey($type,$server)
$regkey = $regkey.opensubkey($key)
$enctypes = $regkey.getvalue("supportedencryptiontypes")
$ret = New-Object psobject
Add-Member -InputObject $ret NoteProperty Server $server
Add-Member -InputObject $ret NoteProperty EncType $enctypes
return $ret
}
And you will need to decode the numeric value from there. A decimal value of 31 is the value for all check boxes being checked except future encryption types.
For further details on dealing with DES, you can look at my previous article on the subject. If you do have everything enabled and you are at SP1 but still have DES issues, please comment on the post.
Thursday, September 8, 2011
Chasing duplicate SPN's
To understand this problem, here is a basic explanation of the Kerberos authentication flow:
1) User accesses a resource application
2) Resource application tells user to authenticate
3) User connects to domain controller looking for a Kerberos service ticket for that service
4) Domain controller searches for an account with that service principal name
    a) If there is one in the same domain, use that one
    b) If there is more than one in the same domain, results may vary
    c) If there is more than one in multiple domains, results may vary
5) User receives ticket from Domain Controller
6) User presents ticket to resource application
7) Resource application account (computer or service account) attempts to decrypt the ticket to verify it.
    a) If the ticket was encrypted to them, authentication works
    b) If the ticket was encrypted to one of the other duplicate SPN accounts, decryption will fail, and access is denied.
Detection:
All your domain controllers will be logging events when duplicate SPN's are encountered. Unfortunately between 2008 and pre-2008 OS's, the event log source data is different. So when searching event logs you will have to account for this in some way. My example below will pull all the duplicate spn events and just strip out only the conflicted SPN record.
2000/2003 Domain controller:
Get-WMIObject -Computer MyDomainController -Filter "Logfile='system' and eventcode = 11 and sourcename='KDC' and type=5" -Class Win32_NtLogEvent | Foreach-Object { $_.insertionstrings[0]
2008 Domain controller:
Get-WMIObject -Computer MyDomainController -Filter "Logfile='system' and eventcode = 11 and sourcename='Microsoft-Windows-Kerberos-Key-Distribution-Center' and type=5" -Class Win32_NtLogEvent | Foreach-Object { $_.insertionstrings[0] }
Note: This type of query is a bit slow, but better than some methods. If you integrate a timewritten >= ########## this may greatly improve the speed of the event log query against the remote machine.
Take this information and you can use something like queryspn.vbs to look at any specific SPN to see what accounts are configured to use it. After that, analyze which account really needs it, then "setspn -D" the invalid entries away.
Wednesday, January 5, 2011
DES encryption, Kerberos and 2008 Server
Identification of DES usage in your environment:
1) Netmon. Using the older Microsoft version of netmon, you can monitor your domain controllers and look for kerberos traffic that is using encryption other than RC4-HMAC. To do this, run a capture and create a display filter as follows:
AND
|---Kerberos: Encryption type (Etype[0]) <> 0x17
|---Protocol == KERBEROS
|---Any <--> Any
This will show all Kerberos traffic that is not using the standard. This may also find other types such as AES if you are using the latest and greatest. Since there are several types of DES encryption formats for Kerberos, this filter method is the simplest, but you can also create a multiple set of OR statements on that Etype value. Refer to the RFC's to get the values of each.
2) Kinit: If you are already seeing problems with kerberos authentication for certain applications, you can use kinit with debugging options to request a ticket for that service. This can show you if you are getting a bad Encryption type error. Here is one example using java and a keytab:
java -Dsun.security.krb5.debug=true -Dsun.security.krb5.krb5debug=all sun.security.krb5.internal.tools.Kinit -k -t HTTP.keytab HTTP/myservice@MYDOMAIN
-result snip-
Found unsupported keytype (23) for
HTTP/myservice@MYDOMAIN
-/result snip-
3) Searching for DES enabled AD accounts:
Ldap filter: (&(|(objectcategory=user)(objectcategory=computer))(userAccountControl:1.2.840.113556.1.4.804:=2097152))
This will find all DES enabled computer and user accounts in your search scope. You can search the serviceprincipalname attribute to see what applications may be using DES.
4) Kerberos event log events on the DC's
5) Errors in application logs. ETYPE not supported and similar. Some logs will show types of encryption configured. Check java versions (1.4.x and earlier don't support RC4-HMAC). Check krb5.ini and krb5.conf files for encryption type configurations.
Fixing the problem
There are two ways to go about getting around this problem during an upgrade. First, you can try to identify all of the uses of DES and get rid of them, or you can enabled DES support.
To get rid of DES usage, look for capability in the application for RC4-HMAC or other support encryption standard. Check the application, check the versions of java it uses, etc. If it is java, anything 1.5 and higher supports RC4. Check all the krb5.conf files to ensure any supplied enctype values allow for RC4-HMAC
[libdefaults]
default_tkt_enctypes = rc4-hmac
default_tgs_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
Do your testing and push the appropriate application owners to do what they can to get away from DES. If keytab's are in use, ensure they are recreated with the appropriate encryption.
If you have applications that cannot get rid of DES, you can look at the steps required to enable DES support on the OS. There are two parts to this. First you will need to patch your 2008 domain controllers with KB978055. This gives the DC the ability to issue DES tickets. If your clients are windows 7 or 2008R2 server themselves, they will need to have some configuration changes. This can be done by a registry fix, or pushed by group policy. Refer to this article for that. When changing the client settings, be careful that you allow all of the required encryption types. If you use a GPO to turn on DES, and don't specify anything else, your machine will only use DES.
UPDATE (2/11/2012): After having the above patch fail to install with an error that it is not applicable, it seems that the patch was rolled into Windows 2008R2 service pack 1. So if you have the service pack installed, you should be fine on the domain controller side.
Tuesday, March 30, 2010
Tired of looking up security codes in events?
function decode-krbTktOpts([string]$code) {
#code provided is in hex format
if (-not($code -match "0x\d{8}")) {
write-error "Invalid entry sent to decode-krbTktOpts function : value was : $code"
return $null
}
##########
#Ticket options are 32 bits of flags. Code comes in as hex. Only bits 0-1,3-5,16-17,20,23,25-31 are used
##########
$results = new-object collections.arraylist
$code = [convert]::toint32($code.substring($code.indexof("x")+1), 16)
if ($code -and 1) {
$results.add("Validate") >$null
} else if ($code -and 2) {
$results.add("Renew") >$null
} else if ($code -and 8) {
$results.add("EncTktInSKey") >$null
} else if ($code -and 16) {
$results.add("RenewableOK") >$null
} else if ($code -and 32) {
$results.add("DisableTransitedCheck") >$null
} else if ($code -and 65536) {
$results.add("Canonicalize") >$null
} else if ($code -and 131072) {
$results.add("CNameInAddlTkt") >$null
} else if ($code -and 1048576) {
$results.add("OptHardwareAuth") >$null
} else if ($code -and 8388608) {
$results.add("Renewable") >$null
} else if ($code -and 33554432) {
$results.add("AllowPostDate") >$null
} else if ($code -and 67108864) {
$results.add("Proxy") > $null
} else if ($code -and 134217728) {
$results.add("Proxiable") > $null
} else if ($code -and 268435456) {
$results.add("Forwarded") > $null
} else if ($code -and 536870912) {
$results.add("Forwardable") > $null
}
return $results
}
function decode-LogonErrorCode([string]$code) {
#decode 32 bit microsoft logon error codes from Hex format (32 bit)
if (-not($code -match "0x\d{8}")) {
write-error "Invalid entry sent to decode-krbTktOpts function : value was : $code"
return $null
}
switch($code.tolower()) {
"0x0" { return "Successful login" }
"0xC0000064" { return "The specified user does not exist" }
"0xC000006A" { return "The value provided as the current password is not correct" }
"0xC000006C" { return "Password policy not met" }
"0xC000006D" { return "The attempted logon is invalid due to a bad user name"}
"0xC000006E" { return "User account restriction has prevented successful login"}
"0xC000006F" { return "The user account has time restrictions and may not be logged onto at this time"}
"0xC0000070" { return "The user is restricted and may not log on from the source workstation"}
"0xC0000071" { return "The user account's password has expired"}
"0xC0000072" { return "The user account is currently disabled"}
"0xC000009A" { return "Insufficient system resources"}
"0xC0000193" { return "The user's account has expired"}
"0xC0000224" { return "User must change his password before he logs on the first time"}
"0xC0000234" { return "The user account has been automatically locked" }
default {return "Unknown code provided, unable to translate" }
}
}
function decode-krbErrCode([string]$code) {
#code provided is required to be in the hex format provided in the system event logs ex: 0x2
if (-not($code -match "x")) {
#if we receive something in invalid format, try to convert to hex
if ($code -match "\d+") {
$code = "0x" + [string]::format("{0:x}",$code)
} else {
write-error "Invalid entry sent to decode-krbErrCode function : value was : $code"
return $null
}
}
switch($code.tolower()) {
"0x0" { return ("KDC_ERR_NONE","No Error") }
"0x1" { return ("KDC_ERR_NAME_EXP","Clients entry in Database has Expired") }
"0x2" { return ("KDC_ERR_SERVICE_EXP","Servers entry in Database has Expired") }
"0x3" { return ("KDC_ERR_BAD_PVNO","Request protocol version number not supported") }
"0x4" { return ("KDC_ERR_C_OLD_MAST_KVNO","Client's key encrypted in old master key") }
"0x5" { return ("KDC_ERR_S_OLD_MAST_KVNO","Servers key encrypted in old master key") }
"0x6" { return ("KDC_ERR_C_PRINCIPAL_UNKNOWN","Client not found in Kerberos Database") }
"0x7" { return ("KDC_ERR_S_PRINCIPAL_UNKNOWN","Server not found in Kerberos Database") }
"0x8" { return ("KDC_ERR_PRINCIPAL_NOT_UNIQUE","Multiple principal entries in database") }
"0x9" { return ("KDC_ERR_NULL_KEY", "The client or server has a null key") }
"0xa" { return ("KDC_ERR_CANNOT_POSTDATE", "Ticket not eligible for postdating") }
"0xb" { return ("KDC_ERR_NEVER_VALID","Requested start time is later than end time") }
"0xc" { return ("KDC_ERR_POLICY","KDC policy rejects request") }
"0xd" { return ("KDC_ERR_BADOPTION","KDC cannot accomodate requested option") }
"0xe" { return ("KDC_ERR_ETYPE_NOSUPP","Kerberos server has no support for this encryption type") }
"0xf" { return ("KDC_ERR_SUMTYPE_NOSUPP","Kerberos server has no support for checksum type") }
"0x10" { return ("KDC_ERR_PADATA_TYPE_NOSUPP","Kerberos server has no support for PADATA type") }
"0x11" { return ("KDC_ERR_TRTYPE_NOSUPP", "Kerberos server has no support for transited type") }
"0x12" { return ("KDC_ERR_CLIENT_REVOKED","Clients credentials have been revoked") }
"0x13" { return ("KDC_ERR_SERVICE_REVOKED","Credentials for server have been revoked") }
"0x14" { return ("KDC_ERR_TGT_REVOKED","TGT has been revoked") }
"0x15" { return ("KDC_ERR_CLIENT_NOTYET","Client not yet valid") }
"0x16" { return ("KDC_ERR_SERVICE_NOTYET", "Server not yet valid") }
"0x17" { return ("KDC_ERR_KEY_EXPIRED", "Password has expired - change password to reset") }
"0x18" { return ("KDC_ERR_PREAUTH_FAILED","Preauthentication is invalid, bad password") }
"0x19" { return ("KDC_ERR_PREAUTH_REQUIRED","Additional Preauthentication required") }
"0x1f" { return ("KRB_AP_ERR_BAD_INTEGRITY","Integrity check on decrypted field failed") }
"0x20" { return ("KRB_AP_ERR_TKT_EXPIRED","Ticket expired")}
"0x21" { return ("KRB_AP_ERR_TKT_NYV","Ticket not yet valid")}
"0x22" { return ("KRB_AP_ERR_REPEAT","Request is a replay")}
"0x23" { return ("KRB_AP_ERR_NOT_US","The ticket isn't for us")}
"0x24" { return ("KRB_AP_ERR_BADMATCH", "Ticket and authenticator do not match") }
"0x25" { return ("KRB_AP_ERR_SKEW", "Clock skew is too big")}
"0x26" { return ("KRB_AP_ERR_BADADDR", "Incorrect net address") }
"0x27" { return ("KRB_AP_ERR_BADVERSION", "Protocol version mismatch") }
"0x28" { return ("KRB_AP_ERR_MSG_TYPE", "Invalid message type") }
"0x29" { return ("KRB_AP_ERR_MODIFIED", "Message stream modified") }
"0x2a" { return ("KRB_AP_ERR_BADORDER", "Message out of order") }
"0x2c" { return ("KRB_AP_ERR_BADKEYVER","Specified version of key is not available") }
"0x2d" { return ("KRB_AP_ERR_NOKEY", "Service key not available") }
"0x2e" { return ("KRB_AP_ERR_MUT_FAIL", "Mutual authentication failed") }
"0x2f" { return ("KRB_AP_ERR_BADDIRECTION", "Incorrect message direction") }
"0x30" { return ("KRB_AP_ERR_METHOD", "Alternative authentication method required") }
"0x31" { return ("KRB_AP_ERR_BADSEQ", "Incorrect sequence number in message") }
"0x32" { return ("KRB_AP_ERR_INAPP_CKSUM", "Inappropriate type of checksum in message") }
"0x3c" { return ("KRB_ERR_GENERIC", "Generic error") }
"0x3d" { return ("KRB_ERR_FIELD_TOOLONG","Field is too long for this implementation") }
default { return ("Invalid code", "not in RFC") }
}
}
Friday, March 26, 2010
Auditing Kerberos Delegation
In the last few weeks I was doing some audits in my environment for accounts that were trusted for delegation. I thought I would share some LDAP searches with everyone, using Joe's great ADFIND tool. For those that are not too familiar with delegation, there are two different bits in the UserAccountControl attribute that are related to delegation. These are TRUSTED_FOR_DELEGATION (0x80000) which uses kerberos forwardable tickets, and TRUSTED_TO_AUTH_FOR_DELEGATION (0x1000000) which allows the delegated person/computer to request a ticket on a user's behalf. Both of these options should be used with extreme caution when the accounts are unconstrained, the second option even more so. If systems that are delegated in an unconstrained manner get compromised, anyone accessing them is basically giving up their account for any purpose to the compromised machine.
Finding all unconstrained delegated computer and user accounts, ignoring domain controllers.
adfind -h DCservername -b dc=mydomain,dc=com -s subtree -bit -f "(&(|(objectcategory=user)(objectcategory=computer))(|(userAccountControl:OR:=16777216)(userAccountControl:OR:=524288))(!(iscriticalsystemobject=TRUE))(!(msds-allowedtodelegateto=*)))" -t 9000 distinguishedname serviceprincipalname useraccountcontrol
Finding all constained delegated computer and user accounts, ignoring domain controllers.
adfind -h DCservername -b dc=mydomain,dc=com -s subtree -bit -f "(&(|(objectcategory=user)(objectcategory=computer))(|(userAccountControl:OR:=16777216)(userAccountControl:OR:=524288))(!(iscriticalsystemobject=TRUE))(msds-allowedtodelegateto=*))" -t 9000 distinguishedname serviceprincipalname useraccountcontrol
Monday, January 25, 2010
mod_auth_vas woes
So we went about the proper process for account creation and moved it to a container, set up all the permissions for the person running the script, and the problems began. The script kept hitting an error of the object already existing. Digging around in the script code pointed to vastool service create. Checking this command only gave an option to create and remove, but not modify. Googling for an answer did not give up anything useful. Most of the discussion was related to the script and how to get it working in certain cases.
Eventually I ran into a document for another Vintella product documentation which had some more detail discussion about setting up the keytab. So after a few attempts, we found a working solution with this:
1) edit AD account UPN to use SPN format
2) setspn -A HTTP/fqdn.of.server Domain\AD-username
3) ktpass -princ HTTP/fqdn.of.server@DOMAIN.DOMAIN.DOMAIN -mapuser AD-username@DOMAIN.DOMAIN.DOMAIN -crypto RC4-HMAC-NT -pass
4) copy the keytab file over to your Vas configuration folder, chgrp daemon HTTP.keytab, chmod 640
5) Configure httpd.conf for mod_auth_vas if it is not done already.