Wednesday, August 21, 2013

Wireless - No networks available (a windows services story)

This morning, I was turning on the company laptop to get ready to start my working day.  Sadly my wifi connection icon was not detecting networks at all (not even trying).  Digging around in the services, I found Wireless autoconfig not running due to Extensible Authentication Protocol dependency failure.  EAP wouldn't start either due to CNG Key Isolation dependency failure.  CNG wouldn't start due to:

Service Control Manager Event 7000
The CNG Key Isolation service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.

After this it occurred to me that I may have broken by system while connected to a LAN wire the day before.  I had been trying to troubleshoot show smartcard -> mstsc.exe client device redirection interactions, and part of this effort had me isolating windows services to separate processes.  So with this, two things learned

1) CNG Key Isolation service needs to be in a shared process and not its own process
2) EAP needs to be in its own process, and won't allow you to configure it as shared.

For those that may have no idea what I'm talking about with shared/own process in related to service, you can see what I'm talking about in task manager, or tasklist /svc.  In task manager you see processes called svchost.exe, which when you right click and say "go to services", it flips you to the services tab and shows some highlighted service names.  These are the services that are attached to that process.  Windows will often stuff several service processes into a single process id.  In some cases this can be a problem, where one service failing can cause multiple services to enter a stopped state of the process crashes.  But in my case, I was trying to isolate activity to individual services using Sysinternal's ProcMon tool.  When you isolate the processes to their "own" process, you can see them as individual PID's here and also in netmon, or other tools that do process monitoring/debugging.  If you ever find yourself wanting to play around with this, you can use:

sc.exe config [servicename] type= own
sc.exe config [servicename] type= shared

Servicename is the actual service name, not the cute display name that most people know the service by.  I add the whole sc.exe (with extension) in case you run it from powershell [sc is an alias for set-content, alias has priority over exe files in the %path%].  Also, mind the space between the equals sign and the process type.  There are a few other service type's which may have some use in other ways, but don't ask me what they do.

Tuesday, August 13, 2013

Kerberos SPN configuration errors for dummies

In a previous article, I had written about the problem of duplicate Kerberos SPN's (Service principal names) and how to identify them.  Since then, I notice a recurring theme in my life where application and database people typically don't understand authentication configurations at all.  As a result, accounts get swapped out, configuration changes are made without any thoughts to what will work, and so on.  In the end the whole application environment may have downgraded itself to NTLM or just stopped working altogether.  So, I thought I would take another shot at trying to simplify kerberos interactions for the typical application web server talking to a database server.

First of all, lets understand what kerberos is doing for us.  Authentication, is how we identify ourselves.  In the example WEB Server->SQL Server, it could be:

1) a service account on the webserver that is logging into the SQL server
2) The end user (at the browser) authenticating to the webserver and the webserver is set to log into the SQL server on the user's behalf (delegation)

Authentication uses protocols to ensure that the various applications and servers are all speaking the same language.  Typically this is NTLM, NTLMv2, or Kerberos v5.  Here we will focus on kerberos.

The way kerberos works is, you have a "Service" that you want to access.  This "Service" has a type and a host machine that it runs on.  Example:

1) Web service on machine  In Kerberos SPN format:   HTTP/
2) MSSQL service on machine  In Kerberos SPN Format:    MSSqlSvc/

There are other variations that include port numbers and domain names, but to keep things simple we will stick to standard ports and windows services here.

So what is the SPN used for?  Lets look at it in less technical terms first:

John wants to call Amy on the phone.  Amy wants to ensure that the people who call her are really who they say they are.  To enable John to meet Amy's requirements, he calls Amy through a phone Operator.  The phone operator has a list of names (account), phone numbers (service) and passwords (secrets/keys) for everyone that calls through their system, including John and Amy.  John tells the operator his password and the number he is calling, the operator looks up the phone number and the operator gives him a temporary code to use for his conversation.  John gets through to Amy on the phone and tells her the code.  Amy uses a special program that takes her password and decrypts the temporary code that John got from the operator.  If she can decrypt the code, she knows that she is talking to John.

And now for the technical terms.  When a client connects to the service, they are told that they need to authenticate.  The Client connects to a KDC (Kerberos Key Distribution Center) and to request a ticket.  In the windows world, the KDC is a domain controller (Active Directory).  During a user's logon (or an application starting running under a service account), the user will log into the KDC to get a Ticket Granting Ticket (TGT).  When it wants to connect to a service, the user will sent a request to the KDC for a Service Ticket.  The KDC will look through its database to see what account holds the SPN for the service that the user wants to connect to.  If it can find one, it will issue a ticket that is encrypted to both the Requestor and the Account with the SPN.  The user will then take this ticket, send it back to the application that they are connecting to and the application will review the ticket to grant/deny access.  (see the previous article for the step by step)

The problem can come in at this point in several ways.  If the SPN was set up on the wrong account...then the ticket is encrypted to the wrong person.

Back to the non-technical example:

When John calls the operator, let us assume there was some bad information in the operators list of names and passwords.  The Operator then provides a temporary code that works for Susan.  When John gives this code to Amy, Amy can not decrypt the code and will have to reject the phone call.

In another form of this problem, if more than one person have the same phone number (duplicate SPN in kerberos), the operator may look up the wrong name.

To solve these problems, it is important to know

  1. What accounts (users or computer objects) are in use
    1. What service they run on
    2. What servers they are configured on
    3. Do they run services on non standard ports
  2. Is there delegation from one service to connect to another service (double hop)
  3. How does authentication from from end to end (have a diagram or documentation as many of the support people you end up working with do not know anything about your application)
From here you can search for the SPN's that would be in use to look for duplicates.  While searching for duplicates, you can find where the SPN's are assigned.  If the SPN's are assigned to the wrong accounts, then obviously it won't work.  Make sure you get things in the right places, and try to avoid changing things once it is set up and working.  Document, Document, Document, and update the document.  Avoid running multiple services and application environments on the same account.

Symptoms of duplication SPN's
1) Log events on domain controllers pointing out the duplicate SPN
2) SCOM alerts from the AD management pack for the duplicate SPN alerts in #1
3) Application running NTLM authentication when it was configured for kerberos
4) Application working some times, and giving access denied at other times

Symptoms of incorrectly assigned SPN
1) Authentication fails all the time.

Symptoms of missing SPN
1) Authentication fails completely
2) Authentication is using NTLM

Tools to use:
1) Queryspn.vbs script from microsoft
2) Setspn
3) Event viewer on multiple machines
4) klist (to view kerberos ticket, or lack of one after connecting to an application)
5) fiddler or some other similar web debugging tool that can show authentication details in the packets to show protocol type
6) Netmon to view kerberos KDC interactions to find any errors (SPN not found, encryption type not supported, etc)
7) Increased debug logging in microsoft OS.  Can turn on kerberos debugging for all machines involved.