Windows Explored

We recently moved from Java 6 to Java 7, specifically JRE 7 Update 25. Immediately, we began get reports of user accounts getting locked after the affected users visited web sites hosting Java applets. For the most parts, the applets would run until the 5th attempt to load-refresh the applet and then the domain account would get locked. The initial look with Network Monitor showed that authenticated users were failing at the proxy level:

The spot-workaround was to create bypass rules for the individual sites to allow all users to pass without authentication, which was by no means a elegant since it was a reactionary approach  that waited for user’s to get locked across various offices and then report the problem to tier 1 and then escalate.

When the support issues began to settle down, I began to look more deeply into the problem. We took a non-production proxy and removed all the rules that were created over the previous days so that any Java applet would begin failing authentication (I used the Java Verify page as my test). I have to admit, network traffic and protocols are not my Zen, but as I began to look at various captures and figure may way around netmon, I saw the same theme each time, Kerberos authentication failing:

Some research pointed me to this Microsoft KB How to force Kerberos to use TCP instead of UDP in Windows. Even though it didn’t apply in this case (Windows Server 2003/XP and earlier only), it shed some light on the possible issue:

The Windows Kerberos authentication package is the default authentication package in Windows Server 2003, in Windows Server 2008, and in Windows Vista. It coexists with the NTLM challenge/response protocol and is used in instances where both a client and a server can negotiate Kerberos. Request for Comments (RFC) 1510 states that the client should send a User Datagram Protocol (UDP) datagram to port 88 at the IP address of the Key Distribution Center (KDC) when a client contacts the KDC. The KDC should respond with a reply datagram to the sending port at the sender’s IP address. The RFC also states that UDP must be the first protocol that is tried.

Note
RFC 4120 now obsoletes RFC 1510. RFC 4120 specifies that a KDC must accept TCP requests and should listen for such requests on port 88 (decimal). By default, Windows Server 2008 and Windows Vista will try TCP first for Kerberos because the MaxPacketSize default is now 0. You can still use the MaxPacketSize registry value to override that behavior.

By default, the maximum size of datagram packets for which Windows Server 2003 uses UDP is 1,465 bytes. For Windows XP and for Windows 2000, this maximum is 2,000 bytes. Transmission Control Protocol (TCP) is used for any datagrampacket that is larger than this maximum. The maximum size of datagram packets for which UDP is used can be changed by modifying a registry key and value.

In a nutshell, it seems Java 7 falling back on is using Kerberos over UDP instead of TCP. I’m guessing the packet exceeds the size that the UDP protocol can handle, resulting in KRB ERROR KDC_ERR_PREAUTH_FAILED (24), i.e. Preauthentication failed. The initial Kerberos TCP packet is here and exceeds 1465 bytes:

After the initial Kerberos request and response from the DC, the java.exe process gets involved and now the initial TCP packet has been transformed into a UDP packet. The KDC error follows I assume because the packet is possibly getting truncated or has been malformed and the domain account passes a bad password because the data is not sufficient for authentication to continue:

Some additional searching pointed me to this OTN forum post where the same issue was reported and referred to exiting Java bug 8009875. Apparently, the solution is to modify Kerberos config settings. However, this does not seem to be working for the poster (its also likely beyond my scope as I do not have the juice or knowledge yet to explore this as a solution)*. I looked more for a client side solution and eventually found this Microsoft KB article Registry Key to Allow Session Keys to Be Sent in Kerberos Ticket-Granting-Ticket (note, it should be a DWORD, not a string). In short:

Previous to the change that is described in the "Summary" section of this article, programs could use the Win32 LsaCallAuthenticationPackage API specifying KERB_RETRIEVE_TICKET_REQUEST and either KerbRetrieveEncodedTicketMessage or KerbRetrieveTicketMessage message types to retrieve a Kerberos ticket-granting-ticket (TGT) and the associated session key.
The registry value to include a session key in the TGT:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters

Value Name: allowtgtsessionkey
Value Type: REG_SZ DWORD
Value Range: 0 or 1 (default of 0)

After applying on a couple labs, the Java applets were no longer passing any Kerberos traffic, avoiding the failed authentication altogether. That being said, I am not ready to call this a solution. There is no doubt a bug (or a change that needs to be documented properly) in Java 7 as we did not see this in any JRE 6 and it needs to be addressed. Further, I haven’t yet fully grasped the security vulnerability involved in applying this to an enterprise environment with thousand of workstations. Perhaps a Kerberos configuration file may actually resolve in our case but that still needs to be investigated. Due to the potential security impact, we have not enabled this registry workaround yet.

Update

*Creating a krb5.conf file and placing it in C:\Program Files (x86)\Java\jre7\lib\security (or C:\Program Files\Java\jre7\lib\security on 32bit OS) seems to correct this issue. See http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html#SetProps. Use a text editor to create it and add your domain where appropriate: