Thursday, May 28, 2026

Unable to login to Katalon Studio in certain enterprise environment conditions

 Another year, another post.

In this episode, my team member who was tasked with scripting up our regression tests in Katalon, had the opportunity to upgrade from v10 to v11.1.3 recently. We'd hopefully be able to take advantage of the newer features and explore any improvements it had to offer. The older version could still run with no issues though. We can also load up their websites just fine on our Chrome browser on that machine.

The installation was just a quick unpacking from their archive. But the built-in login could not connect, since the software had to call home to verify our license. The error came in the form of 

Unable to connect to https://testops.katalon.io.

Please verify your internet connection and Advanced settings, then try again. 

We had a bit of back and forth with their support, they guided us to witness a "fresh" installation, concluding supposedly with suspicions attributed to the ZScaler appliance that fronted our networks. 

We followed their instructions, adding the TLS certificates into the software keystore in our machine. We tried both the root and intermediate certs. It still didn't work. 

They suggested that we point the trust store to use Windows-ROOT instead. Another update involved adding -Djdk.tls.namedGroups for a series of ECC curves next. Their logs still reported a stack trace. It was for a SSLHandshakeException that noted to have "Received fatal alert: handshake_failure" as the complaint. 

 My team spent some time on their own to investigate. Full credits to my team member, a viable resolution was discovered, after feeding the configuration settings of the software to ChatGPT. The primary suspects flagged were these 3 properties observed in the .ini file:

-Djsse.enableSNIExtension=false

-Dsun.security.ssl.allowUnsafeRenegotiation=true

-Dsun.security.ssl.allowLegacyHelloMessages=true 

 The problem went away, and we were able to continue using Katalon Studio. We updated the finding to the support and the ticket was closed.

But wait there's more! Those lines were not found in the .ini file that came out of the box when their installer was unpacking. And neither did my team added those lines in. While I was doing a bit more investigation into this mystery, I executed the installer once more. The first line appeared again on its own. 

By some odd coincidence, it's possible that the framework or some library that their internals were using, had decided to re-append this flag on its own, for our installation. 

The property jsse.enableSNIExtension is part of the Java Secure Socket Extension "JSSE". It is used to toggle (turn off) the extension for Server Name Indication, which in our situation could be due to the client software that's part of Katalon Studio, deciding that our enterprise network was not modern enough, so it tries to compensate by turning off the capability... which inadvertently makes things worse off.

 When I tried installing the same package on my own machine however, the properties did not appear. So it's possible that there are specific conditions that the internals check on, only then will this extremely niche issue will crop up.

No comments:

Post a Comment