Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Recovering from an Invalid System Password

Recovering from an Invalid System Password

You might not want to admit it, but have you ever lost or forgotten the password for the system user within WebLogic Server (WLS) 6.1? Or worse yet, accidentally deleted the fileRealm. properties or SerializedSystemIni.dat files?

If so, you know the consequence - failure to produce a valid system password at server startup prevents the server instance from booting. This article shows you how to recover from this type of situation, keeping production downtime to a minimum.

Losing the System Password
As most administrators know, system password management within the production environment is critical. Root passwords are required to manage a wide array of operational resources, ranging from operating systems to network routers. A production environment is directly impacted when a key system password is lost, rendering the affected resource inaccessible until an administrator recovers or replaces the password.

With WLS, losing the system password within the default file security realm will prevent a server instance from starting up, blocking its boot process until the system password is reset or recovered. The problem is further complicated by the fact that WLS password management is promoted by the browser-based administration console, a valuable interface constrained by the limitation that the server must be up and running in order for it to be accessible. What options are available when the system password is lost but can't be reset from the console? The answer lies in a command-line feature hidden within the file realm itself.

Elements of the File Realm

Security realms play an important role within WLS 6.1; they define the set of users, passwords, groups, and access control lists used by a domain for its base security information. While various types of security realms exist, WLS by default uses the file realm unless configured otherwise.

For a given domain utilizing the file realm, two crucial files work together and constitute the underlying repository of the security realm. The first, a text file, named fileRealm.properties, persistently defines the actual set of users, hashed passwords, groups, and access control lists of the realm. The second, a binary file named SerializedSystemIni.dat, defines the seed input used by the file realm when a cleartext password is hashed in its SHA-based format. Both files exist within the root directory of the given domain (e.g., ./config/mydomain).

Since the SerializedSystemIni.dat file encapsulates a time-variant encrypted key created when a domain is first generated, no two SerializedSystemIni.dat files are created alike for a specific server installation. Therefore, when a cleartext password is hashed for a certain user and stored within the fileRealm.properties file, an explicit relationship is immediately established between the fileRealm.properties repository and its input SerializedSystemIni.dat file. The password stored in the repository and hashed with the SerializedSystemIni.dat input can be successfully compared only to its cleartext equivalent hashed with the same SerializedSystemIni.dat. This element is critical to the validation process of password comparison, where a cleartext password is compared to a stored password at the hash level, since it's impossible to directly reverse the hashed password back out to its original cleartext format.

An Invalid Password - More Than Just Forgetful
With WLS 6.1, the system password can be invalidated in a few ways. Aside from the obvious cases where the password is forgotten or misplaced (considered a lost password), the system password for the file realm of a given domain also becomes invalid when one of the following situations occurs:

  • Errant password modification: The fileRealm.properties file has been modified in such a way that an errant change is introduced to the hashed password value for the system user (intentional or otherwise).
  • Mismatched seed and repository: The SerializedSystemIni.dat file has been modified, replaced, or removed entirely, introducing a change that breaks the relationship between a given salt file and the set of passwords stored within a specific fileRealm.properties file.

An errant modification to a password is equivalent to password corruption. It usually results from one of two possibilities, differing only by the intent - an accidental change by an administrator who inadvertently modifies a password during an edit session, or a planned attack by a malicious user who intentionally corrupts a password. In both scenarios, the original hashed value of the password is changed at the character level so that a new hash is created, making it impossible to determine what the original cleartext of the new hash is and rendering the password invalid.

In the scenario involving a mismatched seed and repository, the relationship between a given salt file and the set of passwords stored within a specific fileRealm.properties file is broken as a result of the SerializedSystemIni.dat seed file having been modified, removed, or replaced entirely by another file (seed or otherwise). When the seed file becomes invalid with respect to its relationship in this manner, password comparison at the hash level will fail, since passwords stored within a given fileRealm.properties file and initially hashed with one salt are now compared to passwords hashed with another salt.

Available Recovery Options
When the system password is invalidated by one of the above possibilities, the recovery options are limited. If the password has been lost or corrupted, an administrator can restore the password from a backup copy of the fileRealm.properties file, if one exists. If a seed file is invalidated and its relationship to the set of passwords within a given repository has been broken, an administrator can restore the salt from a backup copy of the SerializedSystemIni.dat file - again, if the backup exists.

What can be done, however, when backup copies of the seed and repository files don't exist for the affected domain? You can copy the hashed passwords and appropriate files from the file realm of another domain (where the values of those passwords are known) when another domain exists, or create a new domain by installing a clean instance of WLS and copying from that. Of course, this has problems in its own right, as it depends upon the existence of another domain or the creation of a new domain through installation, and involves a fair amount of manual interaction. Besides, you might not want to risk compromising a production environment with passwords from another configuration.

When backup restoration isn't available and domain copies are not possible or desired, the best recovery option is to reset the password from the command line using the features of a hidden utility within the file realm itself. Mastering this utility will ensure proper recovery from disaster and enable a production environment to operate with little interruption. The rest of this article discusses in detail this utility - the FileRealm class.

The FileRealm Class
Every type of security realm within WLS 6.1 is ultimately managed and represented by a pure Java class. Such classes are either provided with the product out of the box or are externally developed to meet specific security needs. The default file realm is no exception, and is governed by the internal weblogic.security.acl.internal.FileRealm class that ships with WLS (packaged as part of the core weblogic.jar file). The primary responsibilities of this class are to manage the users, passwords, groups, and access control lists of the file-based security realm and provide authentication and authorization services for clients, using the underlying fileRealm.properties file as its persistent security repository and master record of data.

By all accounts, the FileRealm class should offer nothing more than a set of interface methods that other internal components of WLS can publicly use when authenticating and authorizing client access within the security realm - and it does. Interestingly enough, however, the FileRealm class also exposes a main method, allowing it to be invoked from the command line as a normal executable program. As a result, the internal class is surprisingly dual-purposed, serving primarily as a supporting library class to other elements within WLS, yet unexpectedly promoting itself as a command-line utility at the same time.

When executed from the command line, the FileRealm class provides a mechanism by which a cleartext password for a given user can be hashed with a specified input seed file, producing an encrypted equivalent output that can be stored within the fileRealm.properties repository and subsequently referenced by the file realm as the actual password for the given user. In this way, the class effectively gives administrators the ability to set passwords from the command line for any user within the file realm, free of the limitations imposed by the browser-based administration console and the password management system it offers. The potential value offered by this internal class is only fully realized in the most disastrous of situations, where the system password has been lost or invalidated and must be reset from the command line when the server consequently fails to start.

Resetting the System Password
Utilizing the command-line feature of the FileRealm class to set or change a password for a user within the file-based security realm is relatively straightforward. You must first identify the set of users that require password changes - the FileRealm class allows you to consolidate all password changes for multiple users at once, so executing the program in multiple iterations to handle a batch of users on an individual basis isn't required. However, in the example below we'll change the password for only one user (the system user).

When the set of users has been identified, a properties file must be created, and the users need to be defined within this file. Comments are allowed within the file (preceded by the # sign), and for each user within the set, a corresponding entry must be defined on its own new line within the file using the following format:


An example follows:

# define the user system to have the
cleartext password weblogic

This file provides the input list of users for the FileRealm class to process and associate each user with its original cleartext password. For this reason, the properties file can also be referred to as the input definition file. This file can be located anywhere in the local file system, and must end with the .src extension. In the example used below, we'll define the input definition file as user.properties.src.

When the input definition file has been created, the location of SerializedSystemIni.dat must be determined before the FileRealm class is executed from the command line. Recall that SerializedSystemIni.dat provides an input seed (or salt) to the hashing phase of the encryption process, and that a password hashed with a specific salt can be successfully compared only to another password hashed by the same exact salt. Therefore, the SerializedSystemIni.dat file, localized at the domain level and present within the config/ directory, has an explicit relationship to the passwords it hashes for the file realm of a given domain. Since WLS can't maintain a file realm in which the set of hashed passwords present have been seeded by different SerializedSystemIni.dat files, you should never mix passwords hashed by different salt files within the same file realm.

Depending upon the situation and the specifics of your problem, the FileRealm class can use an existing SerializedSystemIni.dat file for its input seed or create a new one if none exists. Referencing an existing SerializedSystemIni.dat is beneficial in the scenario in which the system password for a given domain has been lost but the domain-level salt file is still present and valid. This forces the FileRealm class to create a new password with the same seed used to hash all other passwords within the domain. Creating a new SerializedSystemIni.dat is useful when the salt file has been deleted or corrupted and the relationship to the passwords hashed by the original salt has been broken, making them invalid. Note that when a new SerializedSystemIni.dat file is created by the FileRealm class and used to hash new passwords for a domain, all other passwords within that same domain must be reset as well, since the seed for the hash has changed.

Executing the FileRealm class is simple and straightforward. Set your system classpath to include the weblogic.jar file from the lib directory of the WLS installation and invoke the utility from the command line as follows:

java weblogic.security.acl.internal.FileRealm \ <path_to_output_file> \ <path_to_salt_file>

Upon processing the input definition file, the FileRealm class generates its resultant set of users and their hashed passwords into the output file defined by the first parameter. This file is identical to the input definition file in every way except for the format of the passwords - the passwords in the output file are hashed, while those from the input are cleartext. Of course, this is the desired result. The path to the output file given by the first parameter should be identical to that of the input definition file, with the .src extension dropped for the output file. For example, if you had created the input definition file from above at /tmp/user.properties.src, you'd now need to define the output file at /tmp/user.properties. This accommodates the way the FileRealm class internally handles the location of the input definition file - it concludes its location by using the same path and name of the output file, yet it assumes the input file has the .src extension added to it.

The second parameter defines the location of the SerializedSystemIni.dat file. If the relative or absolute path specified references an existing salt file, then that seed is used as the input. If not, a new seed is created at the path specified with the given filename. Note that while it's possible to reference or create a new salt file with a name other than SerializedSystemIni.dat and outside of a given domain directory, WLS will recognize at runtime only seeds named SerializedSystemIni.dat and placed within their appropriate domain directories.

Upon execution, the FileRealm class will generate its hashed password outputs into the file defined by the first input parameter. When the command-line utility has completed, the invalid passwords found within the fileRealm.properties repository of the affected file realm should be overwritten with the new passwords stored within the output file. This last step should be done manually using your favorite text editor and a few select copy/paste operations. Once the changes have been made to fileRealm.properties and the file has been saved, your invalid passwords are valid once again, and must be verified. Start the server and provide the password recently set for the system user. If the password recovery was successful, the server will properly start and WLS will operate as expected. When you're finished, remove the input definition file and its output equivalent from the filesystem to avoid compromising your newly set passwords.

The browser-based administration console, an excellent utility constrained by the requirement of a running server, is the supported mechanism for password management within WLS 6.1. When the system password has been lost or invalidated and WLS won't start, however, the console becomes inaccessible and the only reliable option for resetting a file realm password is to do so from the command line. Utilizing the hidden features of the FileRealm class, you can manage passwords from the command line and recover from an otherwise disastrous situation. Additionally, while the hidden features of the FileRealm class discussed within this article relate to WLS 6.1, they are fully accessible and available under WLS 7.0, and should be used for the same purposes when running in the 6.1-compatible security mode.

More Stories By Steve Mueller

Steve Mueller is a principal
consultant for BEA Systems, where he specializes in the design,
development, and administration of enterprise systems running on WebLogic Server.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

IoT & Smart Cities Stories
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...
If a machine can invent, does this mean the end of the patent system as we know it? The patent system, both in the US and Europe, allows companies to protect their inventions and helps foster innovation. However, Artificial Intelligence (AI) could be set to disrupt the patent system as we know it. This talk will examine how AI may change the patent landscape in the years to come. Furthermore, ways in which companies can best protect their AI related inventions will be examined from both a US and...
Bill Schmarzo, Tech Chair of "Big Data | Analytics" of upcoming CloudEXPO | DXWorldEXPO New York (November 12-13, 2018, New York City) today announced the outline and schedule of the track. "The track has been designed in experience/degree order," said Schmarzo. "So, that folks who attend the entire track can leave the conference with some of the skills necessary to get their work done when they get back to their offices. It actually ties back to some work that I'm doing at the University of San...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...