Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Recovering from an Invalid System Password

Recovering from an Invalid System Password

You might not want to admit it, but have you ever lost or forgotten the password for the system user within WebLogic Server (WLS) 6.1? Or worse yet, accidentally deleted the fileRealm. properties or SerializedSystemIni.dat files?

If so, you know the consequence - failure to produce a valid system password at server startup prevents the server instance from booting. This article shows you how to recover from this type of situation, keeping production downtime to a minimum.

Losing the System Password
As most administrators know, system password management within the production environment is critical. Root passwords are required to manage a wide array of operational resources, ranging from operating systems to network routers. A production environment is directly impacted when a key system password is lost, rendering the affected resource inaccessible until an administrator recovers or replaces the password.

With WLS, losing the system password within the default file security realm will prevent a server instance from starting up, blocking its boot process until the system password is reset or recovered. The problem is further complicated by the fact that WLS password management is promoted by the browser-based administration console, a valuable interface constrained by the limitation that the server must be up and running in order for it to be accessible. What options are available when the system password is lost but can't be reset from the console? The answer lies in a command-line feature hidden within the file realm itself.

Elements of the File Realm

Security realms play an important role within WLS 6.1; they define the set of users, passwords, groups, and access control lists used by a domain for its base security information. While various types of security realms exist, WLS by default uses the file realm unless configured otherwise.

For a given domain utilizing the file realm, two crucial files work together and constitute the underlying repository of the security realm. The first, a text file, named fileRealm.properties, persistently defines the actual set of users, hashed passwords, groups, and access control lists of the realm. The second, a binary file named SerializedSystemIni.dat, defines the seed input used by the file realm when a cleartext password is hashed in its SHA-based format. Both files exist within the root directory of the given domain (e.g., ./config/mydomain).

Since the SerializedSystemIni.dat file encapsulates a time-variant encrypted key created when a domain is first generated, no two SerializedSystemIni.dat files are created alike for a specific server installation. Therefore, when a cleartext password is hashed for a certain user and stored within the fileRealm.properties file, an explicit relationship is immediately established between the fileRealm.properties repository and its input SerializedSystemIni.dat file. The password stored in the repository and hashed with the SerializedSystemIni.dat input can be successfully compared only to its cleartext equivalent hashed with the same SerializedSystemIni.dat. This element is critical to the validation process of password comparison, where a cleartext password is compared to a stored password at the hash level, since it's impossible to directly reverse the hashed password back out to its original cleartext format.

An Invalid Password - More Than Just Forgetful
With WLS 6.1, the system password can be invalidated in a few ways. Aside from the obvious cases where the password is forgotten or misplaced (considered a lost password), the system password for the file realm of a given domain also becomes invalid when one of the following situations occurs:

  • Errant password modification: The fileRealm.properties file has been modified in such a way that an errant change is introduced to the hashed password value for the system user (intentional or otherwise).
  • Mismatched seed and repository: The SerializedSystemIni.dat file has been modified, replaced, or removed entirely, introducing a change that breaks the relationship between a given salt file and the set of passwords stored within a specific fileRealm.properties file.

An errant modification to a password is equivalent to password corruption. It usually results from one of two possibilities, differing only by the intent - an accidental change by an administrator who inadvertently modifies a password during an edit session, or a planned attack by a malicious user who intentionally corrupts a password. In both scenarios, the original hashed value of the password is changed at the character level so that a new hash is created, making it impossible to determine what the original cleartext of the new hash is and rendering the password invalid.

In the scenario involving a mismatched seed and repository, the relationship between a given salt file and the set of passwords stored within a specific fileRealm.properties file is broken as a result of the SerializedSystemIni.dat seed file having been modified, removed, or replaced entirely by another file (seed or otherwise). When the seed file becomes invalid with respect to its relationship in this manner, password comparison at the hash level will fail, since passwords stored within a given fileRealm.properties file and initially hashed with one salt are now compared to passwords hashed with another salt.

Available Recovery Options
When the system password is invalidated by one of the above possibilities, the recovery options are limited. If the password has been lost or corrupted, an administrator can restore the password from a backup copy of the fileRealm.properties file, if one exists. If a seed file is invalidated and its relationship to the set of passwords within a given repository has been broken, an administrator can restore the salt from a backup copy of the SerializedSystemIni.dat file - again, if the backup exists.

What can be done, however, when backup copies of the seed and repository files don't exist for the affected domain? You can copy the hashed passwords and appropriate files from the file realm of another domain (where the values of those passwords are known) when another domain exists, or create a new domain by installing a clean instance of WLS and copying from that. Of course, this has problems in its own right, as it depends upon the existence of another domain or the creation of a new domain through installation, and involves a fair amount of manual interaction. Besides, you might not want to risk compromising a production environment with passwords from another configuration.

When backup restoration isn't available and domain copies are not possible or desired, the best recovery option is to reset the password from the command line using the features of a hidden utility within the file realm itself. Mastering this utility will ensure proper recovery from disaster and enable a production environment to operate with little interruption. The rest of this article discusses in detail this utility - the FileRealm class.

The FileRealm Class
Every type of security realm within WLS 6.1 is ultimately managed and represented by a pure Java class. Such classes are either provided with the product out of the box or are externally developed to meet specific security needs. The default file realm is no exception, and is governed by the internal weblogic.security.acl.internal.FileRealm class that ships with WLS (packaged as part of the core weblogic.jar file). The primary responsibilities of this class are to manage the users, passwords, groups, and access control lists of the file-based security realm and provide authentication and authorization services for clients, using the underlying fileRealm.properties file as its persistent security repository and master record of data.

By all accounts, the FileRealm class should offer nothing more than a set of interface methods that other internal components of WLS can publicly use when authenticating and authorizing client access within the security realm - and it does. Interestingly enough, however, the FileRealm class also exposes a main method, allowing it to be invoked from the command line as a normal executable program. As a result, the internal class is surprisingly dual-purposed, serving primarily as a supporting library class to other elements within WLS, yet unexpectedly promoting itself as a command-line utility at the same time.

When executed from the command line, the FileRealm class provides a mechanism by which a cleartext password for a given user can be hashed with a specified input seed file, producing an encrypted equivalent output that can be stored within the fileRealm.properties repository and subsequently referenced by the file realm as the actual password for the given user. In this way, the class effectively gives administrators the ability to set passwords from the command line for any user within the file realm, free of the limitations imposed by the browser-based administration console and the password management system it offers. The potential value offered by this internal class is only fully realized in the most disastrous of situations, where the system password has been lost or invalidated and must be reset from the command line when the server consequently fails to start.

Resetting the System Password
Utilizing the command-line feature of the FileRealm class to set or change a password for a user within the file-based security realm is relatively straightforward. You must first identify the set of users that require password changes - the FileRealm class allows you to consolidate all password changes for multiple users at once, so executing the program in multiple iterations to handle a batch of users on an individual basis isn't required. However, in the example below we'll change the password for only one user (the system user).

When the set of users has been identified, a properties file must be created, and the users need to be defined within this file. Comments are allowed within the file (preceded by the # sign), and for each user within the set, a corresponding entry must be defined on its own new line within the file using the following format:

user.<username>=<cleartext_password>

An example follows:

# define the user system to have the
cleartext password weblogic
user.system=weblogic

This file provides the input list of users for the FileRealm class to process and associate each user with its original cleartext password. For this reason, the properties file can also be referred to as the input definition file. This file can be located anywhere in the local file system, and must end with the .src extension. In the example used below, we'll define the input definition file as user.properties.src.

When the input definition file has been created, the location of SerializedSystemIni.dat must be determined before the FileRealm class is executed from the command line. Recall that SerializedSystemIni.dat provides an input seed (or salt) to the hashing phase of the encryption process, and that a password hashed with a specific salt can be successfully compared only to another password hashed by the same exact salt. Therefore, the SerializedSystemIni.dat file, localized at the domain level and present within the config/ directory, has an explicit relationship to the passwords it hashes for the file realm of a given domain. Since WLS can't maintain a file realm in which the set of hashed passwords present have been seeded by different SerializedSystemIni.dat files, you should never mix passwords hashed by different salt files within the same file realm.

Depending upon the situation and the specifics of your problem, the FileRealm class can use an existing SerializedSystemIni.dat file for its input seed or create a new one if none exists. Referencing an existing SerializedSystemIni.dat is beneficial in the scenario in which the system password for a given domain has been lost but the domain-level salt file is still present and valid. This forces the FileRealm class to create a new password with the same seed used to hash all other passwords within the domain. Creating a new SerializedSystemIni.dat is useful when the salt file has been deleted or corrupted and the relationship to the passwords hashed by the original salt has been broken, making them invalid. Note that when a new SerializedSystemIni.dat file is created by the FileRealm class and used to hash new passwords for a domain, all other passwords within that same domain must be reset as well, since the seed for the hash has changed.

Executing the FileRealm class is simple and straightforward. Set your system classpath to include the weblogic.jar file from the lib directory of the WLS installation and invoke the utility from the command line as follows:

java weblogic.security.acl.internal.FileRealm \ <path_to_output_file> \ <path_to_salt_file>

Upon processing the input definition file, the FileRealm class generates its resultant set of users and their hashed passwords into the output file defined by the first parameter. This file is identical to the input definition file in every way except for the format of the passwords - the passwords in the output file are hashed, while those from the input are cleartext. Of course, this is the desired result. The path to the output file given by the first parameter should be identical to that of the input definition file, with the .src extension dropped for the output file. For example, if you had created the input definition file from above at /tmp/user.properties.src, you'd now need to define the output file at /tmp/user.properties. This accommodates the way the FileRealm class internally handles the location of the input definition file - it concludes its location by using the same path and name of the output file, yet it assumes the input file has the .src extension added to it.

The second parameter defines the location of the SerializedSystemIni.dat file. If the relative or absolute path specified references an existing salt file, then that seed is used as the input. If not, a new seed is created at the path specified with the given filename. Note that while it's possible to reference or create a new salt file with a name other than SerializedSystemIni.dat and outside of a given domain directory, WLS will recognize at runtime only seeds named SerializedSystemIni.dat and placed within their appropriate domain directories.

Upon execution, the FileRealm class will generate its hashed password outputs into the file defined by the first input parameter. When the command-line utility has completed, the invalid passwords found within the fileRealm.properties repository of the affected file realm should be overwritten with the new passwords stored within the output file. This last step should be done manually using your favorite text editor and a few select copy/paste operations. Once the changes have been made to fileRealm.properties and the file has been saved, your invalid passwords are valid once again, and must be verified. Start the server and provide the password recently set for the system user. If the password recovery was successful, the server will properly start and WLS will operate as expected. When you're finished, remove the input definition file and its output equivalent from the filesystem to avoid compromising your newly set passwords.

Conclusion
The browser-based administration console, an excellent utility constrained by the requirement of a running server, is the supported mechanism for password management within WLS 6.1. When the system password has been lost or invalidated and WLS won't start, however, the console becomes inaccessible and the only reliable option for resetting a file realm password is to do so from the command line. Utilizing the hidden features of the FileRealm class, you can manage passwords from the command line and recover from an otherwise disastrous situation. Additionally, while the hidden features of the FileRealm class discussed within this article relate to WLS 6.1, they are fully accessible and available under WLS 7.0, and should be used for the same purposes when running in the 6.1-compatible security mode.

More Stories By Steve Mueller

Steve Mueller is a principal
consultant for BEA Systems, where he specializes in the design,
development, and administration of enterprise systems running on WebLogic Server.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
Cell networks have the advantage of long-range communications, reaching an estimated 90% of the world. But cell networks such as 2G, 3G and LTE consume lots of power and were designed for connecting people. They are not optimized for low- or battery-powered devices or for IoT applications with infrequently transmitted data. Cell IoT modules that support narrow-band IoT and 4G cell networks will enable cell connectivity, device management, and app enablement for low-power wide-area network IoT. B...
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully ...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...