Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Recovering from an Invalid System Password

Recovering from an Invalid System Password

You might not want to admit it, but have you ever lost or forgotten the password for the system user within WebLogic Server (WLS) 6.1? Or worse yet, accidentally deleted the fileRealm. properties or SerializedSystemIni.dat files?

If so, you know the consequence - failure to produce a valid system password at server startup prevents the server instance from booting. This article shows you how to recover from this type of situation, keeping production downtime to a minimum.

Losing the System Password
As most administrators know, system password management within the production environment is critical. Root passwords are required to manage a wide array of operational resources, ranging from operating systems to network routers. A production environment is directly impacted when a key system password is lost, rendering the affected resource inaccessible until an administrator recovers or replaces the password.

With WLS, losing the system password within the default file security realm will prevent a server instance from starting up, blocking its boot process until the system password is reset or recovered. The problem is further complicated by the fact that WLS password management is promoted by the browser-based administration console, a valuable interface constrained by the limitation that the server must be up and running in order for it to be accessible. What options are available when the system password is lost but can't be reset from the console? The answer lies in a command-line feature hidden within the file realm itself.

Elements of the File Realm

Security realms play an important role within WLS 6.1; they define the set of users, passwords, groups, and access control lists used by a domain for its base security information. While various types of security realms exist, WLS by default uses the file realm unless configured otherwise.

For a given domain utilizing the file realm, two crucial files work together and constitute the underlying repository of the security realm. The first, a text file, named fileRealm.properties, persistently defines the actual set of users, hashed passwords, groups, and access control lists of the realm. The second, a binary file named SerializedSystemIni.dat, defines the seed input used by the file realm when a cleartext password is hashed in its SHA-based format. Both files exist within the root directory of the given domain (e.g., ./config/mydomain).

Since the SerializedSystemIni.dat file encapsulates a time-variant encrypted key created when a domain is first generated, no two SerializedSystemIni.dat files are created alike for a specific server installation. Therefore, when a cleartext password is hashed for a certain user and stored within the fileRealm.properties file, an explicit relationship is immediately established between the fileRealm.properties repository and its input SerializedSystemIni.dat file. The password stored in the repository and hashed with the SerializedSystemIni.dat input can be successfully compared only to its cleartext equivalent hashed with the same SerializedSystemIni.dat. This element is critical to the validation process of password comparison, where a cleartext password is compared to a stored password at the hash level, since it's impossible to directly reverse the hashed password back out to its original cleartext format.

An Invalid Password - More Than Just Forgetful
With WLS 6.1, the system password can be invalidated in a few ways. Aside from the obvious cases where the password is forgotten or misplaced (considered a lost password), the system password for the file realm of a given domain also becomes invalid when one of the following situations occurs:

  • Errant password modification: The fileRealm.properties file has been modified in such a way that an errant change is introduced to the hashed password value for the system user (intentional or otherwise).
  • Mismatched seed and repository: The SerializedSystemIni.dat file has been modified, replaced, or removed entirely, introducing a change that breaks the relationship between a given salt file and the set of passwords stored within a specific fileRealm.properties file.

An errant modification to a password is equivalent to password corruption. It usually results from one of two possibilities, differing only by the intent - an accidental change by an administrator who inadvertently modifies a password during an edit session, or a planned attack by a malicious user who intentionally corrupts a password. In both scenarios, the original hashed value of the password is changed at the character level so that a new hash is created, making it impossible to determine what the original cleartext of the new hash is and rendering the password invalid.

In the scenario involving a mismatched seed and repository, the relationship between a given salt file and the set of passwords stored within a specific fileRealm.properties file is broken as a result of the SerializedSystemIni.dat seed file having been modified, removed, or replaced entirely by another file (seed or otherwise). When the seed file becomes invalid with respect to its relationship in this manner, password comparison at the hash level will fail, since passwords stored within a given fileRealm.properties file and initially hashed with one salt are now compared to passwords hashed with another salt.

Available Recovery Options
When the system password is invalidated by one of the above possibilities, the recovery options are limited. If the password has been lost or corrupted, an administrator can restore the password from a backup copy of the fileRealm.properties file, if one exists. If a seed file is invalidated and its relationship to the set of passwords within a given repository has been broken, an administrator can restore the salt from a backup copy of the SerializedSystemIni.dat file - again, if the backup exists.

What can be done, however, when backup copies of the seed and repository files don't exist for the affected domain? You can copy the hashed passwords and appropriate files from the file realm of another domain (where the values of those passwords are known) when another domain exists, or create a new domain by installing a clean instance of WLS and copying from that. Of course, this has problems in its own right, as it depends upon the existence of another domain or the creation of a new domain through installation, and involves a fair amount of manual interaction. Besides, you might not want to risk compromising a production environment with passwords from another configuration.

When backup restoration isn't available and domain copies are not possible or desired, the best recovery option is to reset the password from the command line using the features of a hidden utility within the file realm itself. Mastering this utility will ensure proper recovery from disaster and enable a production environment to operate with little interruption. The rest of this article discusses in detail this utility - the FileRealm class.

The FileRealm Class
Every type of security realm within WLS 6.1 is ultimately managed and represented by a pure Java class. Such classes are either provided with the product out of the box or are externally developed to meet specific security needs. The default file realm is no exception, and is governed by the internal weblogic.security.acl.internal.FileRealm class that ships with WLS (packaged as part of the core weblogic.jar file). The primary responsibilities of this class are to manage the users, passwords, groups, and access control lists of the file-based security realm and provide authentication and authorization services for clients, using the underlying fileRealm.properties file as its persistent security repository and master record of data.

By all accounts, the FileRealm class should offer nothing more than a set of interface methods that other internal components of WLS can publicly use when authenticating and authorizing client access within the security realm - and it does. Interestingly enough, however, the FileRealm class also exposes a main method, allowing it to be invoked from the command line as a normal executable program. As a result, the internal class is surprisingly dual-purposed, serving primarily as a supporting library class to other elements within WLS, yet unexpectedly promoting itself as a command-line utility at the same time.

When executed from the command line, the FileRealm class provides a mechanism by which a cleartext password for a given user can be hashed with a specified input seed file, producing an encrypted equivalent output that can be stored within the fileRealm.properties repository and subsequently referenced by the file realm as the actual password for the given user. In this way, the class effectively gives administrators the ability to set passwords from the command line for any user within the file realm, free of the limitations imposed by the browser-based administration console and the password management system it offers. The potential value offered by this internal class is only fully realized in the most disastrous of situations, where the system password has been lost or invalidated and must be reset from the command line when the server consequently fails to start.

Resetting the System Password
Utilizing the command-line feature of the FileRealm class to set or change a password for a user within the file-based security realm is relatively straightforward. You must first identify the set of users that require password changes - the FileRealm class allows you to consolidate all password changes for multiple users at once, so executing the program in multiple iterations to handle a batch of users on an individual basis isn't required. However, in the example below we'll change the password for only one user (the system user).

When the set of users has been identified, a properties file must be created, and the users need to be defined within this file. Comments are allowed within the file (preceded by the # sign), and for each user within the set, a corresponding entry must be defined on its own new line within the file using the following format:

user.<username>=<cleartext_password>

An example follows:

# define the user system to have the
cleartext password weblogic
user.system=weblogic

This file provides the input list of users for the FileRealm class to process and associate each user with its original cleartext password. For this reason, the properties file can also be referred to as the input definition file. This file can be located anywhere in the local file system, and must end with the .src extension. In the example used below, we'll define the input definition file as user.properties.src.

When the input definition file has been created, the location of SerializedSystemIni.dat must be determined before the FileRealm class is executed from the command line. Recall that SerializedSystemIni.dat provides an input seed (or salt) to the hashing phase of the encryption process, and that a password hashed with a specific salt can be successfully compared only to another password hashed by the same exact salt. Therefore, the SerializedSystemIni.dat file, localized at the domain level and present within the config/ directory, has an explicit relationship to the passwords it hashes for the file realm of a given domain. Since WLS can't maintain a file realm in which the set of hashed passwords present have been seeded by different SerializedSystemIni.dat files, you should never mix passwords hashed by different salt files within the same file realm.

Depending upon the situation and the specifics of your problem, the FileRealm class can use an existing SerializedSystemIni.dat file for its input seed or create a new one if none exists. Referencing an existing SerializedSystemIni.dat is beneficial in the scenario in which the system password for a given domain has been lost but the domain-level salt file is still present and valid. This forces the FileRealm class to create a new password with the same seed used to hash all other passwords within the domain. Creating a new SerializedSystemIni.dat is useful when the salt file has been deleted or corrupted and the relationship to the passwords hashed by the original salt has been broken, making them invalid. Note that when a new SerializedSystemIni.dat file is created by the FileRealm class and used to hash new passwords for a domain, all other passwords within that same domain must be reset as well, since the seed for the hash has changed.

Executing the FileRealm class is simple and straightforward. Set your system classpath to include the weblogic.jar file from the lib directory of the WLS installation and invoke the utility from the command line as follows:

java weblogic.security.acl.internal.FileRealm \ <path_to_output_file> \ <path_to_salt_file>

Upon processing the input definition file, the FileRealm class generates its resultant set of users and their hashed passwords into the output file defined by the first parameter. This file is identical to the input definition file in every way except for the format of the passwords - the passwords in the output file are hashed, while those from the input are cleartext. Of course, this is the desired result. The path to the output file given by the first parameter should be identical to that of the input definition file, with the .src extension dropped for the output file. For example, if you had created the input definition file from above at /tmp/user.properties.src, you'd now need to define the output file at /tmp/user.properties. This accommodates the way the FileRealm class internally handles the location of the input definition file - it concludes its location by using the same path and name of the output file, yet it assumes the input file has the .src extension added to it.

The second parameter defines the location of the SerializedSystemIni.dat file. If the relative or absolute path specified references an existing salt file, then that seed is used as the input. If not, a new seed is created at the path specified with the given filename. Note that while it's possible to reference or create a new salt file with a name other than SerializedSystemIni.dat and outside of a given domain directory, WLS will recognize at runtime only seeds named SerializedSystemIni.dat and placed within their appropriate domain directories.

Upon execution, the FileRealm class will generate its hashed password outputs into the file defined by the first input parameter. When the command-line utility has completed, the invalid passwords found within the fileRealm.properties repository of the affected file realm should be overwritten with the new passwords stored within the output file. This last step should be done manually using your favorite text editor and a few select copy/paste operations. Once the changes have been made to fileRealm.properties and the file has been saved, your invalid passwords are valid once again, and must be verified. Start the server and provide the password recently set for the system user. If the password recovery was successful, the server will properly start and WLS will operate as expected. When you're finished, remove the input definition file and its output equivalent from the filesystem to avoid compromising your newly set passwords.

Conclusion
The browser-based administration console, an excellent utility constrained by the requirement of a running server, is the supported mechanism for password management within WLS 6.1. When the system password has been lost or invalidated and WLS won't start, however, the console becomes inaccessible and the only reliable option for resetting a file realm password is to do so from the command line. Utilizing the hidden features of the FileRealm class, you can manage passwords from the command line and recover from an otherwise disastrous situation. Additionally, while the hidden features of the FileRealm class discussed within this article relate to WLS 6.1, they are fully accessible and available under WLS 7.0, and should be used for the same purposes when running in the 6.1-compatible security mode.

More Stories By Steve Mueller

Steve Mueller is a principal
consultant for BEA Systems, where he specializes in the design,
development, and administration of enterprise systems running on WebLogic Server.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
Organizations do not need a Big Data strategy; they need a business strategy that incorporates Big Data. Most organizations lack a road map for using Big Data to optimize key business processes, deliver a differentiated customer experience, or uncover new business opportunities. They do not understand what’s possible with respect to integrating Big Data into the business model.
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, will discuss how they b...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
Digital transformation is changing the face of business. The IDC predicts that enterprises will commit to a massive new scale of digital transformation, to stake out leadership positions in the "digital transformation economy." Accordingly, attendees at the upcoming Cloud Expo | @ThingsExpo at the Santa Clara Convention Center in Santa Clara, CA, Oct 31-Nov 2, will find fresh new content in a new track called Enterprise Cloud & Digital Transformation.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, will discuss how given the magnitude of today's applicati...