Welcome!

Weblogic Authors: Pat Romanski, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

'HTTP Session Replication Failure' Issues

Moving away from the primary thought

Sometimes, HTTP session states are not replicated from the primary server to the secondary server.

Symptoms

  1. The application using HTTP session does not function as designed and you see a loss of session data.
  2. You might be asked to re-log into the application even when the session has still not timed out.
  3. You see errors and warnings related to HTTP session failures in the server's log file.
  4. The request is not failed over to another server properly.

There are several reasons as to why session replication fails. We will look at ways to diagnose the issue, possible reasons for it, and the ways to address them.

Types of HTTP Session Replication
There are five different implementations of session persistence:

  • Memory (single-server, non-replicated): When you use memory-based storage, all session information is stored in memory and is lost when you stop and restart WebLogic Server.
  • File system persistence: Session information is stored in a file under the PersistentStoreDir specified.
  • JDBC persistence: Session information is stored in a database table.
  • Cookie-based session persistence: Session information is stored in a cookie.
  • In-memory replication (across a cluster): Session data is copied from one server instance to another into memory.

Diagnosing the Issue
Consider a scenario with two servers (MyServer-1, MyServer-2) in a cluster. When you enable the debug flags (see the "Enable the Debug Flags to Track Session Replication Failures"section), you will see the messages below when a request from a client is sent the very first time.

On MyServer-1:

<Oct 9, 2003 12:38:21 PM PDT> <Debug> <Cluster> <000000>
<Creating primary 5165892837402719733>
<Oct 9, 2003 12:38:21 PM PDT> <Debug> <Cluster> <000000>
<Created secondary for 5165892837402719733 on -7957889153726652135S:
192.168.11.112: [9001,9001, -1, -1,9001, -1, -1]: mydomain: MyServer-2>

This logging message means that the primary server is MyServer-1 and a secondary has been created on MyServer-2. A message like the one shown below will be logged on MyServer-2 to confirm that.

ExecuteThread: '1' for queue: 'Replication'> <kernel identity> <>
<000000> <Creating secondary 5165892837402719733>
####<Oct 9, 2003 12:38:21 PM PDT> <Debug> <Cluster>
<rvimala-c840> <MyServer-2> <ExecuteThread:
'1' for queue: 'Replication'>
<kernel identity> <> <000000>
<Updated local secondary of 5165892837402719733>

If you check the JSESSIONID it looks like:

JSESSIONID=1E9Xwn7nLYfOsc1obgRZIwW5s72an7HPPvSD7iaWHMXzpHga5cQj
!-1587343083!-1587348922

JSESSIONID is the default name of the cookie, which could be changed to anything in weblogic.xml. The format of JSESSIONID is

SessionId!PrimaryServer JVM Hash!SecondaryServer JVMHash

Every time data is changed (either set/get or removed) in the session you'll see the logging message.

On MyServer-1:

<Oct 9, 2003 12:38:21 PM PDT> <Debug> <Cluster> <000000>
<Updated remote secondary for 5165892837402719733>

On MyServer-2:

####<Oct 9, 2003 12:38:21 PM PDT> <Debug> <Cluster>
<rvimala-c840> <MyServer-2> <ExecuteThread:
'1' for queue: 'Replication'> <kernel identity> <> <000000>
<Updated local secondary of 5165892837402719733>

If for any reason session replication fails, you will see the message below in the MyServer-1 log:

<Nov 6, 2003 12:59:12 PM EST> <Debug> <Cluster> <000000>
<Unable to create secondary for -5165892837402719733>
<Nov 6, 2003 12:59:12 PM EST> <Debug> <Cluster> <000000>
<Error creating secondary 5165892837402719733 on -7957889153726652135S:
192.168.11.112:[9001,9001,-1,-1,9001,-1,-1]:mydomain:MyServer-2>

And, the JSESSIONID would appear as

JSESSIONID=1E9Xwn7nLYfOsc1obgRZIwW5s72an7HPPvSD7iaWHMXzpHga5cQj!-1587343083!NONE

The secondary server hash would become NONE

Enable the Debug Flags to Track Session Replication Failures
You can enable the flags DebugCluster, DebugClusterAnnouncements, DebugFailOver, DebugReplication, and DebugReplicationDetails.

To Enable
Use the weblogic.Admin command-line utility to dynamically turn the debug options on and off. For example, to turn on DebugCluster on all administration instances of ServerDebug MBean (i.e., Admin Server or al Managed Server):

java weblogic.Admin -url t3://localhost:7001 -username system -password weblogic
SET -type ServerDebug -property DebugCluster true

Or, edit the config.xml and the MBean element in the <ServerDebug/> stanza for each server that you want to debug and set it to a value of "true" to enable or "false" to disable. Then you must restart the Admin Server. Managed Servers will reconnect to the Admin Server and the debug flags will then dynamically take effect. For example:

<ServerDebug DebugCluster="true" Name="myserver"/>

At the end, with all the flags set, in your config.xml the ServerDebug tag would look like:

<ServerDebug ClassFinder="true" DebugCluster="true"
DebugClusterAnnouncements="true" DebugFailOver="true"
DebugReplication="true" DebugReplicationDetails="true" Name="MyServer1"/>

Make sure the stdOutSeverity level of the server is INFO and StdoutDebugEnabled is set to "true". The debug information will be logged into the server log as well as to the standard out.

Checklist for Each Session Persistence Type
Memory (Single Server, Nonreplicated)

  1. When you use memory-based storage, all session information is stored in memory and is lost when you stop and restart BEA WebLogic Server.
  2. Make sure you have allocated sufficient heap size when running WebLogic Server; otherwise, your server may run out of memory under heavy load.
  3. Not a recommended type for Cluster configuration (because the data is kept in heap and is not available to any other server).

File System Persistence

  1. Verify that the directory where BEA WebLogic Server stores the sessions is correctly specified in weblogic.xml. You must also create this directory yourself and make sure appropriate access privileges have been assigned to the directory.
  2. Make sure you have enough disk space.

JDBC Persistence
Make sure the connection pool that connects to the database has read/write access for the database table used.

Cookie-Based Persistence

  1. Make sure you have not stored anything other than java.lang.String in the HTTP session.
  2. Do not flush the HTTP response object in your application code.
  3. Make sure that the content length of the response exceeds the buffer size set (default is 8192 bytes).
  4. Make sure that cookies are enabled in the browser.
  5. Make sure that you do not use commas (,) in a string when using cookie-based session persistence.

In-Memory Replication

  1. Make sure that the BEA WebLogic Server is accessed only via a proxy server or a hardware load balancer
  2. The hardware load balancer should support a compatible passive or active cookie persistence mechanism, and SSL persistence.
  3. Recommended type in a cluster.

Validate the weblogic.xml Entries
Make sure weblogic.xml has all the parameters that need to be set for each Session Replication type. For example, when using in-memory replication the sample weblogic.xml would look like:

<session-descriptor> <session-param> <param-name> PersistentStoreType </param-name> <param-value> replicated </param-value> </session-param> </session-descriptor>

Note: The debug files are subjected to change with each version of BEA WebLogic Server.

Session Data Must Be Serializable
To support in-memory replication of HTTP session states, all servlet and JSP session data must be serializable or else session replication would fail.

When debugging is enabled, BEA WebLogic Server would output the warning messages below by indicating that the session is not replicated. After this, session replication would stop.

Debug Message:

<Oct 8, 2003 2:10:45 PM PDT> <Error> <Cluster> <000126>
<All session objects should be serializable to replicate.
Please check the objects in your session.
Failed to replicate non-serializable object>

On the subsequent request the JSESSIONID will be missing the secondary server and will be marked NONE, e.g.:

JSESSIONID=1E9Xwn7nLYfOsc1obgRZIwW5s72an7HPPvSD7iaWHMXzpHga5cQj!-1587343083!NONE

Solution: Find out the page the error is thrown from and make sure that all the data put into the session is serializable.

Check for Network/Multicast Issues
Make sure that the network is fine and there are no multicast issues. Do the multicast test to make sure that the multicast IP is working fine.

Syntax

java utils.MulticastTest -n name -a address [-p portnumber] [-t timeout] [-s send]

(http://edocs.bea.com/wls/docs81/adminref/utils.html#1199798)

Validate Cluster Configuration
The primary and secondary servers are selected from the cluster list. In a cluster of two servers, if the cluster doesn't have all the servers, then a secondary might not be chosen, resulting in the session data not being replicated.

To verify, execute the following commands:

  1. Make sure weblogic.jar is in the classpath.
  2. To get all the servers in a cluster:

java weblogic.Admin -username weblogic -password weblogic -url
http://oneofthemanagedserverurlinthecluster:7001/ GET -type ClusterRuntime -pretty

This will list all the servers in a cluster. The URL could be changed to every server in the cluster to make sure they all have the same entries.

Application Code Diagnostics
Make sure you use only setAttribute/removeAttribute methods of the HTTP session in your application code to update the HTTP session. If you use other set methods to change objects within a session, BEA WebLogic Server does not replicate those changes.

Please do not use the methods putValue and removeValue of the HTTP session as they are deprecated and there could be issues with session data replication when using such methods in your application. Instead, use only the setAttribute/removeAttribute methods of the Http session.

Cookies vs URL Rewriting
In some situations, a browser or wireless device may not accept cookies, which makes session tracking with cookies impossible. URL rewriting is a solution to this situation that can be substituted automatically when WebLogic Server detects that the browser does not accept cookies. Enable URL rewriting in BEA WebLogic Server by setting the URLRewritingEnabled attribute in the WebLogic-specific deployment descriptor, weblogic.xml, under the <session-param> element. The default value for this attribute is true.

Performance Issues

  • Consider serialization overhead: Serializing session data introduces some overhead for replicating the session state. The overhead increases as the size of the serialized objects grows. If you plan to create very large objects in the session, test the performance of your servlet to ensure that performance is acceptable.
  • Control frame access to session data: If you are designing a Web application that utilizes multiple frames, keep in mind that there is no synchronization of requests made by frames in a given frameset. For example, it is possible for multiple frames in a frameset to create multiple sessions on behalf of the client application, even though the client should logically create only a single session.

    To avoid unexpected application behavior, carefully plan how you access session data with frames. You can apply one of the following general rules to avoid common problems:
    - In a given frameset, ensure that only one frame creates and modifies session data.
    - Always create the session in a frame of the first frameset your application uses (for example, create the session in the first HTML page that is visited). After the session has been created, access the session data only in framesets other than the first frameset.

  • Storing larger amounts of data in the session: JDBC persistence and File persistence won't be faster as the session data has to be stored and retrieved from an external resource. There is also a performance overhead because of JDBC access for each session update. If you want to store large objects in the session, then JDBC or File persistence should be considered.
  • Storing small amount of data in the session: Cookie-based session persistence is most useful when you do not need to store large amounts of data in the session. Cookie-based session persistence can make managing your BEA WebLogic Server installation easier because clustering failover logic is not required.

    Further Information
    For additional information go to http://support.bea.com for some published solutions on session replication failures. You can also query ASK BEA at http://websupport.beasys.com/index.jsp.

    If none of these help you towards a solution or an identifier in your application, then contact BEA Customer Support for further diagnosis. You can open a case with a valid support contract by logging in at http://support.bea.com/login.jsp

  • More Stories By Vimala Ranganthan

    Vimala Ranganathan is a backline developer relations engineer with BEA Systems. She specializes in troubleshooting and solving complex customer issues with their mission-critical applications on BEA products. Vimala holds a bachelor's degree in computer science.

    Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    @ThingsExpo Stories
    The Transparent Cloud-computing Consortium (abbreviation: T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data processing High speed and high quality networks, and dramatic improvements in computer processing capabilities, have greatly changed the nature of applications and made the storing and processing of data on the network commonplace.
    TechCrunch reported that "Berlin-based relayr, maker of the WunderBar, an Internet of Things (IoT) hardware dev kit which resembles a chunky chocolate bar, has closed a $2.3 million seed round, from unnamed U.S. and Switzerland-based investors. The startup had previously raised a €250,000 friend and family round, and had been on track to close a €500,000 seed earlier this year — but received a higher funding offer from a different set of investors, which is the $2.3M round it’s reporting."
    Predicted by Gartner to add $1.9 trillion to the global economy by 2020, the Internet of Everything (IoE) is based on the idea that devices, systems and services will connect in simple, transparent ways, enabling seamless interactions among devices across brands and sectors. As this vision unfolds, it is clear that no single company can accomplish the level of interoperability required to support the horizontal aspects of the IoE. The AllSeen Alliance, announced in December 2013, was formed with the goal to advance IoE adoption and innovation in the connected home, healthcare, education, aut...
    The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, data security and privacy.
    All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. Over the summer Gartner released its much anticipated annual Hype Cycle report and the big news is that Internet of Things has now replaced Big Data as the most hyped technology. Indeed, we're hearing more and more about this fascinating new technological paradigm. Every other IT news item seems to be about IoT and its implications on the future of digital busines...
    Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water, are pursuing SmartGrid initiatives that represent one of the more mature examples of SAE. We have s...
    Software AG helps organizations transform into Digital Enterprises, so they can differentiate from competitors and better engage customers, partners and employees. Using the Software AG Suite, companies can close the gap between business and IT to create digital systems of differentiation that drive front-line agility. We offer four on-ramps to the Digital Enterprise: alignment through collaborative process analysis; transformation through portfolio management; agility through process automation and integration; and visibility through intelligent business operations and big data.
    The Internet of Things needs an entirely new security model, or does it? Can we save some old and tested controls for the latest emerging and different technology environments? In his session at Internet of @ThingsExpo, Davi Ottenheimer, EMC Senior Director of Trust, will review hands-on lessons with IoT devices and reveal privacy options and a new risk balance you might not expect.
    IoT is still a vague buzzword for many people. In his session at Internet of @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, will discuss the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. The presentation will also discuss how IoT is perceived by investors and how venture capitalist access this space. Other topics to discuss are barriers to success, what is new, what is old, and what the future may hold.
    Swiss innovators dizmo Inc. launches its ground-breaking software, which turns any digital surface into an immersive platform. The dizmo platform seamlessly connects digital and physical objects in the home and at the workplace. Dizmo breaks down traditional boundaries between device, operating systems, apps and software, transforming the way users work, play and live. It supports orchestration and collaboration in an unparalleled way enabling any data to instantaneously be accessed on any surface, anywhere and made interactive. Dizmo brings fantasies as seen in Sci-fi movies such as Iro...
    There’s Big Data, then there’s really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at 6th Big Data Expo®, Hannah Smalltree, Director at Treasure Data, to discuss how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other mach...
    This Internet of Nouns trend is still in the early stages and many of our already connected gadgets do provide human benefits over the typical infotainment. Internet of Things or IoT. You know, where everyday objects have software, chips, and sensors to capture data and report back. Household items like refrigerators, toilets and thermostats along with clothing, cars and soon, the entire home will be connected. Many of these devices provide actionable data - or just fun entertainment - so people can make decisions about whatever is being monitored. It can also help save lives.
    All major researchers estimate there will be tens of billions devices – computers, smartphones, tablets, and sensors – connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be!
    Whether you're a startup or a 100 year old enterprise, the Internet of Things offers a variety of new capabilities for your business. IoT style solutions can help you get closer your customers, launch new product lines and take over an industry. Some companies are dipping their toes in, but many have already taken the plunge, all while dramatic new capabilities continue to emerge. In his session at Internet of @ThingsExpo, Reid Carlberg, Senior Director, Developer Evangelism at salesforce.com, to discuss real-world use cases, patterns and opportunities you can harness today.
    Arrow Electronics Inc. announced its Internet of Things Immersions Roadshow that will showcase how “Interconnected Intelligence” is changing the way the world interacts and solves problems with technology. The Immersions tour will engage the world’s top technology leaders to discuss comprehensive Internet of Things (IoT) building blocks and how businesses can leverage Interconnected Intelligence to improve lives throughout the world. With forums in four key U.S. markets, Arrow connects technology developers with leading-edge suppliers to provide insights about IoT technologies and services,...
    The Internet of Things is not new. Historically, smart businesses have used its basic concept of leveraging data to drive better decision making and have capitalized on those insights to realize additional revenue opportunities. So, what has changed to make the Internet of Things one of the hottest topics in tech? In his session at Internet of @ThingsExpo, Chris Gray, Director, Embedded and Internet of Things, will discuss the underlying factors that are driving the economics of intelligent systems. Discover how hardware commoditization, the ubiquitous nature of connectivity, and the emergen...
    Trick question: What did Thomas Edison do for the lightbulb? He didn’t invent it – working light bulbs existed in laboratories for 80 years before Edison arrived. Instead, Edison's team made electric light scalable. They turned a theoretical possibility into a daily, lived reality – for billions of people. Today, we are at the same point for behavior change. Psychologists, economists, and other behavioral researchers have shown that it’s possible to nudge habits, but almost no one has been able to deliver sustainable, reliable behavior change at scale.
    Chris Matthieu is Co-Founder & CTO at Octoblu, Inc. He has two decades of telecom and web experience. He launched his Teleku cloud communications-as-a-service platform at eComm in 2010 which was acquired by Voxeo. Next he built an opensource Node.JS PaaS called Nodester which was acquired by AppFog. His new startup is Twelephone (http://twelephone.com). Leveraging HTML5 and WebRTC, Twelephone's BHAG (Big Hairy Audacious Goal) is to become the next generation telecom company running in the Web browser. In 9 short months, Twelephone has nearly achieved feature parity with Skype.
    SYS-CON Events announces a new pavilion on the Cloud Expo floor where WebRTC converges with the Internet of Things. Pavilion will showcase WebRTC and the Internet of Things. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices--computers, smartphones, tablets, and sensors – connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.
    Things are being built upon cloud foundations to transform organizations. This CEO Power Panel at 15th Cloud Expo, moderated by Roger Strukhoff, Cloud Expo and @ThingsExpo conference chair, will address the big issues involving these technologies and, more important, the results they will achieve. How important are public, private, and hybrid cloud to the enterprise? How does one define Big Data? And how is the IoT tying all this together?