September 15, 2009, 12:12 pm
Your application just slipped into unconsciousness, and now any request you make just hangs. To make matters worse, your logs don’t show anything useful. So how can you check your application’s vital signs when it drops on you like this?
All your production applications should be under critical care with monitoring systems that continually look at key statistics and life signs. Thus, if an application becomes unresponsive, you just need to glance at the monitor on your office wall to see the statistics for your environment.
But what if somehow you find yourself in a position where you have a critical problem with an application but don’t have any monitoring infrastructure? How can you quickly check the following vital signs?
- Pulse and respiration (data source usage, how many threads are used by HTTP, and AJP connector)
- ID information (check server information like JVM, OS, and JBoss versions)
- Signs of shock (high CPU usage, which threads took the most time on the server, and what threads are currently doing)
Before You Start
Ideally, you should never be in this position with production systems. There should always be some type of monitoring infrastructure in place to help you respond to these types of problems. Only development—and sometimes functional testing QA environments—should be routinely setup without monitoring in place. Monitoring is an essential part of production environments as well as very useful feature to have in load testing environments.
Having said that, if you’re experiencing application problems and don’t have monitoring in place, you’d undoubtedly prefer solutions to lectures about best practices that haven’t been followed. So, let’s cut to the chase and show one possible solution.
Meat & Potatoes
In most situations you should have a monitoring system setup to look at all the information covered below. This article is written for the rare situation in which you do not having a monitoring system.
If your JBoss installation is mostly unaltered from its defaults, it is likely you still have the JMX Console, Tomcat status and Admin Console enabled, and you can use them to get answers to key questions that will help you check your application’s vital signs. If you have them enabled, good for you. But if you leave them on in a production system, don’t forget to at least secure them—the mechanisms presented here should be at least secured if not disabled in production systems.
Note that there are many reasons why an application might hang, and also that there are many other things you can check (e.g. EJB pools) that are not covered here. This article is not intended to be a detailed course in JBoss and application brain surgery, but rather just a brief paramedic’s manual covering the most common signs and causes of unresponsive behavior. In other words, this article is intended to help you determine what’s wrong, not how to cure it – that might be a topic for subsequent articles.
Where Are Consoles Located?
To access the JMX Console go to http://localhost:8080/jmx-console, where localhost is your server’s name. Tomcat status is available at http://localhost:8080/status?full=true. And (if you’re using a JBoss version that includes it) the admin console is at http://localhost:8080/admin-console.
Note that there are differences between JBoss 4 and JBoss 5.1. We will first talk about methods that work in both JBoss 4 and JBoss 5, and then present a somewhat easier method that works only in JBoss 5.1.
Checking DataSource Usage
Investigating data source usage is useful when you’re unsure if you have enough data sources available for application. If you ran out of the available data source connections, your system is unresponsive because your queries are stuck.
To find statistics about data source usage, you need first to find the name of your data source. Let’s suppose that it’s named MyDataSource.
Go to the JMX console, and find the section called jboss.jca. You should see MBean there with the object name:
name=MyDataSource,service=ManagedConnectionPool
Note: There are multiple similarly named entries in the JMX console, so make sure that you use right one in the right section for the right data source!
Click on that link, and the following arguments are especially useful:
- MaxConnectionsInUseCount – displays the maximum number of connections that have ever been used since the server started or the data source was deployed.
- InUseConnectionCount – displays the number of connections currently used.
- MaxSize – displays the maximum number of connections available in the pool.
Another interesting operation here is listStatistics(). The example below shows the listing for one data source:
Checking Connector Thread Usage
Checking connector thread usage is useful when you want to find out how many of the available request threads are used. If all connector threads are blocked, your incoming requests are waiting on the current requests to proceed before their processing can be started.
To find statistics about connector usage, the best way is to look at http://localhost:8080/status?full=true. This will give you the following statistics:
- Max threads – displays the max number of threads the connector can spawn.
- Current thread busy – indicates how many threads are currently used by application users.
There are many other useful things available here, including a list of all currently-deployed applications and per-servlet statistics.
Be sure that you are looking at the statistics for the right connector for your setup – AJP connector if you are using AJP (e.g. setup with Apache HTTPD, mod_jk, and JBoss) or HTTP connector if your HTTP requests go directly to JBoss. The default configuration has AJP connector listening on port 8009 and HTTP connector on port 8080.
The picture below shows statistics for the HTTP connector on a recently-started server with only one user:
Obtaining Server Information
Server information is useful when your questions are:
- How do I get the thread dump?
- What’s the cumulative time spent in my threads?
- Which JVM, OS, and JBoss versions are used?
- How was the server configured? Which configuration (e.g., all, default, or minimal) am I working with?
This line of questioning is helpful to collect basic version information for the software you are using, as well as data about what is your server is currently doing.
You probably know the versions and configuration if you were the person who installed the system, and this information is always available in the JBoss boot.log file. You can also double-check this information with only browser access by finding the following three entries in the jboss.system section of the JMX Console.
These three beans provide answers to all of the questions listed above:
- Server – provides information about the version of JBoss server in use.
- ServerConfig – provides information about the server configuration in use.
- ServerInfo – provides information about the JVM and OS version you are using to run the server. Additionally, the listThreadCpuUtilization() method that you can call from this screen provides you with information about the CPU usage of your threads, and listThreadDump() allows you to execute a thread dump.
Is CPU Usage Going Crazy?
If your CPU usage is stuck at 100% all the time, it might be an indication of multiple problems such as pooling when you shouldn’t, memory leaks, and CPU-intensive code.
A thread dump can help you detect pooling or infinite loop types of problems. Do multiple thread dumps and compare the results to determine if you’re still stuck in the same place on some threads. If these threads have high CPU usage as well, they’re doubly suspicious. Both of these can be checked in the SystemInfo bean, as discussed in the previous section.
Memory leaks are slightly more complex. For additional details on fixing memory leaks please see the tutorial, How to Fix Memory Leaks in Java.
What About JBoss 5?
All of the techniques covered so far apply to both JBoss 4 and JBoss 5, but the JMX Console is somewhat intimidating and may provide the impression of a “big, difficult to navigate pile of MBeans.” JBoss 5.1 comes to the rescue here with an admin console that delivers a more user-friendly way of obtaining important information.
You can access the JBoss 5.1 admin console at http://localhost:8080/admin-console. All of the previously-mentioned information is readily available within the admin console. Look at the tree on the left side.
At the root of the tree you can get information about the OS version.
To get information about the JBoss version and configuration, click on the server in the left side tree view (e.g., JBoss AS 5, which is the default entry in the tree).
Information about the data source is located under JBossAS Servers/YourServer/Resources/Datasources in the tree on the left side. Find your data source, and then look at both the Summary and Metrics panels on the right side. The example above shows the DefaultDS data source, but your data source is probably different. Be sure to look at statistics for your own data source—not, for example, of DefaultDS!
To find connector statistics, look in the tree on the left side under JBossAS Servers/YourServer/Resources/JBoss Web/Connectors.
Can I Get This Information From A Command Line?
If you prefer to work from a command line, you can get the information covered above by using the twiddle.sh command line utility located in the $JBOSS_HOME/bin. The names of the MBeans that you should use in twiddle.sh are the same as the names of the MBeans you are using in JMX Console.
If Something is Maxed, I Just Increase It, Right?
Sometimes, but not always. A lot depends on the situation. What to do in each situation involves some degree of “it depends” and may be the subject of future articles.
Finishing Up
What if you don’t have monitoring installed, can’t get monitoring information from any application, and your consoles and twiddle.sh don’t work? Or if all of the vital signs covered above look normal but you are still stuck with a comatose application? Well, in this case there’s very little that JBoss paramedic techniques can do to help, and you might need to perform a little exploratory surgery to get more information.
You have couple of options here:
- You might try to dig out the information yourself (as a hint, you can change log levels in JBoss for a lot of things without necessarily needing to restart).
- You might reproduce the problem with more monitoring in place, and then try to dig out additional information.
- You might look at additional places that are outside of the scope of this article (e.g., EJB pools, JMS behavior, etc.).
- If all else fails, most JBoss consultants who are worth their salt should be able to do all of the above for you and have at least few more tricks that apply to your specific situation and are not mentioned in article.
References / For More Info
The JBoss Group: “JBoss 4.0 – The Official Guide”, Sams Publishing, April 30, 2005, ISBN 978-0672326486
Javid Jamae, Peter Johnson: “JBoss in Action: Configuring the JBoss Application Server”, Manning Publications, January 28, 2009, ISBN 978-1933988023
http://www.jboss.org/community/wiki/SecureTheJmxConsole
http://www.jboss.org/community/wiki/AccessControlForJMXConsole
About the Author
Veljko Krunic
Not only did Veljko emigrate to the US from Serbia, his idea of fun involves some fairly extreme environments: he’s happiest gliding about at high altitude (through the clouds, we mean, in a teeny tiny lightweight glider plane, while the rest of us clutch at our parachutes and gape through the window of the tow-plane) or flippering through the ocean depths dependent on canned air (scuba). We know he’s intrepid. So, that’s two things, ’cause we also know he’s smart. He’s working away at a Ph.D in CS from the University of Colorado at Boulder, where he’s already gained his BS and MS in the same field. Veljko can be reached at veljko.krunic at openlogic dot com.
