Out of Memory during Passivation of View Object with LOV

Have you ever faced an Out Of Memory Error (OOME) with your ADF application? You probably had so please go ahead and read the rest of this post...

Sometimes it seems so very easy to create problems, but it is so damn hard to find them out and fix them. In this post we will start from the easy part: how to create a problem. The main ingredients of the recipe that may lead to some serious problems in production environments are summarized here:

• An old code base that was migrated from earlier versions of JDeveloper (e.g. 11.1.1.4)
• A View Object with an Attribute that has a List of Values (LOV) defined
• The View Object has a View Accessor that is used in the LOV
• The List Data Source is a View Object based on SQL Query. The View Object is configured to Passivate State Including All Transient Attributes

View Object - Bad Tuning Example

In addition to the above we should also consider the fact that the LOV was build based on a table with thousands of rows in the production environment. No provision was made in order to filter data and limit the results the LOV could fetch from the database.

If you forget for the moment that all the above configuration smells some kind of trouble and should have been avoided at the beginning, think that was just something in production without any tool or procedure (code review, testing etc) ever noticing it.

The result of the above was either very long (a day or so...) running requests and sometimes an OOME error. The worst part was that this problem happened only twice... A "stuck thread" and an OOME incident and no more... No one could re-produce the problem. That was the worst part since another fix included some modifications through ADF customization that really buried the problem for the specific installation. This was proved just by accident...

In the rest of the post I will try to explain the ADF behavior with the above configuration. In another post I will show how to take advantage of any trace of the problem you may have in your hands like a JFR.

In the following figure you can have a look of the UI that I have created in order to demonstrate the behavior described above.

In order to reproduce the problem I have created two View Objects: EmployeeVO and DepartmentsVO. Based on the DepartmentsVO I have created a ViewAccessor and an LOV for the DepartmentId attribute.

For both the ViewAccessor and the LOV I have left the default values that JDeveloper generates, but for the LOV I have chosen the "Input Text with List of Values".

Now if you choose to disable Application Module (AM) pooling and run the application you would get a nice AM activation passivation cycle. So far so good, the application is running fast and each time we press one of the form's buttons we are happy with the result, but shall we?...

Let's dig a little more and study what's going on behind the scenes. In order to get directly to the point of this post, let start from checking what is going on with the activation/passivation cycle, since we have deliberately disabled AM pooling. (By the way this is also recommended by Oracle while developing). In my case I have increased the logging level for the oracle.jbo.server.Serializer to FINEST. At the log file the Serializer has produced the following XML during the passivation of the AM.

There are two ViewObjects that get passivated: EmployeeVO1 and _LOCAL_VIEW_USAGE_com_soapplied_blog_ap_model_EmployeesVO_DepartmentsVA. For the first things look normal, but for the second things are not so pretty... Just image what may happen if the VO was not querying the Departments table, but something quite bigger!

So in a case where the query behind the Departments VO could fetch thousands of records from the DB, this would lead to an enormous XML document that the Serializer has to build and persist to the DB. This is a very CPU and memory hungry procedure that would certainly cause trouble to a production environment. To add to this very bad configuration we have left the fetch size of the VOs to 1. This means that all records of the Departments VO will be fetched one by one from the DB. This is also a very slow procedure with thousands of round-trips to the DB.

References:
https://blogs.oracle.com/ADFProgrammers/entry/restrict_the_number_of_items

http://adfpractice-fedor.blogspot.gr/2013/07/passivation-and-activation-of-view.html