In the real world, the migration between two OBIEE environments is really take time .The last time we did this for a customer migrating from 10g to 11g, we had to spend roughly 2 weeks just running through the dashboards clicking on all objects to do the 1st pass of the upgrade. When a customer asked us to migrate from 11g to 12c we were bracing ourselves for a similar ordeal. But the folks at Oracle have really done a great job this time with the Baseline Validation Tool (BVT). It allowed us to very quickly do a 1st pass through all the catalog objects and do a quick sanity check of the catalog and RPD after the upgrade. 

The challenge is you need to make sure the content (analysis, catalog object, charts…) in two environments is same after the migration. BVT is FREE tool that enables the creation of a baseline set of data that can be compared with data from another system. While we used it for the upgrade it can be used in any situation where we need to run through 2 environments 

• May be the result of an upgrade, patch or migration, or any change
• Automated regression testing on any two compatible Oracle BIEE environments

BVT as a tool is quit well thought thru and allows for several checks to be done in an automated manner. 

Data Validation: Allows to check the numbers in the reports between the baseline and new environment. 

1. Check on actual data values exported from OBI
2. Exported to CSV format
3. Comparison report generated

BVT even downloads the data from the report as excel or CSV and it can be used to analyze the differences later. 

Visual Verification:  The purpose of this is to make sure that visually the 2 environments look the same. During the launch they spoke about using some sophisticated analytics to do the comparison, but whatever they have done it does pretty good job to compare the outputs. BVT generates reports for both analyses and dashboards

• Comparison report allows quick scan of (many) results in addition to a threshold test
• Use in conjunction with data validation
• Can be used to check various prompt combinations. 

The tool checks all the reports and dashboards in the catalog and produces a report which can even be shared with the customer. 

The output is exported and can be manually checked later of passed to the customer as evidence. 

BI Catalog Verification: Compares the metadata of catalog objects (rather than objects themselves)

Logical Query validation: Used to validate the logical query between the 2 environments. Used in conjuction with the data and visual validation gives a complete picture of the 2 environments. 

The logical query query is captured as evidence and for the failed ones can be used to determine the reason for the failure. Using the logical query in conjuction with the data and visual verification gives us complete confidence on the outcome of the testing. 

Requirements

• Java 1.6 or higher
• For Windows: Windows 7 or higher, or Windows Server 2012 or higher
• For Linux: Oracle Linux 6 U5 or Oracle Linux 7
• Mozilla Firefox or Internet Explorer (need to enable javascript)

Configuration of BVT 

The following are the steps we followed to get it all working in our environment 

1. Download & install DVT

First, you have to download OBI Developer Client Tool (12.2.1.0.0) if you don’t have one. The tool zip file is located in [Installation Folder]\bi\components\oracle.bi.bvt

2. Configuration

After unzip the tool, you need to fill in 2 OBIEE instance detail in testconfig.xml file such as

• username, password (you can leave it blank & be prompted later when you run the tool via command line)
• AnalyticsURL
• Name of instance (used for running the tool)

The testconfig.xml controls the behaviour of the BVT and is present in the root folder of the tool. 

You can tweak the various tests to be done by tweaking the settings in the config file. 

Catalog plugin config: Use this to setup the folders to be used for comparison

UI plugin config: Use this to tweak the algorithm for the visual verification algorithm. The  main setting to be done on this is the Threshold score with a value between 0 and 1, where a value of '1' indicates a 100% pixel-by-pixel match, and a value of '0' indicates a complete mismatch. 

Repots plugin config: Set a limit on the maximum tolerance for differences in floating point numbers in report data. Ex:"0" for no tolerance. "0.1, "0.001". It’s very useful for percentage measures. 

Logical query plugin config: Set the folder to be used for the comparison 

Dashboard plugin config: You can set the prompt combinations to be used for the BVT testing. 

3. Running the tool 

You need to run DVT three times. First & second for 2 OBIEE instances. The third for compare them & get the comparison result.

Examples of running :

bin\obibvt.bat -config testconfig.xml -deployment BI11g
bin\obibvt.bat -config testconfig.xml -deployment BI12c
bin\obibvt.bat -compareresults Results\PrePatch Results\PostPatch -config testconfig.xml

4. Getting the comparison results 

After running 3 test, you will get the Results folder, which contain information of each OBIEE instance and Comparisons folder, which contains comparisons results along with all the detailed evidence.

5. Things to be aware of 

• BI-Publisher reports are not included in the comparison
• Oracle Baseline Validation not working well with recent firefox release when running UI Plug-in test, recommended one is firefox version 34
• Run the script from the parent folder of the .bin folder, and not from the .bin folder itself.
• Set the timestampFolders = true to prevent overwritten on subsequent runs of the tool

The goal of this post was to document our experience with the new regression testing tool shipping with OBIEE 12c; Setting up the configuration and the running comparisons is fairly straightforward; the results are detailed and handy. There is no doubt that this tool will be useful for OBIEE regression testing. In conclusion, the Oracle BI development team is making big strides in addressing pain points that have made working with OBIEE and migrating and maintaining different Oracle BI environments a challenge.  

Click below to engage us learn more about 12c upgrade and its benefits.