For example, someone asked a question along the lines of the following:

I have a document with three data providers but there are four reports. How can I know which data provider goes with which report, and why are there more reports than data providers?

That wasn’t the exact question, but that was essentially what they were asking. For someone that is new to BusinessObjects it’s a reasonable question to ask. Especially if they come from a spreadsheet background where everything is right up in front. I thought it might be interesting to set up an Entity – Relationship Diagram for the various document components to help clarify how things work together.

Entities In A Web Intelligence Document

First I want to identify the various entities (components) of a document. A brief note before I start: the concepts in this post apply equally to Desktop Intelligence and Web Intelligence, but not to Crystal.

• Document
  This is the container object. Everything starts here. The parallel for Excel would be the workbook.
• Report
  This is the page or “tab” in a document. The parallel for Excel would be a worksheet. In Web Intelligence a report tab starts out as a blank canvas. I can include various other report structure items as required.
• Block
  Here is where the spreadsheet comparison starts to fall apart a bit. A report is a canvas that starts out empty, whereas a spreadsheet starts out with rows and columns. For Web Intelligence the canvas is completely blank and the structure only comes into existence when I create a block. More on that below, but for now let me close by saying there are different types of blocks such as tables, crosstabs, and charts. Is there a parallel with Excel? Sort of. Data in a sheet goes into the existing row / column structure. But it is possible to create a chart that “floats” above the worksheet. BusinessObjects uses this floating concept for every block on a report, not just charts.
• Cell
  I use this term to refer to two things: it can either be a stand-alone cell on a report, or it can be a cell that participates in a block structure. Either way, a cell is something I can click on. With a stand-alone cell I can click on a single cell. In a table I can also, even if it doesn’t look that way. It may look like I am clicking on an entire column but that’s not what happens. Viewing the report in Structure Mode helps explain that; more below.
• Variable
  Ah, variables. Variables are like named formulas in Excel. Variables might be included (displayed) in a cell on a report, but then again, they might not. Crystal does a good job of showing which variables are currently being used. Desktop Intelligence will try to warn me if I try to remove a variable that is in use, but it’s not always correct. Web Intelligence happily allows me to delete the variable and find out for myself. In any case, a variable is a local calculation of some kind. It is not tied to a specific block or report, but it does exist in a single document.
• Data Source
  There is a subtle difference between a data provider (see below) and a data source. The source is, well, the actual source of the data. It could be a universe. In XI 3.x and earlier versions, a universe generally points to a single database. (There could be exceptions to this statement if Data Federator or techniques like database links are involved, but generally this is a true statement.) In a personal data provider it could be a workbook or a text file.
• Data Provider
  A data provider, on the other hand, is an “instance” of a data source. I can use the same source more than once in the same document simply by creating more than one data provider. To make life really interesting, each data provider can contain more than one SQL statement depending on how your universes are created and configured. Life is never simple, is it?

Relationships In A Web Intelligence Document

Now that I have established my list of entities I need to show how they are related to each other. There is essentially a “many-to-many” relationship between data providers and blocks. (Please forgive the “ASCII Art” version of the ER Diagrams; I wanted to leave everything as text.) The overall structure of a document looks like this:

Document -< Report -o< Block -< Cell
                   -o< Cell

-< is a one to many join
-o< is a one to many outer join, where the right side of the relationship can be missing

What does the above diagram tell me? Each document has to include at least one report (much like a workbook has to include at least one worksheet). One document can have 1 or more reports, and a report can have zero or more blocks. A document can also have (but is not required to have) free-standing cells. A block, if it exists, must contain at least one cell (if it is a tabular format) or chart.

There are various block types, but each block must be exactly one type. These include tables (horizontal or vertical), crosstabs, charts, and forms.

For data it looks like this:

Document -< Data Source -< Data Provider -< SQL Statement

What this means is that a document can have one or more data sources. Each source can be used in one or more data providers. And each data provider might generate one or more SQL statements. (I'm not really going to talk about why I might see multiple SQL statements per data provider in this post.)

Finally:

Data Provider >o-o< Block

This is what I talked about earlier. Each block can have show results from zero or more data providers, and each data provider can be used in zero or more blocks. Meaning a data provider does not *have* to be used in a block, and a block can consist entirely of made up data (constants, formulas, or variables) and not reference a data provider at all.

What About Crystal?

Crystal is a more structured report writing tool. In fact, there is no concept of report blocks in Crystal at all. Instead Crystal uses what I have seen called a "banded" report structure, similar to the way Microsoft Access worked year ago when I last used it for anything. However, there are also similarities in that Crystal can also have more than one data provider, more than one data source, and those sources can be merged onto a single report.

Conclusion

I hope this helps new Web Intelligence users understand more about the structure of a document. Some of the concepts are similar to spreadsheets, others are frustratingly different. It all gets better with a little experience.

Related Links

• Wikipedia on ASCII Art

I am going to show screenshots from Web Intelligence 3.0 for this blog post but the same process works in XI R2 as well.

Setting Up The Scenario

To start I need to make sure that my preferences are set for viewing Web Intelligence documents via Infoview in the “interactive” viewer. This is done by clicking on the preferences option (in XI R2 it’s an icon, in 3.x it’s a text link in the menu bar) and selecting the proper viewer under the Web Intelligence configuration area.

Next I created two documents from the eFashion universe. Or rather I created one document, saved it, made a small change to one formula, and saved it under a new name. Now I have two very similar documents and need to find out what is different. The formula in question is simple enough for this scenario: I take the Sales Revenue and multiply by some factor to get a projection. I specifically did this as a formula rather than a variable for this demonstration. For the second version of the report I changed the projection factor.

Interactive Viewing Sidebar

Here’s the trick that Joe shared with me. While viewing the document in interactive mode there are a number of choices that can be made as far as what to show on the left sidebar. The sidebar for 3.0 is shown in the screenshot to the right. It starts out in “Document Summary” mode. (Before I go too much further I should note that the same features in XI R2 are selected with a drop-down selector instead of buttons as shown here, but the results are the same.) Document summary shows the name, author, keywords, and other information that can be updated from the “save” dialog box or the document properties. It’s interesting enough, but it doesn’t help me check for document changes.

But before I go further, look very carefully at what appears just under the sidebar heading… there is a printer icon. And just in case that isn’t clear enough, the word “Print” also appears. Clicking the print icon doesn’t actually print the data shown on the sidebar, at least not right away. What it does is invoke the print process so you can determine how much of the data you want printed.

I will come back to that in a moment. But first, I want to review each of the options presented here as icons and in XI R2 as drop-down selection options.

Here are the rest of the icons from 3.0 enlarged to show detail.

The icons are (from left to right):

• Document Summary
  This pane shows overall document properties, keywords, and other summary information
• Data Summary
  This pane shows data provider details as well as formulas and variables.
• Chart and Table Types
  This pane shows templates for various block types. This information is available when creating documents too, of course, but having it available here allows an interactive viewer to change how a report is displayed. Cool stuff.
• Available Objects
  This pane allows an interactive viewer to see (and therefore drag-and-drop) available objects from a data provider, formulas, or variables onto the report. An interactive viewer can even create new variables on this pane.
• Document Structure and Filters
  This pane allows an interactive viewer to see the document in “outline mode” which shows both page and block layout configuration settings.
• Navigation Map
  This pane allows a user to navigate through sections on a report. This is driven by the content of the report rather than the structure pane listed above.
• User Prompt Input
  This pane will allow a user to review prompt selections.
• Find
  This pane is probably self-explanatory. It’s nice to know that it’s there and I expect to talk more about how it works in a later blog post.

So which one of these is the “magic button” as far as deciphering document differences?

Data Summary

The second button (Data Summary) is where I want to spend the rest of my time for this blog post. When I click that button the outline for my simple document looks like this:

Data source
 Query 1 

	Universe: eFashion (Oracle)
	Last execution time: 5 s.
	Nb of rows: 1016 

 Objects 

 Query 1 

 	Lines
 	Product line. Each line contains a set of categories. 

 	Quarter
 	Quarter number: Q1, Q2, Q3, Q4. 

 	State
 	State located. 

 	Year
 	Year 1999 - 2001. 

 	Sales revenue
 	Sales revenue $ - $ revenue of SKU sold 

 Formulas 

 	=[Sales revenue] * 1.025 

Now did I sit here and type all of that into my blog post? Nope.

What I did was click the “print” icon. I mentioned this earlier when talking about the Document Summary panel, but it appears on the Data Summary panel as well. Clicking the print icon generates this popup window.

If I leave everything turned on and click the OK button the following content shows up in another window.

Now here’s the best part. Clearly I can print from this screen. But I can also grab everything and copy / paste into another application. This is very cool stuff.

Comparing Reports

I use a text editor called UltraEdit that I am very happy with. One of the many features it has is I can compare two different files. I’m not saying you have to run out and purchase the same tool I have. All you need is some sort of program that will let you compare the contents of two different files. (If you run unix you have one built in; see the manual page for the diff command for more information.)

I copied the entire contents of the data summary output for version one of my document and saved it as document_v1.txt. Then I did the same to version two and saved it as (of course) document_v2.txt. Finally I ran the compare tool against these two files. I mentioned earlier that the only thing I changed was a single value in one formula. Here’s what the compare results showed.

The only difference between these two Web Intelligence documents is clearly shown. This is nice.

What About More Complex Changes?

There was only one change made for this simple scenario. I added some new objects and a new variable to version 2 of the document and reran the compare using the same steps outlined above and here’s what I got.

6	 <! Last execution time: 14 s.
7	 <! Nb of rows: 10272
	 !> Last execution time: 5 s.
	 !> Nb of rows: 1016
22	 <!  Store name
23	 <!  Name of store.
24	 <!
28	 <!  Quantity sold
29	 <!  Quantity sold - number of SKU sold
30	 <!
34	 <!  Document Variables
35	 <!
36	 <!  Test
37	 <!  =[State]
38	 <!
41	 <!  =[Sales revenue] * 1.035
	 !>  =[Sales revenue] * 1.025

It captured the different run times and row counts. It showed that the Store name object had been added to the second version. It showed that a new variable existed in the second version, and of course it still captured the fact that the constant value was changed from 1.025 to 1.035 in the second version of the document. The way I know this is everything with <! is from the newer “v2″ version, and everything tagged with !> is from the older “v1″ text file. That’s how UltraEdit works. Other tools will likely be different.

Now imagine if I were working with a complex document with hundreds of variables and formulas and maybe a dozen or more data providers. I am told that there is “something different” about the two documents but I can’t tell exactly what just from a quick look. I now have a process that will identify exactly what is different as far as data providers, variables, and formulas. That’s a huge savings in time and effort.

So thanks, Joe, for sharing your tip.

As a final note, doesn’t Joe have a great name for working as a Business Objects consultant? His brother Frank won a drawing at one of the Insight conferences in Florida a few years back and the emcee had fun calling out his name. Frank… ZABO? Is Frank here? Frank Full-Client maybe?

What About Desktop Intelligence?

One of the advantages of Desktop Intelligence is that it’s open to manipulation via VBA. It’s outside of the scope of this post to provide full details, but I have in the past written code to extract information from various full-client reports and save it into various formats. It would not be a major stretch to apply the same text compare process in order to determine differences between two versions of the same desktop report.

Conclusion

Usually when I learn something new I immediately try to see how far I can take it. It turns out that this trick doesn’t go much further. Of the eight icons on the sidebar only two of them have a print feature: Document Summary and Data Summary. The same techinques could be applied to the output from the document summary but I think the data tab is much more useful.

I should also remind folks that the text editor UltraEdit is not required; it’s just a tool that I have available. There are many free or inexpensive tools for comparing the contents of two text files. The output from UltraEdit is handy in that it can be color coded (as I showed in the first screenshot) or saved to a text file (which is what I used for the second set of output) instead.

Finally, I hope that I have accomplished one last but important thing with this blog post. Joe seemed really surprised that I had not encountered this feature before. To be honest, I do not spend much time in Infoview. I am usually using the java panel for report development or debugging efforts. Since I don’t spend much time in Infoview I haven’t spent that much time checkout out all of the features, and therefore had never seen the “print” icon that was available even back in XI R2.

My point is this: everyone has something to share. Some of the techniques that you may use every day without thinking twice about them might be new and exciting to someone else. For every person who read this blog post and thought to themself, “I already knew about that” I bet there are a dozen or more that are going to say, “Wow, I never knew I could do that!” So go ahead and do a presentation for your local user group. Write a blog post. Share something on BOB. Pay it forward. Pass it on. Your mileage may vary. Batteries not included. All that stuff.

You might just save someone else a lot of time, and feel good doing it.

Filling Out A Calendar

My report requirement states that all months – whether they have sales data or not – must appear in my output. Because Web Intelligence cannot make up data to fill out the missing months, I need to have an alternative source. In the first post in this series I showed how to do that with a personal data provider as I built out the data in a spreadsheet. This time I am going to show how to do the same thing using only a universe. In order to provide a complete (clean) solution I am going to have to make a new object in my universe.

The first step is to create a query that returns some data. I am going to use the same data that I started with on the last post, which is six months of sales revenue data from the eFashion database. I am also limiting the data to the year 2001 in order to match the example from the last post.

Next I will create a second data provider from the calendar table that includes all of my month values. Since this query has no restrictions (conditions) and no measures (no joins to any fact table) it will be guaranteed to return every month. At this point in the last post I detailed how to merge (link) two data providers together. This time that step is not required. That’s the first advantage of using universe data to perform this step: the dimension objects that are shared across my two queries are already merged (linked). When I used personal data, I had to do the merge step myself.

Once I drag the merged month into my report block I will see all 12 month names. As I did last time I will also replace the existing Month Name detail object with the one that comes from my calendar data provider. The results from these operations are shown in the screenshot below. The “Discontinued” value comes from a custom format used in the eFashion universe. The universe was designed with the assumption that if any cell was empty (as opposed to zero) then there were no sales for that item, and it must have been discontinued.

I would like to insert a zero value into those “Discontinued” cells. I could make it look like it has a zero by changing the format. But if I want an average of that column to be impacted by the fact that I had zero sales for six months out of the year, I need to have an actual zero in the cell, not just a custom format. It’s at this point that I have gone as far as I can without talking to my universe designer.

Creating and Using Placeholder Objects

As a universe designer I can help solve this problem. I might create a special class called “Placeholder Objects” that includes new objects used to solve this type of problem. (I have also seen these objects called “dummy” objects because they are only going to work in very specific situations. They’re not “smart enough” to be used by themselves.) When I used personal data to solve this issue I created a new column in my spreadsheet and put a zero in for every month. In my universe I will create just one object with a zero value in the select clause and nothing in the tables. Note that this object will not parse; that is expected.

Since this object does not have a table associated with it I will need to make sure that it’s used with at least one “real” object in a query. As a universe designer I can’t enforce this. But if a report writer tried to use this object by itself the query would not run. I have seen some folks suggest that objects like this should be pointed to the DUAL table (in Oracle) or something similar. I disagree. The only benefit to doing this is that the object will parse. But at the same time I am fixing that issue I would be introducing an entirely new set of issues because the DUAL table isn’t typically going to join to any other table. It’s much more effective to leave this object in a state where it won’t parse than to deal with all of that mess.

Rebuilding the Query

Once I have created my dummy object and exported the updated universe I can fix the problem. In fact I can fix it a couple of different ways. First, I could mimic the solution from the last post by adding the zero object to my calendar-only data provider. Since I have already covered all of those steps I thought I would show a different technique for this post. I am going to use a UNION query.

A union requires that all of the queries (you can have two or more) match. Each query has to include the exact same number of objects, and the object types (character, date, number) have to match as well. For the scenario I am using for this post each query will have three objects. Both queries will include the Month and Month Name. The first query will also include the new Zero object. The second query will include the regular revenue object. Here are the two screen shots showing what this query looks like.

Query Panel Union Part 1

Query Panel Union Part 2

Notice that the first query does not have any conditions at all? The second query includes the condition that I created to drop out months (it only shows months with an “e” in their name) as well as a limit to show data only for 2001 as mentioned earlier. When I run the combined queries I see this output.

At this point the data looks great, and it was obtained with a minimal amount of fuss. All of the months show up as required. There are no merged dimensions here since everything is returned in a single data provider. However, there is one item that needs to be fixed. Check out the column headers… they’re not exactly “user friendly” at this point. What happened? It turns out that column headers from a union query are taken from the first pass of the data provider. As a result my column header says “Zero” instead of “Sales revenue” as I would expect. Fortunately Web Intelligence (the Rich Client in this case) makes this process extremely simple. All I have to do is open the query panel and drag “Query 2″ in front of “Query 1″ in the union area of the query panel and rerun. After rerunning the query I am left with a report block that is missing a measure. This is okay, because the object named “Zero” has been removed from the query, and a new object “Sales revenue” has appeared. All I have to do is drag that new measure onto the block and I see these results.

Notice that I have added an Average calculation in the footer to show that the zero values are being included during the calculation. The result obtained via this process is now essentially identical to that shown in the first post about making up data.

Conclusion

So far I have shown two different techniques that can be used to “make up” data for a report: personal data and universe data. Which is better? As with many questions the answer is “It depends.” Personal data has an advantage in that it can be built on the fly by a report user and does not require any universe maintenance. Universe data has an advantage in that the extra objects are reusable and are available to anyone. With proper training users can build queries that can include extra data that does not exist in the database.

Personal data requires me to know how to merge the results. With universe data the merge step (as long as I use the same objects) is performed automatically.

In this post I also showed two different techniques for getting the “filler data” into my report. With personal data I have to create two data providers and merge the results. With universe objects I have the option of creating a single data provider using a Union technique and my placeholder object.

The placeholder measure object that I created is not limited to this particular report. I can use (and reuse) it for any other report with similar requirements. Since the object is not tied to the calendar (or any other dimension) I can use it anywhere. It can be used to merge data providers or in a union query. I can create similar “dummy” objects for dates or text fields as needed. Ultimately I can decide if I want these special objects to be visible to all users or to a selected subset of “power user” report writers using the standard security process.

Next Time

For Part III of this post I am going to create a more complex example where data might be missing from more than one dimension (instead of just month as I have used in the last two posts.) I am also going to show cross tab and a section output because each of those can cause some interesting challenges as well. Maybe I’ll even do a sectioned cross tab.

Related Posts

• Making Up Data Part I: Personal Data Providers

How can I display each individual value selected in the UserResponse() function on its own row in a table?

I believe the reason for wanting this behavior is fairly clear. If you have a bunch of complex data it would be much easier for the report consumer to read this:

XYZ123ABC
DEF456HIJ
BOB111BOB

Instead of this:

XYZ123ABC;DEF456HIJ;BOB111BOB

As is often the case, the strict answer to the question is “you can’t do it that way” with Web Intelligence. Despite the “you can’t…” answer I am going to show how it can be done.

Printer History Lesson aka Carriage Return + Line Feed To The Rescue

Before I show my solution, I have a history lesson to share. In the old days (and I mean really old, like before laser printers) most business printing was done with impact printers. An impact printer was essentially a typewriter connected to a computer. It would print one character at a time. Some really expensive printers could write an entire line at a time. Some even more expensive ones would be able to write forwards and backwards to save time. Fancy stuff. But the point is, everything was characters and lines. If a program needed to print something bold, the printer would back up and print the same characters more than once on the same line. That’s where special printer control characters came in.

The print head or “carriage” was where the next character would be printed. Normally it would flow across the page printing one character at a time. At the end of the line, a carriage return signal was sent. This would tell the printer to send the carriage all the way back to the beginning of the line. If the printer started printing again at that point, it would overwrite the existing line of output. (As mentioned above, that’s one way that bold type was done.) To avoid this, another special printer instruction character was sent just before the printing started: the line feed. This character told the paper to roll the paper forward one line so that the new output would be on the next line.

To recap: carriage return sends the print head back to the left side of the page, and line feed moved the paper forward one line. Carriage return was (and still is) character 13 in the ASCII sequence, and line feed is character 10. (These are also often abbreviated CR and LF.) There are plenty of others. If my memory is correct a tab is character 9 and the “bell” if you wanted the printer to make a noise was character 7. But I digress.

Fast forward to today. The characters still retain some of their meanings, even if we don’t use impact printers anymore. So if I want to push some content down and start on a new line, I can include these characters as part of my output and they will behave the same way. Web Intelligence offers the function Char() which will allow me to introduce a specific character using the proper number. Web Intelligence also offers the Replace() function which will allow me to replace one character (or string) with another character (or string). You might see where all of this is heading…

Prompt Processing

I created a simple query using the Prestige Motors universe. The query includes a prompt that asks the user to select one or more automobile manufacturers when the report is refreshed. The text of the prompt is “Car Makers.” I can easily display the results of the prompt (the values the user selected) on my report using the UserResponse() function. What I want to do is display each selected value on its own line.

How am I going to accomplish this? I’m not going to put each value on an individual row in a table. As I already said that can’t be done. Instead, I’m going to make them show up on different rows in a single cell. The readability should be the same, even if the applied technique is different.

The final piece of the puzzle is to observe just how Web Intelligence captures and displays prompt responses. Here is a screen shot showing the prompt after I have selected four different car makers. Notice that the four values are separated by the semicolon ; character.

With a simple formula I can echo the prompt selections in my report title cell like this:

Notice that the exact same format is used in both the prompt window and the output in the title cell. Each selected value is separated by a semicolon. I can work with this.

The Solution

I created a variable named “Picked Cars” with this formula:

=Replace(UserResponse("Car Makers");";";Char(13)+Char(10))

The inside portion of the formula looks like this:

UserResponse("Car Makers")

That bit captures the selections and returns them in the format shown above, where each car name is separated by a semicolon. The outer portion of the formula uses the Replace() function to find those semicolon characters and replace them with the combination of character 13 (CR) and 10 (LF). The net result?

Everywhere there is a ; it gets replaced by the CR+LF and therefore the output moves down to a new line and back to the left side of the page. Strictly speaking there is no page nor is there any paper to “line feed” but the functionality is the same. Simple enough. Here’s what the results looked like after I created this variable and used it in my header.

What about the formatting? It turns out I didn’t need to do too much. I made sure that my cell was set to automatically fit the content (since it could grow or shrink based on how many cars were selected) using the “Autofit Height” setting as shown here.

The next item I planned to update turned out to be set for me by default. (This example was documented using the Web Intelligence Rich Client so other tools may require this step.) I clicked on the edge of the block and made sure that if the title cell above it grew too big that the block would move down the page. This is done by setting the relative positioning of the block, as shown here.

As I mentioned, this dialog box did not require any changes for this scenario. It was already set the way I needed it to be. Since the block is positioned relative to the cell that contains my [Picked Cars] variable it will move up or down the page based on the size (height in this case) of the cell.

Conclusion

Can I split out each individual user response and display it on a separate row in a table? No. Can I achieve the same visual result using other techniques? Sure. That’s what it’s all about, finding a different way to answer a question. This solution does have one problem… it will not work cleanly if there are semicolon characters in the data returned from the prompt. I did not test this with Desktop Intelligence, but since it includes all of the functions used in this example I am confident that it would work the same way.

I was challenged when I posted this solution on BOB as to why I used both CR and LF characters in my formula. To be honest, when I changed the formula so that it only included one or the other, either seemed to work just fine individually. Despite this, I opted to leave both in my formula.

What can I say… old habits die hard.

I have only six months of data, but I have to show all twelve months in my report. How can I show the missing data?

The trick is to understand that Business Objects does not make up data. It just doesn’t have that capability. And you can’t edit the report directly in order to add those extra values to the result set. However, there are a couple of tricks that I can share that will help solve this issue.

Special Note to Blog Readers: This post is fairly long because it has quite a few screen shots to illustrate the process. As a result it will be presented in several pages. When you get to the bottom of a page just click on the next page number from the links provided to continue reading the post. Since this is the first time I have used this feature I thought I would warn you about that. Now on with the solution.

Using Personal Data

One of the easiest ways to make up data for a report is to use a personal data provider. This technique can be used in all versions of Desktop Intelligence and also in the newer “rich client” version of Web Intelligence. While the specific steps are unique to each tool the general concept is the same, so I will only show how to do this using the rich client. Screen shots for this blog post were taken from version 3.0 of the Web Intelligence Rich Client.

The first step is to determine what is missing and therefore what you need to make up as filler information. It’s also important to know that the format of your personal data has to match what is coming from your enterprise data source or the data won’t merge properly. For this scenario I will show some data coming from the eFashion database. I am going to restrict the data to months that include an “e” in their name, just to make sure that I have holes in my data. If you’re wondering, that includes February, June, September, October, November, and December. Why did I pick that? No reason, really, just thought it would be fun.

After applying that condition (along with another to restrict to one specific year) I see the following data on my report.

In the eFashion universe the Month is a dimension object and is a numeric value. The Month Name is a character string containing the month name spelled out (October rather than just Oct) and is configured as a detail object. This is important. When I want to merge two different data providers together I have to perform that process (the merge or link step) using only dimension objects. I cannot use detail objects. That means my personal data provider will have to include the month number as a dimension, and it should also include the month name as a detail.

After building up this query I have six months of data, as described in my original scenario. To meet the report requirements I have to show all twelve months, including those that don’t have sales that meet my current criteria. How can I make this happen?

Making Up Data

I am going to use a spreadsheet to make up my data, but I could just as easily use a delimited text or CSV file created with any text editor. The trick is that I have to structure my personal data so that it matches what I already have from the universe. To solve the scenario I am currently working on that means that I need the month number and the month name in my personal data. Here’s how that looked when I was done.

I saved this data on my computer as a standard spreadsheet file. I could also have used a comma separated values (CSV) or other structured text file just as easily. The next step is to import this data into my document.

Adding Personal Data

In order to add personal data to my document I open the query panel, right click on the current query tab, and select the option to add a new data provider. After I do that I see this dialog box.

After making sure I select “Local data source” I move on to the next step, which is to browse my computer and find the spreadsheet file I saved earlier. Once I find the file I am presented with a few configuration options as shown here.

The filename is there, of course. Since I am using a spreadsheet I can pick which sheet from the workbook contains my data. I can also determine whether to use a named range, to enter the range myself, or to let the client try to figure out where all of the data is for me. For this simple case I mark “All fields” and let the client do just that. Finally, I confirm that the box for “First row contains column names” is checked and click the Next button. Here is what I see at this point in the process.

There are a couple of problems with this result. The client is trying to be helpful, but it’s making the wrong decisions for me. The Month column contained all numeric values so it was imported as a measure. The Month Name contained character data so it was imported as a dimension. Both of these assumptions are wrong. I need the month number to be a dimension so I can link it to the value I already have, and I need the name to be a detail in order to match the existing structure on the report. Fortunately, since this is my own personal data, I am allowed to make changes to the configuration.

How can I calculate the number of business days between two dates?

The easiest answer is to use a calendar table with a flag for business days because that takes care of everything quite easily. You can even mark holidays along with weekend days and make the results more useful.

But suppose you don’t have a calendar table. Suppose you don’t care about holidays, you simply want to count the number of Monday – Friday days between a range of dates. Can you do that?

The answer is coming if you read the rest of the post.

Note: This technique was shown in public on Monday at the GBN conference. I had written this blog post some time ago but waited to release it until after the presentation had been delivered. The presentation has been uploaded to my blog and is available for download from the conference page.

Web Intelligence Functions Required

This formula is going to use the following functions, so I will introduce and describe each of them a bit here.

• DaysBetween()
  Input values: two date values
  Output value: an integer number (no fractions) showing the number of days between the two dates
• DayNumberOfWeek()
  Input value: date
  Output value: Number from 1 to 7 denoting the day of week (1 = Monday, 7 = Sunday)
• Truncate()
  Input value: numeric expression or value, number of desired decimal places
  Output value: numeric value truncated to specified number of decimals, for example Truncate(3.14159, 2) = 3.14
  Truncate does not round, so Truncate(5.9; 0) returns 5 which is what is needed for this solution
• Mod()
  Input value: two numbers
  Output value: the remainder of the first value divided by the second
  Example: Mod(10; 2) = 0 and Mod(11; 2) = 1 and Mod(15; 7) = 1
• Substr()
  Input value: string, starting position, length
  Output value: part of the string
  Example: Substr(’1234′;2;1) = ‘2′
• ToNumber()
  Input value: string representation of a valid number
  Output value: number
  Example: ToNumber(’2′) = 2

Counting Weeks

One of the first steps in solving any sort of calculation is to look for shortcuts. Clearly there will be an infinite number of input values for my calculation so I need some way to avoid too many possible choices. Because of this I am going to start by counting the number of weeks between two days.

Take a look at the calendar on the right-side of my blog and try an experiment. It turns out that no matter which day of the week you start on, if you run for seven days you are guaranteed to get exactly two weekend dates. No more, no less, exactly two. Try it out and see. Start on a Sunday and count forward seven days and what do you get? Exactly one Sunday and one Saturday = two weekend days. Start on Monday and count forward seven days and the same thing happens. Start on Tuesday… on Friday, or even Saturday… and any range of seven days always has exactly two weekend dates included.

I can extend this to any multiple of seven days. If I have 14 days I get 4 weekend days. With 21 days I have 6 weekend days. If I multiply by five instead of two I get the number of weekdays in the range, therefore I am going to start my calculation using this pattern. Here is the start of the formula:

Truncate(DaysBetween([Start Date]; [End Date]) / 7 ; 0)

Let me break that down. First the DaysBetween() function is used to determine the number of days between the starting and ending dates for my range. Next I divide the number of days by seven which will result in a decimal result. For example, 15 / 7 = 2.14. Finally I use the Truncate() function to remove the decimal, which converts 2.14 to 2. That gives me the number of full weeks between my two days. Here’s a table showing results for some sample day ranges.

Based on the conclusion I made earlier, I can use the number of weeks and multiply by five to get the number of week days included in the range. In order to get that number I make one small modification to the formula presented above; I multiply the result by five. The result:

Truncate(DaysBetween([Start Date]; [End Date]) / 7 ; 0) * 5

All I have left to deal with are the leftover of days that don’t make up a full week. The decimal part of 2.14 is 0.14 which represents one extra day in my range. In order to know whether to count that day or not, I have to know the name (or alternatively the number) of the day my range starts on.

How Many Extra Days

There is a function called DayNumberOfWeek(). It numbers Monday as 1 and Sunday as 7 so I’m going to pretend that my calendar looks like this:

M T W T F S S
1 2 3 4 5 6 7

What I need to get next is the number of days that are not a full week. I used the Truncate() command earlier to get the number of full weeks. It removes the decimal part of the division result. What I need now is the decimal part, and the Mod() function can be used to get it. This function does a division and returns the remainder. The first argument will be the total number of days, and the second argument will be seven as there are seven days in a week. The result will be a number that ranges from zero (0) to six (6) because those are the only possible remainder values when dividing by sevent.

For example, Mod(15; 7) = 1. 15 / 7 is 2 with 1 left over, therefore 1 is the return value from the Mod() function. To apply that to my current problem: 15 days is made up of two full weeks and one extra day. Twenty days would be Mod(20; 7) which results in 6 remaining days. Mod(21; 7) is zero because there are exactly three weeks and no extra days.

Can I put these two pieces of information together? I can easily get the number of the day in the week, and I know how many extra days there are. I should be able to do some math…

So how do they work, exactly? What do I mean by unbalanced data providers? How is it fixed with these functions? Are there any special conditions to be aware of?

Merged Data Provider Review

I recently published a blog post that described unbalanced data providers. If you want the full description, I have included a link at the end of this post. If you want the executive summary, here it is:

When you have two (or more) data providers in the same document and want to merge the data, you have to link them using dimension objects. If you don’t have the same number of dimensions from each data provider, then they are unbalanced. That can cause issues.

It took me twelve hundred words (and six pictures) to provide a detailed explanation in my earlier post. In the sample report I used for that post I had one common dimension (Year) and one unique dimension from each side (Resort and Region). I showed how I could use Year with either measure since it was a shared (and linked) dimension.

I also showed how (and explained why) trying to use the “extra” dimensions like Region or Resort caused problems. I don’t intend to repeat everything here; please use the link at the end of this post to read the prior post if you are unclear on anything so far.

What About MultiCube()?

One of the comments I got on that post suggested that MultiCube() could be used to fix the issue. My response was accurate as far as I was concerned, but later when I read it again I realized it could also be quite confusing. Here’s what I said:

…the MultiCube() function will help if data providers are unbalanced “upwards” but not “downwards” …

After reviewing my response, I realized that it was quite likely that I was the only person that understood what I meant by that statement. While at times I do enjoy talking with myself that’s not why I have this blog. So I am going to try to explain that a bit further.

The problem with trying to use MultiCube() or the Web Intelligence equivalent ForceMerge() to solve this specific case is that the data doesn’t support the solution. There is no hierarchy or relationship between Region and Resort. (Yes, they are both related via links to the fact table but that’s not what I mean.) Resort is a place. Region is an attribute of the customer. There is a many to many relationship between regions and resorts, and that sort of relationship is essentially worthless. If I can’t determine how to put things together, how are the numbers going to make sense? Neither of these functions can create a relationship from nothing.

Yet there are some cases where I can fix unbalanced data providers. I need to talk about the data first, then I need to create a new document with the proper structure in order to show how the ForceMerge() function works.

Hierarchical Data

Here is what I consider to be a really good example of why this is a problem. Most people think of time as a really clean hierarchy. On the surface the hierarchy is simple: Months go into quarters, and quarters go into years. Yet if I create a query with the Quarter and Revenue from the Island Resorts database here is what I get.

Take a look at the data and tell me how you would break out the quarter data into different years.

Go ahead and think a bit, I can wait.

One-Mississippi.

Two-Mississippi.

You can’t do it, can you? You don’t have enough information. As defined, with the format “Q1″ and “Q2″ and so on, there is no indication as to which year the quarter revenues are coming from. You can’t “break down” the quarter by year with the information that you have. That is what I meant by data being unbalanced “downwards” in my comment on my prior post. I can’t break data down because the information simply isn’t there.

What I can do, however, is roll up or move “upwards” via a hierarchy. That’s where the ForceMerge() function comes into play.

Introducing the ForceMerge() Function

Here is what the help text says for the ForceMerge() function.

Forces Web Intelligence to account for synchronized dimensions in measure calculations when the synchronized dimensions do not appear in the calculation context of the measure

I will explain what this means, but first I am going to build my example. My document will have two data providers from the Island Resorts Marketing universe. My first data provider has dimension values Country and Resort and the Revenue measure. My second data provider has the dimension Resort and the measure Guests. I will link the two data providers by Resort as it is the only common dimension.

Data Provider 1

Data Provider 2

Data Provider Results

Merged Dimensions

From this point I will create a merged block using all of the measures and both dimensions. Here are the results from that experiment.

Everything is fine so far. The issue becomes evident when I remove the Resort object from the block. Remember that my two data providers are merged (synchronized) by Resort. When that object is no longer present in the block, the measures roll up. Revenue is still okay because the Country object was part of the query context of that data provider. The Guests measure fails because it is now being viewed out of context. (At the risk of becoming tedious, I will point out one last time that this issue is covered in much more detail in the first post in this series.)

I have created a problem. Next, I will explain how to fix it.

ForceMerge() In Action

All of the information I need is there in the document. It just does not show up in the block. Here once again is the help text for the ForceMerge() function:

Forces Web Intelligence to account for synchronized dimensions in measure calculations when the synchronized dimensions do not appear in the calculation context of the measure

Simply put, the function tells Web Intelligence to use all of the information from the cube, rather than being limited to what is shown in the block. I will create a new variable with this formula:

=ForceMerge([Number of guests])

I will replace my “broken” measure with my new variable and observe the results.

I haven’t changed my dimension linking. I haven’t changed my data providers. I didn’t even change the block structure, other than to replace the broken measure with my corrected calculation. If you were looking for something more complex, I’m afraid that’s all there is. The complicated part is understanding just what the function does and when to use it, rather than how to use it. I am glad this function is now available in Web Intelligence starting with XI 3.x. In my opinion, it should have been there since merging data providers was an option.

Summary

The magic of the ForceMerge() function (or if using Desktop Intelligence the MultiCube() function) is that it changes the behavior of the report engine. Normally when dimension values are removed from a block they are taken out of context. By using one of these functions I can specify that linked dimensions still have to be considered by the measures even if those dimensions no longer appear in the block structure.

Even with this magic function in my arsenal, I cannot rely on it to fix every problem that I have. If the data can be “rolled up” by the calculation process then I am fine. If the data has to be “broken down” then I am in trouble, and these functions will not fix the issue. Each resort exists in exactly one country so I can move up the hierarchy from the resort link and the results are valid.

Special Note
If there are smart measures in a universe, then ForceMerge() will not work. In order to calculate the results correctly, smart measures (using the “database delegated” projection function) need to have the block structure in place to define the query. According to the documentation, the ForceMerge() function will return a #MULTIVALUE error if used in this situation.

Related Links

• Blog Post: What Are Unbalanced Data Providers… And Why Should I Care?
• Blog Post: “Smart” aka Database Delegated Measures

What was that “Mississippi” thing earlier?
From Wikipedia:

Children in the United States and Canada often count ‘One-Mississippi, two-Mississippi’ during informal games such as hide and seek to approximate counting by seconds.

So now you know.

Merging Data Providers in Web Intelligence

Dimensions are the “keys” of a data provider. When two database tables are linked in a universe, certain columns are used to join them together. Dimensions serve this role in linking two data providers. When two (or more) data providers are built in Web Intelligence from the same universe they are automatically linked together using any common dimension objects. What happens when two different data sources are used? In that case, the report writer has to merge the information manually. And if the dimensions from each data provider are not identical, the report has what I call “unbalanced” data providers. That’s what I want to describe in this blog post.

For my sample I will join two data providers from the Island Resorts Marketing universe. I am going to structure my queries in such a way that they will result in an unbalanced scenario. For my first query I will return Resort + Year + Revenue. For the second query I will return Region + Year + Number of Guests. Resort, Year, and Region are all dimension objects, of course, and Revenue and Number of Guests are my measures. Here are the raw results from each individual data provider. (I am using Web Intelligence XI 3.0 for this demonstration.)

I have one common dimension. I also have two dimensions that are unique, one from each data provider. Since both data providers come from the same universe, the common or shared dimension value (Year) has already been linked, as shown here.

Because the Year object was already merged, I can create a report block that contains Year, Revenue, and Number of Guests. I can also do local variable calculations like Revenue per Guest using simple division.

This is what happens when everything works. What if something doesn’t work?

Mixing Dimensions Generates Unexpected Results

This block shows what happens when I add Resort to the block shown previously.

Notice anything interesting about the data? About the Number of Guests column specifically?

When the year repeats, the Number of Guests repeats. The Revenue measure seems fine. Before I explain this, let me exchange the Region object for the Resort object and show what happens.

In this case, the Revenue column is strange and the Number of Guests looks okay. To make matters even more confusing, here’s what I get if I apply a Sum() to the Revenue column after doing a break on the Year…

The sum on the footer is identical to the values displayed in the block! Any idea what’s going on?

Measures Do Not Sum, They Project

The secret is that Web Intelligence does not sum a column of values like Excel does. A few weeks ago I discussed the difference between SQL Aggregation and Report Projection which helps explain this. Simply put: when a measure is combined with a set of dimension values in a block, the measure will project (roll up) based on the structure of the block. This is also called the context of the measure within the block. (Please note: The use of the word “context” in this case should not be confused with the concept of a context — set of joins — in a universe design.)

Because of the way I structured my queries, the Revenue context is determined by combinations of the Resort and Year objects. The context of the Number of Guests was set by Region and Year. The Year value is the only common dimension between the two data providers, which is why the block with only the Year dimension object showed valid data. However, if I include a dimension that is not part of a measure context, a problem occurs. When Region was added to the block the Revenue was wrong. When Resort was added the Number of Guests was wrong. In each case, the measure was out of context for at least one dimension.

To repeat: When I created the block with Year as the only dimension everything was fine. Each measure includes Year as part of its context. When I added Resort, the Revenue measure was still in context (and therefore showed the correct results) but the Number of Guests object was out of context. As a result, it used the only context it could (Year) and rolled up to that value, then repeated that value each time the year appeared on the block.

What about that weird behavior of the Sum() function? It turns out that a Sum() doesn’t really add up the column values. It does a projection instead. When the block included both Year and Region, the Revenue measure was valid for only the Year therefore it projected to that value. As a result, the measure value is repeated for each different Region in the body of the block as long as the Year is the same. Adding a break on Year created a footer for the block. Since the context for the footer was Year only, that made the Revenue projection (total) for the Year the same on the footer as it was for the body.

Unbalanced Dimensions

Now that I have been through all of that, let me finally explain what I mean by “unbalanced dimensions” as used in the title of this blog post. In my data providers I have Year as a common dimension and Resort hanging off of one side and Region hanging off of the other. Web Intelligence will prevent me from adding both Resort and Region to the block at the same time, but it will allow me to add one of them. Assuming I do that (add an extra dimension) I will have two dimensions from one data provider and one from the other. Two against one is unbalanced.

Resort, Year <-> Year
        Year <-> Year, Region

It doesn’t matter which side the extra dimension comes from either.

If you have more than one data provider the following rules apply.

1. All common dimension values should be linked
2. Measure context is defined by the dimensions in their source data provider
3. A block with linked dimensions and measures works fine
4. Unlinked dimensions from one data provider can be used; others will be designated as incompatible
5. Measures from the data provider that includes the extra unlinked dimension will be fine
6. Measures from the other data provider will repeat for each unlinked dimension value because they are at least partially out of context

With all of these complications, Web Intelligence does what it can to show the data properly, even to the point of having the footer totals be correct despite the presense of an unlinked dimension. It’s better to manage your data providers to avoid this challenge, but it’s nice to know how things work if you can’t do that. There are other techniques that can help (like creating variables to demote a dimension to a detail object, if that action is appropriate) but this post is long enough already.

We have some fairly complex reports at my current client. Of course the word “complex” is not very clear; what is simple to me might be complex to someone else. So let me quantify what I mean by “complex” in this case. The report in question has multiple data provders. Each data provider generates multi-pass SQL because of contexts. Altogether there are 29 data providers with an overall total of over 100 SQL passes. That’s what I mean by a “complex” report, and I think most folks would agree with that assessment.

The challenge is that for this (and other) complex reports we have been encountering severe issues. In some cases we are unable to edit the queries without restarting our browser session. We also have encountered a number of problems during the process of repointing the document from one universe (development) to another (user acceptance testing) within the same environment.

Current versions of Web Intelligence offer the ability to repoint a document using the following steps:

• Open the report
• Edit the query
• Click the [...] button next to the universe in use
• Select the desired destination universe from the list
• Review the object mapping and click OK to continue

Side note: I really really like the fact that I can remap individual objects rather than having the process fail as a result of something being missing, but that’s another blog post…

Our problem is that after performing the steps listed above, we wait. And wait. And wait. Eventually the session times out. Finally (after much beating of head on desk) I remembered to activate the Java console. (Click Tools and then Sun Java Console from the Internet Explorer menu bar to start the console.) The console shows what’s going on within the Java environment. It seems that I was running out of memory during the attempt to switch universes, and Webi was not capturing the exception properly. Instead of letting me know there was an issue, it was simply… not responding. Nice.

Setting Java Heap Size

Long story short, the memory exception was related to the heap size. So I did some searching on the Sun Java forums and found a number of tips about how to increase this parameter. The exact process will be different based on the version of the JRE that you are using, but the desired parameter is the same either way. If you open a command window and type java -X (which is case sensitve, it must be a capital “X”) the program will return a list of valid parameters. For my version I got this:

java -X
    -Xmixed           mixed mode execution (default)
    -Xint             interpreted mode execution only
    -Xbootclasspath:
                      set search path for bootstrap classes and resources
    -Xbootclasspath/a:
                      append to end of bootstrap class path
    -Xbootclasspath/p:
                      prepend in front of bootstrap class path
    -Xnoclassgc       disable class garbage collection
    -Xincgc           enable incremental garbage collection
    -Xloggc:    log GC status to a file with time stamps
    -Xbatch           disable background compilation
    -Xms        set initial Java heap size
    -Xmx        set maximum Java heap size
    -Xss        set java thread stack size
    -Xprof            output cpu profiling data
    -Xrunhprof[:help]|[:=, ...]
                      perform JVMPI heap, cpu, or monitor profiling
    -Xdebug           enable remote debugging
    -Xfuture          enable strictest checks, anticipating future default
    -Xrs              reduce use of OS signals by Java/VM (see documentation)
    -Xcheck:jni       perform additional checks for JNI functions

The -X options are non-standard and subject to change without notice.

The one I am interested in is Xmx which can be used to adjust the maximum heap size used by the JVM. Based on some reading, setting this value too high can be just as bad as having it set too low, so I didn’t want to go nuts right away. I tried several different values, using my complex report as a test case, and ultimately found that 128m was just fine for me.

The screen used to update this changes based on the version of Java that you are using. I have included screen shots for both 1.4.2 and 1.5 at the end of this blog post.

Conclusion

After making this simple change I was not only able to change the data provider source for my complicated report, but other operations became much more stable as well. I also think that many operations were faster.

Or maybe it was just the fact that they completed at all that made them seem faster.

Java 1.4.2 Heap Configuration

Java 1.5.0 Heap Configuration

A few months ago about half of the reporting team at my current client called me at various times over a week saying they were having problems running reports. Eventually I figured it out… they all had one thing in common: automatic Java updates. Turns out that was a Very Bad Thing™ to do.

As these folks were using their computers it seemed that the Java application was busy in the background checking in with the mother ship. Since version 6.0 (aka 1.6) had been released, the Java-bot dutifully went about upgrading the system. The problem is, version 1.6 of the JRE doesn’t play nicely with Web Intelligence. The primary symptom? You can’t get a prompt window to display.

I have seen this manifest it self in two ways. First, you cannot run queries that include prompts. These can be prompts built in the query panel itself, predefined conditions, or even automatic prompts embedded in the universe. Second, if you have the “refresh on open” property turned on you cannot even click on the “Modify” link to open the document in the Java query panel. It wants to work, and it tries to load, but it will get stuck on the “Building interface” step shown on the splash screen. Very frustrating, and not really a good indication of what the underlying issue is either.

Turning Off Automatic Updates

So here’s how to turn off Java automatic updates. If you look in your Windows system tray (it’s the part of your task bar where all the icons appear that show you how much of your system RAM you’re wasting on trivial stuff like the Weather Bug ) there will be a Java cup. If you right-click on that icon you can open the properties. Here is where you want to go to disable the automatic updates:

The screen capture above is taken from Java 1.4.2_04 and shows where the checkbox is hidden. If you clear that checkbox you will only get upgraded when you want to, and you also will not be nagged about pending updates. Java 1.4.2_04 is what you get from Web Intelligence XI (current versions as I write this, that may change in the future) when you first load the Java query panel.

Getting Java 1.4.2 back

Finally, here is a link to download older Java versions. What you need are the JRE (Java Runtime Environment) 1.4.2 or 1.5 (aka 5.0). Version 1.6 (aka 6.0) is not supported by Web Intelligence as of XI Release 2 Service Pack 2.

http://java.sun.com/products/archive/

And on additional bonus: if you don’t want to see the java cup taking up space in your system tray, there is an option for that as well. Here’s a screen shot showing where that setting is:

Edited on September 4, 2007
It just occurred to me while answering a post on BOB that you don’t really need the download links I provided above. If you remove all of the Java JRE programs from your system then Web Intelligence will automatically push down a supported version the next time you load the java query panel.