Xcelsius Challenge for Community Poll Results

If you’re interested, you have until April 30, 2010 to enter your submission.

I’m thinking of creating a pie chart.

Some databases require you to use actual column names in an ORDER BY clause. Like this:

select first_name, last_name, phone
from employee
order by last_name, first_name

Other databases let you take a shorter approach and sort by the position of the column in the select clause, Like this:

select first_name, last_name, phone
from employee
order by 2, 1

To be honest, I don’t like the shortcut. I would rather see explicit column names in my order by because that way I know exactly what is being sorted without having to refer back to the select clause. Another advantage is that if the objects in my select ever change, my order by is not affected.

There is a parameter found in the .PRM file for each database named SORT_BY_NO. When you see that name, what do you think it is? Every time I see it I assume that it is used to determine whether the SQL will contain numbers in the ORDER BY clause like order by 2, 1 instead of order by last_name, first_name. But that’s not what it does at all. Instead of doing what I described above, this parameter is used to determine if a query can be sorted by a column that does not appear in the select clause. That makes sense, doesn’t it? It should be called SORT_BY_IN_SELECT or something. But it’s not, and here’s how it works.

Sorting By “Something Else”

I have a period calendar table where the period names are P01, P02, and so on. The years are fiscal years 2009, 2010, 2011, and on from there as you would expect. The user expects to pick a period and year combination. However, they want to see it in that order… period, and then year. As a designer I can easily combine the two values together in the format PPP YYYY with a concatenation operation. But then the LOV displays in alphabetical rather than chronological order. So I see this:

P01 2008
P01 2009
P01 2010
P01 2011
P02 2008
P02 2009
…

Instead of this:

P01 2008
P02 2008
P03 2008
…
P01 2009
P02 2009
P03 2009
…

This is not what the user expects or requires, but is easily solved by editing the LOV query and adding a custom ORDER BY clause like this:

select period || ' ' || year
from fiscal_calendar
order by period_start_date

Sorting by the period start date would cause the alphabetical list to be sorted chronologically instead. However, doing any sort of manual editing – even in a simple LOV query – is something I want to avoid. Any time I have to click the “do not regenerate SQL” option it leaves me open for problems later on. I could add the start date to my query and sort it normally. However, I don’t want to do that as it would clutter the display.

Setting SORT_BY_NO=NO

This parameter is found in the .PRM file that belongs to the database engine referenced by a universe. In the old days the format of the .PRM file was the same as that found in a Windows .INI file. Today they use an XML structure instead. By default the SORT_BY_NO parameter is set to YES, so the line in the file looks like this:

<Parameter Name="SORT_BY_NO">YES</Parameter>

Clear as mud, yes? no? What it means is that yes, it is true; I cannot sort by something that does not appear in the select clause.

First I need to determine if my database allows me to sort by a column that does not appear in the select; Oracle and Teradata both do and I imagine others do as well. Changing this parameter won’t do me any good if the database does not support the technique. Next, I can find this file on my computer where I have Designer installed. The actual location will vary based on the installed path. The file name will be DBNAME.PRM, or in my case teradata.prm. I opened the file with a simple text editor, found the line shown above, and changed the value from YES to NO. It’s now a double-negative. It says, if I can paraphrase:

“It is NOT TRUE that I CANNOT sort by something NOT in the select”

or rather

“I is TRUE that I CAN sort by something NOT in the select”

Very clear, I am sure.

The Results

Before this change was made the “manage sorts” button on the query panel for editing LOV definitions in Designer was never available. After making this change, saving the file, and restarting Designer, I can now click this button.

When I click that button I get a list of objects from my universe. These objects do not have to appear in the select clause but can now appear in the sort clause. Problem solved.

But only after I set SORT_BY_NO equal to NO rather than YES in my parameter file.

By finally writing this down as a blog post, I hope that I will remember this the next time I get a new laptop and won’t have to spend time searching for the parameter setting that allows me to do this. I hope it helps someone else as well, but mainly this one is just for me. It happens that way sometimes.

• Step 1: Upgrade to Teradata version 13
• Step 2: Recognize that with this version a “distinct” query no longer returns sorted results
• Step 3: On the advice of Teradata, reconfigure your box with the “regression” parameter that makes distinct queries behave the way they did in 6.2
• Step 4: Send a Business Objects LOV query to the database that includes a DISTINCT keyword and a where clause with a couple of constant values
• Step 5: Watch the system reboot

That’s about what happened to us a few days ago. It wasn’t pretty. It took a long time to get our production box upgraded (and this after seeing development and Q/A roll through the upgrades with flying colors). Once the upgrade was finally completed, we had catch-up work as far as batch processing to do. Once that was complete the users got back into the system… only to see it sporadically reboot.

With a personal computer or laptop, a sporadic reboot is often a loose connection or faulty piece of hardware. We had not experienced anything like this on our database servers. Ultimately someone figured out that the following query was at fault:

select DISTINCT table.column FROM table WHERE table.column in ('A','B')

That’s a fairly innocuous query, isn’t it? At first someone thought the table was corrupt. Nope, it checks out fine. Next someone suggested that the data in the table was bad. Nope, I can query it just fine. Then we thought maybe the fact that there were some special characters in the where clause was the problem. Nope, they work fine too. Finally it was narrowed all the way down to the fact that we had a DISTINCT clause with the where clause and the regression parameter set on our database.

Once we had identified the specific cause, there were three factors to consider. First, we needed the where clause on the LOV in order to deliver the proper list of choices to the business. So that could not really change. The Teradata DBA team had set a parameter to make TD13 work like TD6.2 as far as providing a “distinct” and sorted LOV result. Could we work with that instead? If I removed the “distinct” from the LOV query then Teradata no longer rebooted. However, the results of the LOV were no longer sorted, and that presented a new challenge.

Universe Parameters To The Rescue

There are a couple of universe parameters that were used to solve this issue. Side note: it was back in 6.0 (I believe) that the ability to override universe parameter settings within the Designer application first appeared. Prior to that version any parameter changes were applied via a configuration file. Any changes made then affected every universe that used that parameter file (there was a different one for each database). Today I can change individual universes, which is good. So what changes did I make?

Universe Parameter: DISTINCT

The first change was to change the setting for the DISTINCT parameter. By default this parameter value is set to DISTINCT, which seems redundant. It essentially becomes DISTINCT=DISTINCT. However there is another option, that being GROUPBY. Note that there are no underscores or spaces in that phrase. With the setting updated to DISTINCT=GROUPBY my LOV queries no longer included the “distinct” keyword at the top of the query. Instead of this:

select DISTINCT table.column from table

I see this:

select table.column from table group by 1

There is another parameter that controls whether I see group by 1 or group by table.column and that depends on whether a database supports that syntax or not.

This change solved the first half of the issue. Without a “distinct” the LOV queries would no longer cause the server to reboot. This was certainly a positive step. However, making this change had an important side effect: the LOV results were unique because of the GROUP BY clause, but they were no longer sorted.

Universe Parameter: FORCE_SORTED_LOV

The second parameter that I changed was FORCE_SORTED_LOV. The default value is No and I changed it to Yes instead. The theory behind this parameter is simple: it would force the LOV results to be sorted. Before I changed the parameter my LOV query looked like this:

select table.column from table group by 1

After applying this change, my LOV query looked like this:

select table.column from table group by 1

Hm. Not what I expected. I was looking for an ORDER BY clause at the end of the query and I was not seeing one.

I went ahead and saved the universe and exported it to the Q/A system. We ran a few test queries (including one using the problem LOV definition mentioned above) and everything worked fine. Short LOV results were sorted. Long LOV result sets were also sorted, and they were still “paged” like I expected them to be. It still bothered me that I wasn’t seeing an ORDER BY clause in the LOV query definition, but obviously the sort was happening somewhere. We speculated that perhaps the Web Intelligence server was applying the sort which raised some concerns. LOV performance definitely suffered during the upgrade from 6.5 to XI and I didn’t want to introduce anything that would cause it to degrade further.

Fortunately one of the Teradata DBA team found the LOV queries in the logs, and despite the fact that no ORDER BY was showing up in the SQL when I viewed it via Designer, it was really there by the time Teradata got the query request.

Wrap Up

Ultimately the two universe parameters (DISTINCT and FORCE_SORTED_LOV) were used to change how LOV queries were generated, which allowed the Teradata DBA team to revert back to a standard version 13 installation (without the regression parameter turned on). I imagine that the Teradata engineering team is busy working on the bug and it will be fixed soon, so if you are using Teradata and looking at a version 13 upgrade at some point later this year you probably won’t encounter the same issue.

But it’s nice to know that we can help solve the issue by changing a few parameters on the universe side.

The ASUG board of directors announced that they intend to open a new seat dedicated to a Business Objects customer. This board position is intended to be filled by a customer that is not running any SAP enterprise products but has heavily invested in Business Objects technology. If you are a dedicated Business Objects customer and have a high-level executive that would be interested in sharing your viewpoint with the rest of the board and ensuring that your voice is heard then ASUG wants to hear from you.

I am no longer a member of the GBN steering committee. I didn’t get fired there just isn’t a need for that group anymore. However, ASUG has asked each of the steering committee members to stay on in a new role: I am now a Global Ambassador for Business Objects within ASUG. Or something like that. There were some details about the new ambassador program released at the planning meeting, but I expect that a more formal press release might be coming out at some point once the program has more details. In an interesting twist the global ambassadors (former steering committee members) have also been granted SAP Mentor status, which I already had. I certainly feel that’s a positive step because in my experience the Mentor program has remained primarily focused on the SAP enterprise products. The amount of content related to “classic” Business Objects products and services has been in the minority.

There were several activities going on in parallel during the summit. One group was focused on finalizing the selections for the spring ASUG conference which is co-located with SAPPHIRE in Orlando. Another group was working on plans for year-round content such as webinars. There were discussions about the influence councils. (I asked about the results from the Web Intelligence influence council from last year and was told it was essentially done and was being reviewed by the legal department. Hopefully we’ll see something soon.) I am assuming that a next step will be to request abstracts for the fall conference and repeat the planning process for that event.

I did a bit of work tonight on BOB to remove the GBN logo from the user profile screen. It has been replaced by an ASUG profile icon instead. I also swapped out the GBN logo from the sidebar of my blog and replaced it with the ASUG logo and a link to the organization home page. This will be my last blog post that gets the “GBN” tag as well.

Save the date.

I now imagine blog readers who have been working with the tools for a while have started nodding their heads in agreement, only to stop and say, “Hey, wait a second, is Dave talking about universe contexts or report calculation contexts?”

And the answer is, of course, yes.

Both concepts can be quite complicated and take a while to fully understand. Since I have already written a number of blog posts about universe contexts I thought it was time to turn my attention to the report technique of calculation contexts. Note that this concept is used in Desktop Intelligence and Web Intelligence but not in Crystal. This post is intended to be an overview only. I have plans for a detailed post on each of the various context operators where I will go into much more depth. For this post my goal is to introduce the concept and provide some basic definitions.

Introducing Calculation Context

The default calculation context is defined by the dimensions that participate in a report block. It is this feature that makes the dynamic nature of measures possible. When dimensions are removed from or added to a block the context changes and the measure amounts also change. This is controlled by the projection function set on each measure by the universe designer.

However, a report writer sometimes wants a measure to be fixed at a particular value. Percentages are a really simple example. To calculate a contribution percentage I need to divide each individual value by the overall grand total. In order to do that, I need to know how to generate the grand total anywhere I want it. In order to do that, I need to understand how to use the various calculation context operators.

Context Operators

There is a simple (but powerful) set of context operators that I can choose from. I intend to briefly define each one in this blog post and then dive into more detail in future posts. As I mentioned when I started, the default context for a measure is based on the dimension values present in the block structure. I can alter which of those dimensions impact the calculation using one of the following three context operators.

• In is used to specify exactly which dimensions to include in a context. Other dimensions in the block are ignored. Adding / removing elements from the block does not impact this calculation unless a removed dimension was specified in the context. In that case a #MULTIVALUE error is displayed.
• ForEach is used to make include a dimension in a context. The calculation context is still affected by other values in the block.
• ForAll is used to exclude a dimension from a context. Other dimensions of the block will still be considered. Adding or removing values from a block might change the value, but it will always ignore the ForAll items.

The operators listed above all require one or more dimensions as part of their syntax. However, calculation context can be specified by more than a set of dimension values. There are a series of report structure keywords that can be used as well.

• In Report sets the context at the report or “grand total” level. Any formula with these keywords for the context will return an overall total. Note that the total may still be affected by report filters.
• In Block sets the context for each block section. For years I have been waiting for this setting to be renamed to reflect how it really works, but it hasn’t. I suppose it’s for backwards compatibility. If there is only one block on a report then “In Block” and “In Report” are going to be the same. But when a block is broken up into sections, then this context will generate a total for each section value. For that reason I submit that a better name might have been “In Section” instead, but as long as I know what it does the specific word does not really matter.
• In Body is the standard default context for each row of data.

With Desktop Intelligence we have the option of In CurrentPage which can be quite useful and unfortunately not yet available in Web Intelligence. I probably don’t have to explain how the “CurrentPage” total is derived.

Finally, these keywords can appear in several places in a formula. The most common are the “input” and “output” context settings, but certain functions (like RunningSum() for example) also have a “reset” context.

• Input context is used to determine the input values for the formula.
• Output context is used to determine the output scope of the formula.
• Reset context is used to determine when a running function starts over at zero.

Dimensions referenced in the input context do not have to appear in the block. Output and reset dimensions must appear in the block in order to function correctly. For many calculations the default input context works just fine. I will show an example where it’s crucial to understand input context later on. For now I would like to move on to a simple example.

Simple Calculation Context Example

It’s hard to fully grasp some of the subtleties of calculation context without good examples. For this post I will provide one simple but common example that I mentioned earlier: contribution percent. I will start with a simple block with one dimension (resort), one measure (revenue), and a grand total as shown here.

This block already has a total on it. If this were a typical spreadsheet I could reference the total value using a row/column address, but that doesn’t work in Web Intelligence. That means I have to use another technique to generate the grand total. I mentioned the solution earlier… all I have to do is create a simple formula that looks like this:

Here’s the formula again:

=[Revenue] / [Revenue] In Report

Remember that the In Report context projects the measure up to the overall report total. By using that calculation context operator in the denominator of my division I ensure that I will always be dividing by the grand total in this formula. The results?

What’s Next?

This post barely scratched the surface of the calculation context question. It provided a few brief definitions and one simple example. I have several other posts planned that will dive into much greater detail about this often confusing subject. First I plan to talk more about In versus ForEach versus ForAll and how they are different. Next I plan to cover input versus output context and when each should be used. As I write this post neither of those two are much more than an outline for now, so I don’t have an estimated publish date.

But they’ll get published. Eventually.

I’m not done yet. I’m done for this year, but I’ll see you again shortly.

Best wishes to you and yours during this holiday season.

Regular blog readers might remember I had posted something about a “universe review service” that I was thinking of offering. I have not posted anything further about that service, mainly because I got a real job instead. But since this universe is a new one (at least it was new to me, I had not seen it used in training sessions or demonstrations before) and since it has a number of interesting mistakes, I decided to go ahead and talk about that process. I will use this universe as my subject and along the way talk about various things that I make sure to cover during a universe review. I got permission to take a copy of the universe from the session laptop and placed it on the memory stick that they had provided with all of the session presentations. (That was handy.) Unfortunately the connection was pointing to a Microsoft SQL Server database so I could hardly grab a copy of that.

Later on I asked around to see if anyone knew what the Foodmart universe was used for. One of the responses I got back included a link to another blog where they had a Microsoft Access database that matched the structure from the universe perfectly. After downloading (and testing) that file I now have a copy of the universe and the matching database. I can work with that.

After further investigation it seems that the Foodmart database is yet another demonstration database, much like the Northwind or Summit Sporting Goods or (dare I say it) Island Resorts Marketing. There should not be any license issues with distributing or using this database.

Since I expect that this universe will be new to most folks I decided to use it for this series of posts detailing the process I go through to review a universe. I am going to start the more detailed blog posts next year but figured that I would go ahead and post the universe and the associated database for folks to be able to download now. Download links are at the end of the post.

And Now, The Rules

I really hope that this series of posts will be useful. In order to provide the most effective use of these materials, I have a few ground rules.

• No spoilers in comments please
  I expect that many blog readers who download the universe and start looking for mistakes are going to find some. I found many in less than five minutes, and I expect no less from some of you. Please don’t spoil the process by posting comments about additional mistakes that I have not covered yet. I have a specific order that I want to cover things in and spoilers will, well, they’ll spoil the process.
• Please keep questions on topic
  This is a standard request that I make for all blog posts but I wanted to repeat it here.
• Don’t ask if you can send me your universe
  Unfortunately I cannot review your universe for you for free (or for a fee) right now. I simply do not have the time to perform this service at the moment.

That’s it. I hope this will prove to be a valuable series of posts for both new and experienced designers, and these few requests will help that happen.

You can follow all of the posts related to the Foodmart universe review using the “foodmart” cateogry tag.

Download Links

Here are the links to download the database and universe. This universe was created in non-secured mode which means anyone should be able to open it. You do not even have to have a CMS running in order to use this universe; you should be able to switch to Standalone (no CMS) mode on the Designer login screen and still be able to use this resource. The database is in Microsoft Access format and is over 20MB when unzipped.

• Microsoft Access Database (zipped – 8.37MB)
• Business Objects Foodmart Universe (zipped – 37KB)

I am not going to cover how to set up the database and connection. I don’t want to sound rude but if you don’t already know how to do these basic steps then the discussions that follow this one are going to be way over your head.

This will be my final blog post for 2009. Happy holidays, and I will see you next year.

In any universe design project I strive for the following goals:

• Deliver the correct result
  In my opinion, this is always the number one goal in any universe design.
• User friendly
  This is quite important but secondary to correctness
• Easy to maintain
  Universe maintenance is always allowed to suffer in order to provide the first two attributes on this list, but it is a worthwhile goal to strive for nonetheless

In this post I will show how all three of these goals are ultimately met by this implementation. When I am done I will have a completed universe. This post will cover slides 26 through 30 from my 2008 GBN Conference presentation. There is a link to download the file at the end of this post.

A Brief Recap

The overall goals of this project were listed in the prior posts in this series. The most important ones for this post are:

• Each report is expected to have multiple time-sliced measures
• The process of splitting each time-slice time period into its own SQL statement should be completely transparent

Current Year and Prior Year are different time slice attributes. Every report is expected to have month-to-date and year-to-date measures that cover both the current and the prior year. That means I expect to see four different measures used in each document. The month and year date ranges are obviously different, as are the current and prior year ranges. In the last post on this topic I described a special time-slice calendar table and set up different aliases for each required time-slice. I set up contexts to keep queries from hitting more than one time-slice at the same time.

That’s how we got here. Where to next? It’s time to build some measures.

Bits and Pieces

I subscribe to the thought that anything that makes the universe designer’s job easier while making the user’s life harder is the wrong approach. However, if I can do something to make my life easier without impacting the user, then whatever I come up with is fair game. That’s how I arrived at the strategy that I am going to outline next.

Normally a “bit” is either a one or a zero. Many of the bit objects used in this solution will be just that. However, I am also going to create some special measure bits that make use of the @Select() function. As a general rule I am not in favor of using this function, but in this case I will and I will justify why in a bit. (heh, pun )

I am going to focus on creating time-sliced objects for the revenue object. In order to do this, I am going to create a measure bit called “Revenue” that contains only the table / columns required to calculate this value. I am not going to include an aggregation function for this measure which violates another one of my universe design rules. Again, I will justify why. The other bit objects will be used to select (or mark) which context is used for each time slice.

Using Measure Bits

In my time-sliced solution everything is made up of combinations. Imagine a cartesian product between measures and time periods, something like this:

Each time-slice on the left is combined with each measure on the right. Rather than create six different references to the revenue formula shown here…

INVOICE_LINE.DAYS * INVOICE_LINE.NB_GUESTS * SERVICE.PRICE

…I am going to create just one. Notice that the code shown above does not have an aggregate function? This is by design… I am going to use the @Select() function to build my visible measure objects. Here’s the initial part of the object definition for the CY MTD Revenue object.

sum(@Select(Measure Bits\Revenue))

By leaving the aggregate function out of the source measure I can also create a minimum, maximum, or other aggregate version of the sales revenue. This is because the code referenced by the @Select() doesn’t have an aggregate… and why is this important? Because I can’t nest aggregate functions. If my source measure included the sum() function then that’s all I can do with it. By keeping it generic (no aggregate) I can reuse it with any aggregate function as needed. So that’s why I am breaking the rule that says every measure must have an aggregate function.

Ultimately the measure bits are going to be hidden from view so they will never be viewable / usable by report writers. They are only used as building blocks in the Designer application.

Time Slice Bits

Now I need to build some time-sliced bits. These are really simple. Here’s what the object definition looks like for the CY MTD bit object.

The select clause contains the number 1 and that’s it. Notice what the help text says? In big CAPITAL letters it says “DON’T FORGET THE TABLE!!!” I wonder what that means?

Here’s what it means. The 1 is just a placeholder to put something in the Select clause. The Where clause is empty. The From clause comes from the table list, and here’s what that looks like.

Stick with me just a little bit longer and this will all make sense. Remember that in the earlier posts I set up aliases for my time-slice calendar table. Each alias became part of its own context. A context is a series of joins that make up a particular path through a universe. What I am working towards is a set of time-sliced objects that a user can drag onto their query panel without having to worry about where they come from or how they’re put together. Here’s what my schema looks like as a reminder. All of the aliases are on the right side.

This structure gives me a way to automatically split out each time-sliced measure into its own path so that a user can combine CY MTD and PY MTD and any of the other options onto one query and run it.