Merge – Join

As I mentioned, the first thing I wanted to do after my initial exploration of my data was compare prior year to current year results. I could not do that (or figure out a way to do that) because of the way I uploaded my data. I had sent a data set that looked like this:

In Web Intelligence I would have created a crosstab report with the year on the top and the period down the left side and the cases (my measure) in the middle. Explorer does not seem to work that way. So what I did was split my file into two parts. The first part contained 2008 data and the second part contained my 2009 data. Instead of one file I had two files like this:

2008 file

2009 file

Once I uploaded the two files I started figuring out how to merge the data.

The menu option is quite easy to find. You can also right-click on a data set and select “Combine” from the context menu that pops up. However, there are some restrictions that are not so obvious. Once I selected my two sets I was presented with a screen showing the merged data. It turns out that the OnDemand site merges data sets by finding matching column names. Since both of my data sets came from the same source, that wasn’t a problem. I did, however, have to rename my measure column from “Quantity” to “2008 Quantity” and “2009 Quantity” in order to be able to tell the difference.

How did I know this was how my sets were going to be merged? There was a “more” link on the page. When I clicked it I got this information.

Any changes you make to the two source datasets are not saved back to those datasets; the changes are only propagated into the combined dataset.

The join occurs on columns that have the same name. The maximum size for a created dataset is 20,000 rows and 256 columns.

When you have finished joining the data, press “Next” to continue to the next step of the wizard where you can finalize how the new dataset will appear.

The changes mentioned in the first point related to the fact that I can change column names during the merge process. My source data sets are not changed, only the merged file contains the changes. I’m fine with that. The second point told me how the join was going to be performed, and also let me know that my combined set was going to be limited to 20,000 rows. So as I mentioned last time, there are limits to the free service but they do not appear to be documented anywhere so I could plan ahead. It turns out that my initial uploads were also silently truncated to 20,000 rows as well.

I went ahead with the merge. I was curious if they would tell me how many rows were being truncated, but they did not. After the merge was complete I had a data set with all of my facets (dimensions) and two measures. I was easily able to create a Variance variable on my data set by subtracting 2008 from 2009. Ultimately I was able to create a visualization showing the trend over periods including both 2008 and 2009 data, along with my variance. Here is what the merged data set looked like:

Merge – Union

Now that I knew how to get 2008 + 2009 data together I wanted to see if I could combine two business units. In this case I would not want new columns added as I did in the prior example, I wanted to see more rows. It turned out to be quite simple. I already had two years of data for one business unit uploaded and merged. I followed the same process for the second. Now I was ready to create a union between the two.

All I had to do was select the same “combine” option and this time make sure that all of the column names were the same, even my quantities (measures). Explorer appeared to merge the sets using a union operation, which was exactly what I wanted.

I guess now is a good time to point out that some of the features I am talking about are features of the OnDemand.com service, rather than the Explorer product itself. You or your company can opt to use the service online (via the web site) or you can license the entire system (Explorer plus the OnDemand portal interface) for internal use.

Need “Sticky” Settings

I am now able to generate year-over-year comparison charts from multiple different business units (data sources). However, there are still some (not so) sticky points to deal with.

The data I uploaded included the fiscal calendar period number. We use 13 fiscal periods so the numbers range (obviously) from 1 to 13. When I picked the period facet as part of my analysis, Explorer showed me the top 10 + a category for “other” instead of showing all 13 periods. I was eventually able to figure out that I could configure the system to show all 13 periods. I was also able to get them to sort chronologically rather than from “best” to “worst” order by quantity. It wasn’t hard.

It also wasn’t sticky.

What do I mean by that? As I played around with (explored) my data I swapped things in and out. When I removed period and then later brought it back it had forgotten all of the settings I had changed. I was back to 10 periods + other instead. I can see where this would be okay for a lot of things, but it would be nice to be able to make changes to a specific item and have them always come up that way by default.

Story So Far…

Normally I would call the final section of my post “Conclusion” but I am far from that point. While I have been a bit frustrated by the limits of the trial (free) service I can certainly see the potential of the OnDemand framework. Explorer is the core of the system, of course, but they have added a front-end interface that is extremely easy to work with. We have been trying to install Explorer internally to prototype some solutions, but have encountered a number of isssues with the installation. By using OnDemand.com I was able to – in a matter of a few days – upload some data and create some Explorer data sets and then build some visualizations for our review.

I wish they published the limitations for the trial membership somewhere. I definitely would have reduced the complexity of my data in order to avoid getting cut off at 20,000 rows. Fortunately the OnDemand site offers a Feedback link that takes you to a new site where you can make suggestions on how to improve the service as well as vote on existing ideas. I have entered this as a suggestion. I also entered the idea of making changes “sticky” as mentioned above.

I still want to get Explorer up and running internally. But OnDemand.com was certainly easy enough to work with, and it allowed me to get my demonstration up and running well within my time constraints.

I first looked all over the site to find out what the restrictions are. Since I was going to be using a free trial account, I didn’t expect to be able to upload my entire two billion row fact table, nor would I want to. However, I did want to stretch the system to show of what Explorer can do. I could not find anything, anywhere, that detailed the specific limits of the free account. I even tried to find out what pricing would be in the event that I decided to host more data, and I could not find that anywhere either. If anyone finds that information online, please post a link.

Other than these site issues, how did my initial exploration go?

Upload Limits

I created an extract with two years of data from one of my fact tables. I denormalized (flattened) the data to get it into a single CSV file, which I then attempted to upload to the OnDemand service. After the upload completed I found out what I was looking for before… it seems there is a 3MB limit on upload file sizes. So at least I was able to cross that question off of my list. It took me a while to decide what to drop out of my extract to get down to that limit, but finally I got a 2.9MB file that I was able to upload and save.

Of course the first thing I tried to do after that was open Explorer. The site gave me an error, so I logged out and logged back in. Still got the error. At this point I had to move on to something else so I tabled my efforts and came back the next day. That time Explorer launched just fine.

Initial Exploration

The data set I uploaded included one measure, two time facets (Year and Period), two geographical facets (Region and Area), two product facets (Brand and Line), and three location facets (Category, Channel, and Segment). Explorer uses the term “Facet” to describe a collection of values. I uploaded approximately 30,000 rows of data.

As I mentioned above, one of the selling points for Explorer is that there are no training classes for the product. It’s supposed to be point-click-insight easy. I failed the test. The first thing I wanted to do was compare current year to prior year results. I figured out how to make my graph show a trend from last year to current year, but only if I included the year on the graph. What I wanted to be able to do was show period or region or product comparison of sales from last year to this year. With the data I uploaded I was not able to do that. I am going to go back and separate out my files by year, so I have 2008 and 2009 data as separate uploads. My thinking is that by doing two uploads I will have two measures, and that way can compare the two years. I will report back on that after I get a chance to try that experiment.

Other Frustrations

One of the data points I uploaded was Period. We have 13 periods in our fiscal calendar. Explorer tried showing me the “top ten” periods sorted by my measure, and then grouped the remaining periods as “Other” with an aggregate total. This was not what I wanted. I figured out how to change the sort to chronological order, and I also figured out how to get Explorer to show me 13 instead of 10 values + Other. But as I continued to explore my data it went back to showing only ten periods. I could not figure out how to make my change “sticky” so that it would retain my settings.

Along the way I clicked the “Create Visualization” button a couple of times. Later on when I tried to remove my data set, I was presented with an error message that said my data set could not be deleted. I figured out that my data set was protected because of the visualizations that I had generated, so I switched to the proper tab and removed all of my visualizations. Problem solved? Not quite. I had to find the “filters” that were used to drive the visualizations, and remove them too. I could not figure out a way to remove them both at once, so I had to remove them one at a time. Again, there may be a way to clean things up easier but I didn’t find it right away.

Ultimately I was able to remove my data set. It would have been nice to be presented with a list of object dependencies and asked if I wanted to delete my data set and cascade the delete to all dependent objects. Instead I had to clean up manually.

Exploration Results

After my initial attempt I’m not ready to rush out and do a demonstration for any executives just yet. I need to learn more about the proper way to model data for Explorer (assuming that there is such a thing). I need to understand more about how I can configure / control certain aspects of the interface, such as what I outlined for Period above. I may have to even <gasp> try to find some documentation in order to find out how to implement some of the analysis I want to make.

Stay tuned for more details.