Spotfire – Column 2

TIBCO Spotfire 4.0

I had a briefing with TIBCO on their Spotfire 4.0 release, announced today and due to be released by the end of November. Spotfire is the analytics platform that TIBCO acquired a few years back, and provides end-user tools for dimensional analysis of data. This includes both visualization and mashups with other data sources, such as ERP systems.

In 4.0, they have focused on two main areas:

Analytic dashboards for monitoring and interactive drilldowns. This seems more like the traditional BI dashboards market, whereas Spotfire is known for its multidimensional visualizations, but I expect that business customers find that just a bit too unstructured at times.
Social collaboration around data analysis, both in terms of finding contributors and publishing results, by allowing Spotfire analysis to be embedded in SharePoint or shared with tibbr, and by including tibbr context in an analysis.

I did get a brief demo, starting with the dashboards. This starts out like a pretty standard dashboard, but does show some nice tools for the user to change the views, apparently including custom controls that can be created without development. The dynamic visualization is good, as you would expect if you have ever seen Spotfire in full flight: highlighting parts of one visualization object (graph or table) causes the corresponding bits in the other visualizations on the screen to be highlighted, for example.

There’s also some built-in collaboration: a chart on the Spotfire dashboard can be shared on tibbr, which has a static snapshot of the chart shared in a discussion thread but links back directly to the live visualization, [Insert obligatory iPad demo here] whereas sharing in SharePoint embeds the live visualization rather than a static shot. Embedding a tibbr discussion as context within an analysis is really less of an integration than just a side-by-side complementary viewing: you can have a tibbr discussion thread viewed on the same page as part of the analysis, although the tibbr thread is not itself being analyzed.

I found that the integration/collaboration was a bit lightweight, some of it no more than screen sharing (like the difference between a portal and a composite application). However, the push into the realm of more traditional dashboards will allow Spotfire to take on the more traditional BI vendors, particularly for data related to other TIBCO products, such as ActiveMatrix BPM.

[Update: All screenshots from briefing; for some reason, Flickr doesn’t want to show them as an embedded slideshow]

TUCON: BPM with Spotfire Analytics

Lars Bauerle and Brendan Gibson of TIBCO showed us how Spotfire analytics are being integrated with data from iProcess to identify process improvement. I hadn’t seen Spotfire in any detail before the demo that I saw on Tuesday, and it’s a very impressive visualization and analysis tool; today, they showed iProcess process runtime data copied and pasted from Excel into Spotfire, but it’s not clear that they’ve done a real integration between the iProcess process statistics and Spotfire. Regardless, once you get the data in there, it’s very easy to do aggregations on the fly then drill into the results, comparisons of portions of the data set, and filtering by any attributes. You can also define KPIs and create dashboard-style interfaces. Authoring and heavy-duty analysis are done using an installed desktop application with (I believe) a local in-memory engine, but light-weight analysis can be done using a zero-install web client and all analysis done on the server.

In addition to local data, it’s possible to link directly from enterprise databases into the Spotfire client, which effectively gives the Spotfire user the ability to do queries to bring data into the in-memory engine for visualization and analysis — in other words, there doesn’t appear to be any technical barriers to establishing a link to the statistics in an iProcess engine. They showed a model of data flowing from the iProcess server to a data mart, which would then be connected to Spotfire; realistically, you’re not going to let your analytics hit your production process engine directly, so this makes sense, although there can be latency issues with this model. It’s not clear if they provide any templates for doing this and for some standard process analytics.

They did a demo of some preconfigured analytics pages with process data, such as cases in progress and missed SLAs, showing what this could look like for a business manager or knowledge worker. Gibson did refer to "when you refresh the data from the database" which indicates that this is not real-time data, although it could be reasonably low latency depending on the link between iProcess and the data mart, and client refresh frequency.

Then, the demo gods reared their heads and Spotfire froze, and hosed IE with it. Obviously, someone forgot to do the animal sacrifice this morning…

They went to questions while rebooting, and we found out that it’s not possible to stream data in real-time to Spotfire (as I suspected from the earlier comments); it needs to load data from a data source into its own in-memory engine on a periodic basis. In other words, you’re not going to use this as a real-time monitoring dashboard, but as an advanced visualization and analytics tool.

Since this uses an in-memory engine for analytics, there are limitations based on the physical memory of the machine doing the processing, but Spotfire does some smart things in terms of caching to disk, and swapping levels of aggregation in and out as required. However, at some point you’re going to have to consider loading a subset of your process history data via a database view.

There was a question about data security, for example, if a person should only be able to drill down on their own region’s data; this is done in Spotfire by setting permissions on the queries underlying the analysis, including row-level security.

iProcess Analytics is being positioned as being for preconfigured reporting on your process data, whereas Spotfire is positioned for ad hoc analysis and integration with other data sets.

Spotfire could add huge value to iProcess data, but it appears that they don’t quite have the whole story put together yet; I’m looking forward to seeing how this progresses, some real world case studies when customers start to use it, and the reality of what you need to do to preprocess the ocean of process data before loading it into Spotfire for analysis.

TIBCO Analyst Summit: Spotfire

Christopher Ahlberg, founder of Spotfire and now president of TIBCO’s Spotfire division, discussed Spotfire’s capabilities and what’s been done with integrating Spotfire into other TIBCO products.

Timely insights — the right information at the right time — is a competitive differentiator for most businesses, and classic business intelligence just doesn’t cut it in many cases. Consumer applications like Google Finance are raising the bar for dynamic visualization techniques, although most of them are fairly inflexible when it comes to viewing or comparing specific data in which you’re interested. In other words, we want the data selection and aggregation capabilities of our enterprise systems, and the visualization capabilities of consumer web applications. Ahlberg sees a number of disruptive BI technologies transforming the platform — in-memory processes, interactive visualization, participatory architecture, mashups — and starting to be able to link to the event-driven world of classic TIBCO.

He did a demo of copying and pasting the contents of a spreadsheet directly into Spotfire, which automatically used the column headers as metadata and created a scatterplot. He filtered and colored the chart dynamically, set thresholds and played around with the data to show what could be extracted from it, then showed a pre-built dashboard of charts that still allowed quite a bit of interactivity in terms of filtering and other view parameters. He also showed a mashup between Spotfire and Microsoft Virtual Earth that allowed a subset of the data to be selected in Spotfire, causing a shortest route between the geographic location corresponding to the data points to be plotted on Virtual Earth.

This puts a much more configurable face on standard analytics, not just in display parameters but also in area like selecting the dimensions to be compared on the fly rather than having them pre-defined in OLAP cubes. Since TIBCO is focused on real-time event processing, the logical step is to see how those events can be visualized in Spotfire: instead of just raising an alert to someone, give them a view of the analytical context behind the alert that makes it easier to close the loop on problem resolution. They’ve packaged this as Spotfire Operations Analytics, which fits most closely into a LEAN Six Sigma manufacturing environment.

There’s a session on Thursday about BPM with analytics which I’ll likely attend to see what they’re doing in that area.