Month: February 2008

Timo Teimonen of StarCut, a mobile media publisher, was up to discuss how the mobile internet is transforming. The incredible growth in mobile internet access over the past few years means that people are no longer happy with seeing full-format web pages on their small screen, but want content tailored to the smaller form factor. A huge portion of the market, mostly in Asia, experience the internet only on their mobile, and almost never on a PC except perhaps in a workplace environment. This creates challenges not just for content presentation, but also for search and monetization, with the added issues of localization and contextualization: creating search results and ad content, for example, that’s based on the user’s current location.

The heart of mobile value is based on contextuality: mobile users are willing to trade a bit of their location and preferences privacy in order to get more targeted product and service information. The mobile phone is the bit of information access that people carry with them anywhere, providing the perfect platform for extremely local advertising based on relationships and information about the user. Traditional advertising is extremely inefficient; mobile advertising can be much more efficient at delivering relevant content to people based on where they are and what they’re looking for. In turn, mobile content providers have an incredible amount of information about how people user their content: the Zagat mobile site, which is based on StarCut’s platform, can gather information about what type of restaurant that a customer searches for, be able to provide recommendations based on location and preferences, and even allow for online reservations through the mobile device. In this way, mobile allows advertising and other content to be directly connected to physical retail: a true bridging from the virtual to physical worlds.

I can definitely relate to this, since I live within close proximity of my Blackberry at all time: a device that I use primarily for data access (email, web access, and specialized services such as Google Maps) rather than voice. Mobile content is a big part of my life, and the lives of many of the people with whom I interact, and I believe the quality of this content will greatly improve as localization and contextualization improves.

Gerry Campbell, president of search and content technologies at Reuters, spoke about how Web 2.0 has set information free from many of the content silos where it used to be entrapped, but there’s a major information overload problem since consumers need related and relevant information presented together in order to provide context. Users want more relevant information, faster; causal relationships between the information and more sophisticated patterns, and the ability to make comparisons.

People don’t just want the “what”, they want the “why”: where relationships are understood in order to provide insight, not just data. The missing element is context: the layer of meaning created through understanding the relationships of facts and entities, including people, companies, places and events. This layer of contextual intelligence vastly improves the quality of search, and hence information accessibility.

To address this, Reuters is creating a transformational platform. Starting with content and metadata as input — Reuters content, partnered content and the entire web — the information is then processed to understand the content, the queries that will be made against that content, and the needs of the users. The output is not just basic content, but insight and knowledge. ClearForest, a recent acquisition of theirs, will do a lot of that processing to automatically extract and tag content. Furthermore, Reuters is using this to create Calais, an open and free web service to discover entities, events and relations in text and automatically generate metadata, transforming basic text to semantic web content.

Sue Feldman, VP of search and digital marketplace technologies at IDC, spoke about taking the pulse of the digital marketplace. She started by defining the digital marketplace as an online gathering place of buyers and sellers, with particular relationships and dynamics between the types of participants. Users interact with 3 types of sites: gateways (a starting point or index to the web, funded by ad revenue; these are portals and search engines such as Yahoo and Google), hubs (aggregators with high-value tools and trusted content, funded by high-value ad networks because of their trusted position as selectors of content, such as eBay or Amazon), and nodes (communities with few outward connections, such as direct consumer-facing corporate websites).

They’ve created a model of the dynamics of feedback in the digital economy: why people visit certain sites, and why they move from one site to another. Along the way, they found a few surprises:

• The web is fragmenting, not consolidating
• More queries are happening on the web than are reported
• Only a third of queries go through web search engines (and hence are monetized); the rest are directly on destination and speciality sites
• More queries go to Baidu than to Google, Yahoo and Microsoft combined, but all are in Chinese (I have no idea why IDC would find this surprising, except she does admit that prior to hiring an analyst covering China, they had an “embarrassingly North American focus” — what I take to actually mean a US focus)

Looking at the trends that are emerging, they see increasing user sophistication, with users participating in social applications and expecting personalized and localized information. There’s a greater demand for selection on trust, as information overload drives users to speciality sites that pre-select information targeted at their needs. They’re seeing a rise in ad networks that provide context-specific and relevant ads, and a rise in social networks that filter the overwhelming amount of content through a trusted network of friends.

There’s some powerful new technology and infrastructure that’s underlying these trends in the digital marketplace, from personalization and recommendation engines  to user data mining tools. To conduct business on the web, you need to exploit this infrastructure, as well as understanding the long tail effects that impact what you offer to your customers. For traditional businesses, the web is an additional channel, not a replacement for other media.

The dynamics of the digital marketplace are changing quickly, but the roles are becoming defined between the gateways, aggregators and nodes. The central role of the user cannot be overestimated, both on an individual level and as a group. Audiences are attracted not just by useful content, but also by compelling tools and usefulness. Selection and trust are key components for the aggregators and nodes, and targeted communities are replacing generic sites.

Safa Rashtchy addressed us in this morning’s first session about the original six trends and impact of the user revolution, from a report that he created while with Piper Jaffray.

1. The emergence of “communitainment”: community + communication + entertainment have collided together and are impacting each other’s growth, generating a new type of activity on sites like MySpace. Advertisers must learn to leverage the community aspect of communitainment.
2. The increasing popularity of Usites: sites where user-generated content makes up most of the site, such as YouTube. Time spent on Usites has increased from 3% to 31% of total time spent online, since it’s not just about the content itself, but also about the community that emerges around the content.
3. Mainstreaming of the internet: it’s becoming a mainstream media channel that people are using as part of their daily routine, and 40% of respondents in a recent survey would actually give up TV and keep the internet, given the choice. It has surpassed print media and radio in terms of reach, and is approaching TV. It’s also a key medium in the workplace, further expanding its reach.
4. Declining usage of traditional media: in particular, TV viewing and TV advertising are both dropping.
5. Fragmentation of content consumption across several media: we’ve moved from having only a few choices in how we consume content (e.g., newspapers, broadcast TV, magazines, broadcast radio) to more than 30 different ways, many of them facilitated by the internet. Furthermore, consumers are multi-tasking, merely “snacking” on any particular medium by doing things such as surfing the internet while watching TV.
6. Evolution of user generated brands, especially when well-known corporate brands allow their consumers to impact the corporate brand: consider the Dorito’s Superbowl ad example, where they allowed consumers to design the ad that was aired during the most expensive time available on TV.

Search is the second most commonly used application on the web, and search is becoming the new portal: it’s how people interact with information on the web. Google is increasing their dominance in web search, but local and enterprise search are still wide open. Traditional media is changing their view of search as well: in 2005, Agence France-Presse tried to sue Google because their headlines appeared in Google searches, then by 2007, the Times of London was training its journalists on how to write in a way that would maximize the probability of a high ranking in a Google search.

He offered up nine trends and how they appear before and after the user revolution

• Online versus offline media are changing from being separate and competitive to being integrated into one medium
• Media sources are moving from a few large content providers to vertically-focused multiple sources
• Content control used to be centralized and controlled by publishers, but is becoming fragmented and controlled by the users
• Internet content is changing focus from text-based to video-based; Rashtchy feels that video will be the next killer app on the web
• Main navigation method is changing from portals to search
• Consumer decision process used to be basic and largely influenced by advertising, but is now sophisticated and driven by reviews and rating
• Competitive advantage was based on exclusive content but is now based on simplicity, speed and interconnectivity
• Media consumption patterns
• Social networking

There are four key impacts of the user revolution on enterprises:

1. Enterprise search is not a tool: it’s the platform that will hold your business
2. Consumers expect simplicity, common sense and accuracy; if it’s not easy to use, they won’t use it
3. If you have to use the search constantly, the website is in very bad shape
4. A good search platform must be built on a great website design, created for simplicity, common sense and accuracy

He covered a huge amount of information, and I couldn’t absorb it all; I’d love to hear his talk again.

My regular live-blogging posts from the FASTforward conference are cross-posted here as well as the FASTforward blog, but check out the FASTforward blog for a ton of other content: other bloggers at the show, and some great video interviews by Jerry Michalski with the conference speakers and some of us bloggers. It’s the next best thing to being here!

Last breakout session of the day, and it was a tough decision: there was one on business intelligence built on search, but I opted for the enterprise mashups session with Arnt Schoning and Espen Sommerfelt of FAST, plus Stefan Sveen from Comperio, one of their partners that builds search solutions on top of the FAST platform.

They started with some background on service-oriented architecture (SOA) and mashups: SOA as a way to expose business functionality from siloed applications for reusability, and mashups as the Web 2.0 equivalent of SOA, using lighter-weight integration protocols such as REST and JSON instead of WS-*. At last year’s FASTforward, they created a business process in BPEL that consumed a couple of FAST services; this year, they wanted to show a more comprehensive mashup using their technology. There are about 20 services available out of the box with FAST, accessible using Java or SOAP/WSDL.

This turned into a fairly deep dive into the mechanics of how to implement mashups with their services, complete with Java snippets; a higher-level view of the importance of this would have been helpful as at least part of the presentation.

Interestingly, they put what they did last year in the context of purely automated integration-centric processes (hence their mischaracterization of their process as SOA rather than integration-centric BPM), whereas I can imagine that there are also uses of their services as called from processes that include human interaction.

They continued on to talk about mashups: visual versus non-visual, client-side versus server-side, assembly versus coding styles of implementation, and consumer versus enterprise usage. Mashups are ad hoc and situational, hence are primarily user-driven and used to empower users and knowledge workers on the long tail of application development: giving them the ability to create applications that IT would never get around to implementing because the audience is too small or because there’s only a temporary need.

They also talked about the difference between SOA (composite applications) and mashups; they distinguish mashups as being user-created and SOA as being IT-created, but I think that they’re off the mark here: SOA is really just the services layer, which can be consumed by a user-created application just as easily as an IT-created application. The true comparison would be between a composite application development environment and a mashup creation environment, and I think that the line between those is becoming increasingly indistinct.

They showed a demo of a mashup created using their enterprise search services, in which the user can enter any text, and any location information in the free form text is detected and mapped on a Google map. Since they used the location extraction service in their engine to parse the text, words similar to locations are not detected as locations as long as there is sufficient context to make that distinction (e.g., the phrase “Denzel Washington is an actor” did not result in Washington being mapped). Once the points are mapped, other information can be gathered related to those locations, such as weather data or Flickr photos at those locations; this is not really specific to a mashup using FAST services.

A second demo showed the creation of a simple blog with automatic tagging: their automatic tagging service generated the most appropriate tags based on the text of a blog entry. They also showed applying the automatic tagging using a large chunk of text from a real FASTforward blog post; for the second time in a year, I see a recent blog entry of mine being used in a demo during a conference presentation — a slightly odd experience, although they likely had no idea that I was in the audience.

Sveen from Comperio then discussed their best practices in developing search applications: they’ve created a lightweight framework, Comperio Front, that repackages FAST search functionality for easier reuse. The framework is service oriented, and models the search logic workflow and rules as well as encapsulating a number of reusable search tasks and simplifying some of the FAST APIs. It appears to be well-layered, such that (for example) additional adapters could be easily added to the core processing engine to augment the existing web services, HTTP and XML interfaces.

He showed a demo of how they could easily improve the user interface on a standard search portal by combining multiple navigation widgets, and a further demo that shows how search data can be dynamically filtered and displayed. Since I don’t know what the same functionality would look like directly in FAST, I can’t really judge how much easier it is using Comperio Front, although they have the definite benefit of exposing additional search functionality as services for consumption by composite applications.

I was delayed getting to the next session on using search as the next generation of enterprise portal by Kara Hansen of Disney, so missed a bit at the beginning. She described how they use FAST to crawl content across multiple enterprise sites to create a virtual view of their content, both enterprise wide and for specific subsites representing divisions within the enterprise. In total, they serve 55,000 employees, plus customers via an externally-facing portal.

One of their most significant challenges was security. First of all, the search crawler needs to have credentials for all of the content locations that it’s going to crawl so as to expose only information that should be exposed on the portal; although these are internal portals, there’s still security issues around content viewing. There’s also an externally-facing portal based on the same technology platform, limited by the specific collections and search profiles to generate a restricted view of the content. When any user executes a search, their security level is combined into the search behind the scenes to deliver only the information to which they should have access. This, in turn, results in different results depending on whether a user is logged in or not, which can frustrate users who don’t understand the implications that their user authentication has on their search breadth.

They also have a challenge with users finding applications that are typically exposed as portlets, but aren’t searchable so don’t end up in the search-based portal. In other words, the user is trying to reach a specific destination on the intranet, not information. To compensate for this, they created a simple XML document that links to the applications and allowed that document to be indexed, although this seems like a bit of a brittle approach since it presumably requires manual updating of that document to add, remove or modify links to applications. However, sometimes a manual workaround can bridge a major gap in functionality; queries for applications now forms a major portion of their executed searches.

They’ve seen a shift in communication culture within the organization: users proactively search for information that they want rather than wait for it to be fed to them. Search also acts as way to introduce user-driven applications, such as wikis and user-created profile pages, a change that Hansen refers to as “profoundly disruptive”, in a good way.

They did all of this with a team of five people — three Disney and two from FAST — although that didn’t include the content providers who manage the content that populates each divisional portal.

She ended her presentation by giving away some nice Disney swag: proving that Mickey is for adults too with some MM-themed bar accessories.

The breakout sessions have started, and I’m sitting in on the financial services track for a couple of sessions, since my primary customer base is in financial services. Adam Sussman at TABB Group is discussing how search technology will change the relationship dynamics of the institutional brokerage industry, where there is a much different relationship between brokers and clients because of the size of the institutional investment portfolios.

The current problem is that even large customers are devaluing the service provided by high-touch (full-service) traders, choosing to make their own trades directly and purchasing research on an ad hoc basis. However, the type of research provided by high-touch firms is really driven by search, providing information about movement in companies that are held within a portfolio, as well as other market conditions that might influence future trades.

More importantly, however, and the focus of Sussman’s talk, is that by implementing sophisticated search technology, the sell side will be able to recognize opportunities to revive high-touch revenues. In other words, use search technology to identify which customers are most likely to be profitable, and focus efforts on those customers.

You can read a bit more about Sussman’s research in this area on their site.

It’s the first session after lunch, and the first time that we’ve heard an actual FAST presentation, by Bjørn Olstad, their CTO, discussing next-generation innovation in search technology.

They have segmented their audience into two main groups:

• Monetization, which is driving revenue by creating unique user experiences that match customers to relevant assets
• Enterprise, which is driving productivity by creating unique user experiences unlocking actionable information across data silos

Unlike traditional content management systems, search focuses on the intentions of the consumers of the content, not the creators of the content. The search experience should constantly be evolving based on what the consumers are searching for and what they’re doing with the results, plus the constantly changing data underlying the search.

He looked at search as a research portal within an enterprise — which is also covered in a breakout session later today — and other uses that don’t fall into the “search box” view of search. Search is becoming the portal for user interaction, mapping the user intent onto the available content.

We then saw a demo of the new Content Integration Studio, which allows for an easy mapping of reader (data sources) to writer (consumption mechanism): a bit like using Yahoo Pipes to create a graphical mapping from inputs to outputs with various filters and transformations along the way, but using corporate databases and a wide variety of internal and external data sources. You can debug the flow on the fly, setting breakpoints then examining and/or changing data at the breakpoints to see how information will flow through.

They also have some new indexing and contextual matching algorithms in a new search core that allow for near-real-time availability of new content, which is essential for their media/news clients. They’re also providing tools for creating search-based portals to allow search to be the fundamental driver of the user interaction, and provide very customized experiences so that individual users will be presented with different content in their view of the portal.

The morning finished with Jared Spataro, a Microsoft product manager for SharePoint, responsible for enterprise search: namely, the group in which FAST will belong after the acquisition completes. He talked about why Microsoft is making this acquisition, how FAST will benefit, and what this means for customers.

Microsoft started to look at enterprise search about 18 months ago when they realized that their customers no longer saw search as just a feature, but that it was actually shaping the user experience. They found a huge gap in the market between high-end and low-end enterprise search segments, and created a mid-range search capability combined within the SharePoint platform. They also attacked the low end of the market by introducing Search Server Express, a free downloadable product. But that left the high end of the market, which Microsoft is approaching through the acquisition of FAST. So the answer to the question “what’s in it for Microsoft?”: total search world domination.

So what’s in it for FAST, besides $1.2B? It gives them scalability both in their ability to innovate and in the massive sales and marketing channel that will expand their product reach. Microsoft has recently bought in to the idea of distributed development, both to find and retain talent and because of non-US acquisitions, meaning that FAST’s development team can stay in Oslo, far from Redmond and hence a bit autonomous.

At the high end, of course, FAST runs on non-Microsoft platforms, and Microsoft is committed to continuing support and innovation across all platforms, a stance that should somewhat calm the fears of existing FAST customers. Spataro also talked about how it’s an advantage to customers that they offer complementary capabilities in enterprise search across the low, medium and high tiers; personally, I’ve never seen a big advantage to having multiple products in the same product class but based on different technologies available from the same vendor, since there will be confusion at the boundaries and some uncomfortable migrations.

He summed up with the overall SharePoint strategy, based on the pillars of business intelligence, collaboration, portal, search, content management and business processes. It’s not their intention to bury FAST inside SharePoint, although it will likely end up as part of the platform at least for branding purposes.