All of a sudden, a development that for too long appeared to be stalled by the chicken-and-egg problem of how website owners could be induced to tag their metadata looks to be in imminent danger of going seriously mainstream.

Marketers, it seems likely, rather than academics, will lead the charge to the VW campers from here on in. And in all probability, publishers and information providers who aren’t already waxing their boards in preparation for this particular wave of technologic change could risk being left behind as it steadily takes on tsunami proportions and thunders beachwards.

Google, Facebook, Twitter embrace semantic technologies

A recent article in New Scientist (subscription required) described how the giants of search and social media are making moves to actualize the semantic web.

1. Google’s recent acquisition of Metaweb’s Freebase, an open-source repository of structured data – or ‘entity graph’ as the company styles it – containing more than 12 million entities, will potentially enable much smarter searching. Entries in Freebase are tagged in such a way that machines can ‘understand’ what they are about and make meaningful connections between them. At the simplest level, computer searches would, for instance, be able to distinguish between David Mitchell the British Novelist and David Mitchell the British Actor, Comedian and Writer (not to mention David Mitchell the Tory politician, David Mitchell the retired American ice dancer, etc. etc.).
2. Twitter has recently released information about its new ‘annotations’ feature, which allows users to annotate a tweet with structured metadata. A tweet about a new book release, for example, might let you link straight to a ‘look inside’ book widget or the Amazon page for the paperback. Launch of a test version is apparently imminent.
3. Facebook is making changes to its Open Graph protocol that have a semantic element. Website owners can add a “like” button to their site, along with semantic tags that tell Facebook’s servers what the page is about. According to Facebook: ‘based on the structured data you provide via the Open Graph protocol, your pages show up richly across Facebook: in user profiles, within search results and in News Feed’. So when a Facebook user clicks the ‘like’ button on a publisher’s site – relating to a particular title, or author, perhaps – a link is established between that site and their Facebook profile.

Advertising goes semantic

Any change in the way Google works has major implications for marketers. If using an entity graph changes the way Google delivers its search results significantly, the dark art of Search Engine optimization will have to respond and weighty volumes of SEO best practice to be revised.

But even more wide-ranging changes will have to be made to practice around online marketing, with micro-writing and metadata tagging becoming ever more critical aspects of the marketer’s art, as websites lose their traffic to Google’s interface, which now not only provides a place for people to enter search terms, but also a place for them to read the answers, with no further click-through taking place.

New Scientist speculates, however, that it is in the Facebook and Twitter changes that the main attraction of these developments may lie for advertisers. With the major players in social media on board, apps are already beginning to be written that can exploit the potential of semantically tagged data.

And – oh dear – here comes another water-based metaphor: mainstream adoption is likely to open the floodgates for such third-party development. This is because it solves the chicken-and-egg incentive problem of how you get website owners to tag their content. There is a clear incentive for any content owner to tag their content appropriately, providing structured metadata, if it means targeted, relevant access to Facebook’s 500 million plus user base.

Why should you care about this?

The implications for publishers are obvious. The opportunity exists, through semantic technologies, to massively improve the discoverability of their content online. But they also present a threat. Those who move fastest stand to gain a march on their competitors, while those who lag could well miss out.

This throws down yet another gauntlet to a traditionally conservative industry that may well feel it already has quite a bit on its plate to deal with. Even more reason, then, for publishers to embrace the world of online in a concerted fashion, if they are to reap the benefits and stay ahead of the competition.

Surf’s up!

If you’re investigating the use of semantic technologies, talk to Semantico first. We offer a Semantic Web consultancy service focused on helping publishers improve the discoverability of their content using the evolving semantic web. Contact us today.

Search should be clairvoyant: like a magical librarian who somehow correctly guesses what it was you were looking for; offering it up within a fraction of a second, along with a wealth of additional filtering options and navigational possibilities.

Without wishing to destroy the magic, here are my seven steps to improving findability:

1. Define what relevancy means in this context

Relevancy is a difficult thing to pin down. A set of search results is more or less relevant on the basis of how well the information retrieved meets the need of the user. Arriving at a definition of relevancy therefore means doing some fairly detailed analysis of your users and content. Some BIG questions need answering.
Is recall or precision more important? How do you go about catering to the competing needs of different user groups?

Clearly, discussions need to be had and decisions made. During this process it will be worth considering a range of scenarios where you might like to boost certain results over others:

• Field weighting
  e.g. results within titles are more relevant
• Recency of data
  e.g. results from recent data are more relevant
• Search phrase density
  e.g. results which contain the most number of uses of the search phrase are the most relevant
• Search phrase term proximity
  e.g. results where multiple terms are nearer to each other are more relevant
• Records which have been bookmarked, cited or linked to
  e.g. results which have already proved themselves to be useful to other users are more relevant

2. Provide lots of options, not just lots of results

Findability is not just about returning relevant results. A good search implementation will also provide lots of additional further options to the user, which they can use to hone in on exactly what they were searching for. Examples of this include providing:

• Meaningful facets through which the results can be filtered
• ‘Did you mean…?’ option – for alternative spellings
• ‘Users who searched for x also searched y’ option – for related searches
• Clustering of search results, e.g. by topic
• Sort options
• Hit-highlighting – to highlight the phrase terms in context

3. Enriching the data

Data not only has to be marked up consistently and correctly, it can often benefit from some enhancement before it goes online. In practice, this can mean additional classification processing or entity extraction through text mining. The goal is to ensure the content itself is rich enough to support the sort of advanced searching and filtering that we want to build within the site.

4. Measure relevancy

It’s worth setting up some relevancy metrics to monitor how search is performing over time. A good method is Mean Reciprocal Rank. To implement this you track click-throughs to search results, giving each click-through to a first result a score of 1, each click-through to a second result a score of 1/2, each click-through to a third result a score of 1/3, and so on. Adding all these together will give you the ability to track an overall relevancy score, with a higher score meaning that top links are performing better.

It is also a good idea to monitor searches that return zero results. A monthly list should be reviewed in case there are some sensible search queries in there which will have resulted in user frustration.

Regular reviews of search analytics are a vital part of ensuring that search is still performing well as the site and its content change over time.

5. Improving the user’s query

Normalising the user’s search phrase (and indeed the search index data) can help to improve findability. The following are all ways in which you do this:

• Converting all letters to lower or upper case
• Removing punctuation, accent marks or diacritics
• Expanding abbreviations
• Removing stopwords or “too common” words

Recall might also be improved upon in certain scenarios by converting the users query into a fuzzy query (to return results for close matches to the search terms in order of how well they match). It may also be worth expanding the user’s search to include synonyms using a thesaurus (to return results where matches have been found for the same or similar concept).

In these ways it is possible to enhance the input query before it has even been sent to the search engine.

6. Tuning the site for third party findability

Lots of users (yes, perhaps even MOST) start their search using a third party search engine. It is therefore essential that the site is Search Engine Optimised, meaning lots of quality in-bound links, the use of semantic markup, micro formats and much, much more.

It may also be worth creating an Open Search API – so that third party use of the site’s search facility is possible.

7. Finding also means re-finding

There is a very good chance that users will want to re-use the entries that satisfy their information need. Consequently, improving fundability should also mean making it as easy as possible for users to re-find what they found before. Helping users in this way can be done with features such as:

• Bookmarks
• Saved searches
• Direct exporting to citation software

So there you have it: seven steps to findability. It is a BIG topic and I’m certain to have missed out important considerations. Please do feel free to publicly rub my nose in some of them by responding below!

The inaugural Semantico Symposium was held recently in London to discuss implications of the shift to mobile for publishers and information providers. An invited audience of publishing industry leaders debated the issues under Chatham House rules, covering the following three themes:

• Devices and technology
• Business models
• Future strategy options

This was a stimulating event with a high calibre guest list, delegates attending from organisations including Oxford University Press, Nature Publishing Group, Macmillan Education, Wiley-Blackwell, CrossRef, CABI, BSI Group and the Institute of Engineering and Technology. To do justice to the discussion, we’re going to report it over a couple of blog posts, starting with initial theme of devices and technology (yes, it’s a partwork!).

Forget devices, focus on the underlying technology

If proof were needed that these are nervous times for publishers, just consider the case of Flash. Not only does Apple not support Flash technology on the iPhone or iPad, but the world’s most popular video-sharing site, YouTube (owned by Google), is quietly in the process of moving away from Flash video. In addition the emerging HTML5 standard, which aims to reduce the need for such proprietary plug-ins, looks likely to make it all but obsolete.  So will Flash die? Almost certainly, say the tech-heads.

This is appalling news for publishers with large amounts of legacy online content in Flash. It also serves as an example of one of the strongest themes to emerge from our Symposium, which is that publishers and information providers who hope to thrive (or at the very least survive) in the turbulent times ahead would be well-advised to disregard, to a certain extent, the hype and wow surrounding high-profile device launches like that of the iPad, and focus on the underlying technology issues in cross-platform delivery. That’s where the real uncertainty lies. Marvellous though they are, it’s not about the devices – but about the content, and the user’s experience of the content.

There is no denying that the iPhone has instituted something of a paradigm shift in the delivery of content, but notwithstanding this undoubted fact, a good deal of skepticism was evinced by our delegates about what is perhaps the most significant innovation to be introduced along with that device, the App Store.

A significant strand of opinion believes that an app is really not that much different from a mobile-optimised website. As far as the user is concerned there is little difference. In the not-too-distant future, it was predicted, you will download something you think is an app but you will actually be interacting with a website optimised for mobile use.

The iPad experience of web surfing (about 42% of our small but select sample had had hands-on experience of the device) might make us question whether we need apps at all, in the opinion of one delegate. Maybe what we need is not apps but better-designed, more mobile-friendly websites.

So far, so heretical

However, there is another strand of opinion. From the user’s point of view, the experience of using an app is utterly different from that of using a PC. One virtue of the app is that it does a very narrow, specific thing. Apps streamline our use of the internet and cut out – or at least reduce – much of the pain associated with PCs (e.g. constant downloads of plug-ins, patches and updates, the state of total war we have to live in with viruses, spyware and spam, etc.).

A website is always going to feel like a place you go to, to harvest a crop of information. In the case of an app, the crop is turned into biofuel: information becomes the petrol that gets your knowledge car from A to B – to a designated destination. A website might be a field of dreams (if you’ll excuse a criminally over-used film reference), but an app helps you actually do something.

These two points of view are not, in essence, irreconcilable. It’s a matter of perspective; of whether you are looking at things from the producer’s end of things or from the consumer’s. If you strip away the wow, yes, an app is no more than a website. But what produces the ‘wow’ is fantastic usability – and that’s a matter of primary importance for most end-users.

Search lags on mobile

… Which is not to say at all that the current generation of mobile devices together embody a giant leap forward for usability. In actual fact they can look like a bad step backwards.

In particular, search took a while to get established on the desktop internet, and to reach its current state of utility. By comparison, search on mobile is very slow at the moment, even on 3G networks. Also, it is not that easy to find the app you want: the discoverability of apps is not great. This situation is liable to get worse before it gets better, as apps and app stores proliferate.

A certain frustration is surely excusable for those who soldiered through the difficult early years of the millennium when publishers were just beginning to build their first sites, and had to cope with the teething troubles of the early web – only to see many of the same problems coming back to them in 2010. There is a new network, and it has yet to organize itself effectively.

Monitoring the Big Tech face-offs

Focusing on underlying technology and networks throws a deal of emphasis on the importance of monitoring and understanding what is going on with some of the major tech companies – and not solely because a few (particularly Amazon and Google) have forged themselves into the publishing value chain, where they are fast becoming almost unavoidable links. We mentioned Flash earlier, owned by Adobe, but there are others to consider as well.

Apple’s new prominence, which has come about largely as a result of the huge success of the iPhone, is beginning to foreground some of the ways it has of going about things that most annoy people. The dead hand of control that it exercises over what can and cannot be offered through the App Store – amounting to censorship – has led to comparisons with China. Will Google’s Android prove to be a viable Open Source alternative?

Apple has become the company to attack, and the company to position against.

Microsoft appears to be positioning against Apple with Windows 7 by placing emphasis on social networking. This is an important battleground if it really can be established as a point of difference. RIM’s Blackberry Curve phone has crossed over into the teenage market not only because it is a lot cheaper than an iPhone, but because it offers their young audience a more effective way of interacting with their online social networks. It is too easy to write off Microsoft and believe that the important dust-up nowadays is between Apple and Google, but there may well be life in the old dog yet – and Microsoft still has significant market share in mobile operating systems.

Publishers likewise dare not forget, in the age of the read/write web, that online publishing is not just about how the stuff gets delivered, but also about how it gets produced, edited, commented, redacted, peer-reviewed … etc., etc. Nowhere is this more true than in the field of academic publishing – because one of the central concerns of academic publishing is scholarly communication.

What this becomes is a debate about how we consume and produce information. Corporate positioning takes on a philosophical, even ideological aspect, the nuances of which publishers have to tune their ears to detect. The first task is to be aware.

Government unhelpful

Someone who seems to have a bit of a tin ear in this regard is the great clunking fist himself – if Gordon Brown can be held responsible for the controversial Digital Economy Bill which, at time of writing, is awaiting Royal Assent. There wasn’t much controversy here: instead it was roundly condemned as a piece of rushed and unworkable legislation that will, nevertheless, no doubt remain on the statute books for many years to come.

The debate continues

Tune in next time for a further report from the Symposium, as we move to discuss business models.

More on this excellent seminar can be found at The International Association of Scientific Technical and Medical Publishers website.

Video by River Valley TV.

It’s always suspicious (to a jaundiced marketing person’s eye) when a show organiser chooses to place a large seated cafe area at the centre of the exhibition floor.

There were some noticeable absences at this year’s Online Information exhibition at Olympia – no doubt the result of crunch-inspired budget caution – and the air of an industry bracing itself for further shocks.

Public sector cuts have come to be seen as one of the few predictable features of a scarily unpredictable outlook for 2010, but the recent news from Dubai has shown that the private sector may not have exhausted its store of nasty surprises yet. Meanwhile, closer to home, there’s one less book retailer on the high street this Christmas with Borders going down – and as print news sales continue in a similarly downward direction, Rupert Murdoch has sounded the retreat, enjoining other Publishers to follow his lead and dig in behind their paywalls.

Small wonder, perhaps, that the on-stand drinks parties spilling out into the aisles had plenty of space this year to spill out into.

Quality not quantity

When a show is in such a period of contraction it’s usually the organisers who like to bandy the phrase ‘quality not quantity’ about, but interestingly it was a Publisher we heard using it this year. A show like this is about conversations, and OK there were a few less conversations happening this year than last, but perhaps they really were more valuable ones. Certainly, there seemed a serious progression from last year in many quarters. Tech industries in boom are notoriously productive of hot air. Sometimes it takes a chillier climate to bring a greater air of reality.

Answers not results

The conference at Online Information is always an interesting one. We at Semantico were too busy with those valuable conversations, perhaps, to catch all the sessions worth seeing, but what we did see confirmed our view that, more than ever, Search is a critical issue for our industry.

Two extremes, perhaps, of the current landscape were visible here. The leading edge of semantic search was represented by Conrad Wolfram, who launched Wolfram Alpha earlier this year. Like all good things it is now available as an iPhone app. Wolfram’s assertion is that users don’t look for search results, but for answers. Being on the leading edge of search he of course denies that what he is doing is search: preferring to say that he is creating ‘knowledge environments’. One thing that really came across as distinctive in his approach is the emphasis on the presentational aspect of data.

At the other extreme, arguably, and at some remove from the leading edge, lies Bing. It is hard to gainsay the notion that Microsoft is playing catch-up with Google here, as in many of its recent offerings, and despite what it says in the marketing books about ‘fast follower’ being the more advantageous positioning in tech markets, surely this can’t seem a great place to be when you used to be … well, Microsoft. It’s hard to think that any sliver of useful innovation Microsoft manages to come up, any incremental enhancements to where we are now, won’t instantly be snapped up, and probably improved upon, by Google.

Which is not to say that Google doesn’t have its own catching up to do elsewhere …

Get real

There’s a lot of talk about real-time search at the moment, some of which was being purveyed at the conference by search guru Stephen Arnold. This interest largely results from the stellar rise of Twitter, and Google’s stated ambition to improve its game in this area. But real-time search is hard. At the moment, Google indexes certain sites hourly; however, there are many others that it indexes only three times per year. Keeping up with the global information stream, and prioritising exactly what should be kept up with, is an extremely challenging computational task.

Keeping up with the local information stream can be a challenging task too, even at a slightly downsized event such as this; so these notes are necessarily impressionistic. Over all, our impression is of a tougher, but perhaps slightly more real, marketplace for online publishing. Which bodes well for the upturn when it eventually comes.

We’d also like to thank all those who thronged our stand on Wednesday afternoon (and spilled out into the aisles) to help celebrate Semantico’s tenth birthday. Here’s to the next ten!

1. Site Copy

Copy has a direct impact on search engine optimisation as it provides the crawlers with meaningful semantic information about the site. The book content of any publishing site will contain excellent copy but marketing and home page material should be carefully considered as this is often where crawlers, users and off-site links find themselves first. The number of keywords per site should be limited to three or four and these should be added to the content in such a way that the content remains natural and readable.

Particular attention should be paid to drafting text for hyperlinks. This is because the words that point to a particular page are heavily weighted by Google when determining keywords for that particular page. Never use phrases like “Click here for our contact form” (for example); instead use words that describe the page you’re linking to by thinking of what a user would search for if they were looking for that page. In our example this might be “Contact us now using our contact form“.

2. Incoming Links

Google rates a pages relevance using an algorithm called PageRank which is calculated based on the number and relevancy of links pointing to the site. The concept is that each link represents a vote of confidence for your site. But the votes are not equal, links from pages with a high PageRank are worth more. The PageRank of any site is shown on the Google Toolbar, which you can download from Google. Wikipedia have a good explanation of the PageRank alogorithm.

3. Google Webmaster tools

Google Webmaster Tools offers a comprehensive selection of tools for configuring a site for improved search results on Google. The main offering is the ability to submit an XML sitemap that improves the crawl across the site making it easier to control what pages are being indexed. The use of XML sitemaps will improve discoverability of the content for any site. In addition to this, is the ability to see when crawls have taken place and when they are next scheduled. This is of particular importance on each content update to ensure that the most recent updates have been picked up. Google Webmaster Tools also provides:

• A top search queries feature, where information about user search queries that have returned pages from your site are shown.
• An interface for quickly finding all links to a site, allowing incoming links to be tracked.
• Keywords and their frequency within the site.
• Internal links.

4. Google Analytics

Google Analytics offers further integration with Google search as the script that enables the tracking of the site provides constant activity feedback back to Google. Google Analytics should be used to determine where users are coming from and where users are landing. This information in itself will not help search ranking and discoverability, but it will provide metrics with which to measure the progress of any search engine optimisation activity. There are other analytics packages, such as Yahoo! Web Analytics (which provides real-time statistics), but Google Analytics has been recommended as it is the leading search provider and integrates well with other Google tools.

4. URL structure

Have a look at many complex publishing sites and you might notice that there are no PageRanks on many of the deeper linked pages within the sites. This is most likely due to Google’s crawlers not reading complex page URLs as these pages are often less meaningfully marked up and are the result of a a query. For more details you can read Google’s explanation of this behaviour.

Many sites can benefit from an overhaul of the URL structure to provide more meaningful and less complex URL structure. Care needs to be be taken in making sure that the original links are still operable and their appearance in search engine results will diminish over time.

5. Error pages

If the user requests an unknown page, return a friendly error message and use the correct HTTP status code. This might sound obvious, but it’s a surprisingly common mistake on publisher platforms. If you don’t return the correct error code (for example “404 Not Found”) then Google will cache dead or bad links on your site, leading to less than optimal search experience for your users. Similarly, link resolution services such as CrossRef can only quality check your DOI metadata if the HTTP codes are used correctly.

Conclusion

In conclusion, a little attention to the copy and structure of your site can pay dividends in terms of search engine rankings and discoverabilty.

Google have recently added support for a very limited subset of the RDFa vocabulary to their search engine [beginners should start with RDFa Basics]. This will allow them to present much more relevant information directly on the search engine results page (SERP), which in turn will allow users to discover the information they are looking for much more easily. It will also allow Google to build on their “show options” feature, which allows users to narrow down search results by selecting from a relatively limited number of different categories.

Tech-heads will be less than happy with the part-baked nature of Google’s support for the RDFa standard, and rightly so in my view. What it means in practice is that the exact tagging needed to make the new Google features work is currently determined by Google alone, and does not build on semantic web community standards already established, such as FOAF or Dublin Core.

Those publishers who currently enable Google to crawl their sites in order to drive user traffic should take note of this development. Since the main benefit will be to allow the end user more options in navigating search results, and the ability to find relevant results more quickly, those who adopt the new standards should stand to gain user traffic over those who don’t. However one corollary of allowing Google to provide more relevant information directly in the SERP is that, for some searches at least, the relevant information will be embedded directly into the SERP and the user need go no further. Clearly a win for the user (and Google), but a loss for the publisher, whose site will inevitably get less traffic.

Directory, data and search publishers have most to loose from this new state of affairs. Because Google stores all of the information it indexes, the key issue is that those who choose to expose RDFa data also choose to let Google (and the world at large) mine their information for later use. This will amount, for many publishers, to a huge giveaway of factual content. If you code your factual information using RDFa then the whole world can take it away and do with it what it chooses. At the basic level, this will include telephone numbers, contact names, organisational relationships, etc. but will also extend to areas of specialist data such as protein structures, DNA … the implications are potentially huge.

The rub is that if you don’t publish at this level of granularity, users may choose the resources that do over yours, so staying out of the game is not a serious option. On the other side, the threat if you do jump in with both feet is that someone else may take away your data and use it in ways that will be completely out of your control. You’re liable to feel that the tight, tactical game of Premier League football you’ve been playing up to now has suddenly regressed to the state of a village knockabout circa 1300, with anyone out of the crowd able to rush in and steal away with the ball.

Neither are there just two teams anymore. This particular match looks to have had a three way result: Users 1, Google 1, Publishers 0.

Search Wizards Speak

I recently had the pleasure to be interviewed by one of the great minds in search, Stephen Arnold, as part of his ‘Search Wizards Speak‘ series. Stephen is a straight talking guy, and it was great to have a chance to spend some time with him discussing where search will go over the next couple of years.

Go direct to Stephen’s site for the full text of the interview.