SkillPages Engineering Introduce “The SkillGraph”

Editor’s note:  This is an update from the SkillPages Engineering Team, just to warn you it’s a pretty detailed technical insight into the technology we’re developing at SkillPages!

When you look for somebody to fix your sink you probably figure you need a plumber.  A handyman might do it, or maybe a janitor, even a drain cleaner could do the job.  Your brain does something amazing here: in split seconds it generates a cloud of many different skill names and professions, guiding you to the best match for the task.  Knowing what you mean and having an idea of how various skills are related is what makes the skill ‘search & matching engine’ in your head extremely powerful.

Moving Beyond Simplistic Mechanical Matching

People’s skills and expertise often do not fall into rigid categories.  People use different terms to describe their skills and these skill terms mean different things to different people (think painter of paintings and painter of houses).  This causes a problem.  Traditional online search uses keyword matching to connect a search term to search results.  But this doesn’t work effectively with skill search (a search for a “carpenter” will never return a “woodworker”).  To create a truly powerful, effective and useful online skill search utility, we need to move far beyond simplistic mechanical matching of keywords.  To capture all the variations and subtleties of skill search, we need something very different.  That is why the SkillPages Engineering team and UCD Clarity Research Centre have developed a unique technology called SkillGraph.

SkillGraph is at the very core of the SkillPages platform, underpinning everything we do.  Essentially it’s a market matching skill relevance engine that connects people looking for skills to people who have the relevant skills in a more accurate way than was previously ever possible.

How SkillGraph Works

Rather than rely on keywords, SkillGraph captures the meaning of skill terms used by members on SkillPages.  For example when an opportunity is created by a user on SkillPages, it is matched to people with relevant skills through SkillGraph.  This sees the terminology used in the opportunity “inflated” into a cloud of relevant terms.  This cloud is then instantaneously matched against equivalent clouds of words pertaining to each and every skill that members have added.  A relevancy threshold is used to identify only the most relevant skills and then those users are sent a notification about the opportunity relevant to their skills available for them on SkillPages.  Simplistically, this has the effect of connecting someone who is looking for a “pencil drawing artist” to an “illustrator”, a “roofer” to a “builder” and so on.

The SkillGraph algorithms are continuously evolved based on research and continuous behavioral and data analysis of our rapidly growing user base.  The underlying data sets are not just based on public data sources such as Freebase, Wikipedia etc. but rather; the real world data of our members – the people who know most about their own skills.  Using techniques such as Map-Reduce, we are able to mathematically decompose and model the probability of relationships and relevance of domain specific vocabularies and indeed cultures.

Having access to this incredible data is of limited value if it cannot be quickly channeled back into the platform for the benefit of our users.  Consequently, the entire SkillPages platform is designed to natively leverage SkillGraph to ensure that both individual and wide scale reclassification actions are instantly applied whilst maintaining the integrity and quality of the repository via feedback loops, moderation and spam detection.

Relevance is key – everything a user does on SkillPages from adding a skill, opportunity, profile, searching, browsing and opportunity matching, these are all driven from SkillGraph.

Finding The Best Matches

SkillGraph is rapidly developing as the definitive catalogue of all the world’s skills. The more data it gets, the longer its tail stretches.  However, to continue to gratify our users, we must ensure that even those at the tip of this tail are found.  Take for example an opportunity looking for someone “to make an interesting piece of furniture as a centerpiece for a back garden”. This is where the power of SkillGraph really kicks in.  Using the initial vocabulary as a starting point, SkillGraph can find many different skills to which this relates.  It not only makes obvious suggestions like “Furniture Maker” or “Landscape Gardener”, but also less obvious ones like Stonemason and even the highly obscure like Pooktre Tree Shaper.

Building a system that can organize all human skills, capture their relations and make them searchable in a meaningful way is not trivial.  In addition to software engineering challenges like scalability and keeping the latency low it requires a good grasp in disciplines like statistical analysis and linguistics. However, it’s very rewarding to see members gain success through SkillPages and our SkillGraph – it makes all the hard work worthwhile.