Why great technology needs 'artists' to thrive

Image caption Technology is the new artist's palette, says PeerIndex CTO Sanford Dickert. The key is finding the right people

Each week we ask high-profile technology decision-makers three questions.

PeerIndex aims to give credible measures of an individual's influence on the world wide web. It monitors social networks to judge audience size, interaction levels and areas of expertise in various subjects.

In doing so, it produces a ranking score designed to separate who is creating waves online and who is not.

Speaking to the BBC, chief technology officer Sanford Dickert explained what challenges the firm faces when finding people who can deal with 'big data' - and why he will never be completely happy with how PeerIndex works.

Image caption A production fault in his early career taught Mr Dickert about quality assurance

What's your biggest technology problem right now?

The biggest problem we're having today is finding really good 'big data' engineers in one physical location.

One of the challenges about 'big data' is that it's a relatively new concept. Granted, people have been data-mining for many years, but the real challenge is finding people who understand, for example, Hadoop, and machine learning and large data science concepts.

The difficultly is that Europe is just beginning to wake up to it in a more accessible fashion. Trying to get a large number of people working with big data is not particularly easy in any one location.

The ability to connect via telecoms has afforded us the ability to bring some of these people together. Our team is based across three countries, and that's primarily due to the fact we have three areas of expertise and the systems are able to be distributed. I'm finding people in those locations, but I keep seeking people in those areas and sometimes it's difficult. That's my biggest challenge.

The other challenge I face is finding technologists - engineers, scientists and so on - who think out of their comfort zone. One of the things I love about the West Coast is that it has become sort of a centre for the really solid, creative technologists to show up and do amazing things.

If you don't go to the West Coast, you sometimes go to the East Coast, New York, and sometimes get paid very well to be kind of creative for financial creatives.

The challenge is finding the artists. Technology is not just about logic, it's about seeing beyond the constraints you see already with the technology and finding a way to do things above and beyond. Technology is another palette for an artist. It's tough to find them.

Technology isn't the limitation right now. It's all about the people.

I will always be unhappy with how PeerIndex works in the fact that I'll always want it to be better - that's part of being an engineer.

We're happy with what we're doing because we're beginning to evolve our knowledge. We're trying to understand predictive analytics, trying to understand more about people who are influential, who is the person you wish to talk to in London about, for example, Arsenal footy - who would actually convince their friends to be interested in maybe some sort of Arsenal product. That is going to occur over time.

What's the next big tech thing in your industry?

Predictive analytics. The ability to understand people and other potential outcomes based on data, because as the algorithms become more and more sophisticated, predictions through algorithmic analysis - and also the ability to use data effectively - will help in terms of making technology both very scarily predictive as well as almost invisible.

If the algorithms get better, you don't need as much data because you will know which data points signal a certain outcome. Instead of having to consume a terabyte of data to figure out 'who using their cellphone is an Arsenal fan', if it only needs two or three data points, it reduces the complexity and cost of generating the outcome.

This is why Reid Hoffman said 'Data is web 3.0', because how to handle, consume and address that large set of data is going to be key.

As we get more data we begin to understand better models and better ways of predicting, which interestingly enough will reduce the cost because you don't need so much data when you know what are the signals which matter the most.

What's the biggest technology mistake you've ever made - either at work or in your own life?

My biggest mistake had to do with quality assurance (QA). Back when I was in grad school, I was working for IBM in the manufacturing line.

My boss had me go over this workcell which took care of wiring up a disk drive's end effector. There's a magno-resistive head that's on the end of an armature that's wired up by this device called a spinner. All it did was wind up the wire a certain number of times so that it would reduce the electro-magnetic discharge.

The problem was that we were running at a certain pace and we needed to speed it up. I found by analysing the workcell how I could optimise it and speed it up. I showed it to my boss - he said go ahead and deploy it. We increased the throughput - it was terrific, everything was great, we deployed it across the whole network.

We had nine work lines and I put it across the nine workcells. Unfortunately what he didn't do - my boss - was QA the fact that we had actually added an additional twist in the wire. The additional twist caused half of the magno-resistive heads to go one way, and half the other way. By the time it was discovered, after thanksgiving, we had essentially $300,000 worth of heads failing in the QA.

What it taught me that you never deploy anything until you test not just the 'I got it to work', but 'does the outcome come out the right way'.

I didn't get fired because my boss had said it was OK. It cost us time and money. The lesson was even a small mistake, such as a twist, could mess up the outcome. That's why testing and QA is so important.

Related Internet links

The BBC is not responsible for the content of external Internet sites