return 42;

by Jan Wedel

India - Scaling People

My experience with off-shoring teams and what went wrong.

I recently read a tweet from @sivalabs that made me think:

There is a huge pool of talented, hard working and energetic software developers in India. They just don't know that they can say NO to some of the unreasonable demands from their bosses and delivering poor quality of work to meet deadline.

Is that the really problem?

After one year of working with an off-shore team, we lost about a million Euros and needed to re-write most parts of the software. What actually was the problem? I've been thinking about that for quite some time. But let's rewind to mid 2015.

Manager: Jan, I need you to go to Bangalore to acquire a team in India.

Jan: Why?

M: Because I've got orders from top-management that we need to employ 10 developers off-shore.

J: Why?

M: Because it scales better.

J: People scale?

That discussion was pointless, as you might have guessed by now and I got my ticket to Bangalore. At least we have a business class policy for long-distance flights.

Honestly, I was not happy about that. Not only because it was a bullshit decision but because I've already had some bad experience in previous jobs. But I thought: Let's be positive. Let's learn from past mistakes and do it right this time.

Preparation

Hierarchy

First, we had a cross-cultural training which was pretty good. I learned that hierarchy still matters a lot in India today and talking across hierarchy levels is considered a no-go.

So we created a mapping table (no kidding, we really did that):

Role Us Them
Developer Person A Person E
Team Lead Person B Person F
Project Manager Person C Person G
Department Manager Person D Person H

The rule was to not talk cross-levels. E.g. I was at team lead level and I wouldn't talk to their developers or their Project Manager. Instead, I would talk to our counterpart first which then talks to them.

Small-talk and Direct Approach

Second, we learned that you always start your conversation with some small talk. I hate small-talk.

You would never directly criticize people but always say something in a roundabout way. I hate that, too. Looks like I'm the right guy for this job, huh?

The Plan

Now, we needed to think about how we would actually do it. We planned to go with four people including me, another developer from my team, our project manager and the lead developer of an in-house framework that was supposed to be used in that project. We would spend three weeks in total in India. We would have one week to recruit developers from our company's off-shoring pool and then two weeks to coach the team. Recruiting would be a two step process, first an online assessment and then a face-to-face interview.

Bangalore

After a very pleasant flight, the next day we came to the office to meet everyone. We got a driver to us to "Cyber Park, Electronic City". I was already smiling at that point.

To our surprise, there was no one to meet in the office except of the project manager. We (as Germans) learned, that punctuality is not exactly top priority in India. After just waiting for 45 minutes, almost everybody joined our meeting.

We made it very clear, that we were looking for a team that is able to think independently, speak up if anything seems wrong from their perspective and that delivering high quality is more important that delivering in time.

Online Assessment

To the developers from the talent pool, we assigned a prepared online assessment that incorporated a couple of technical questions, writing code, tests, and some SQL.

That was a disaster. There was certainly no clear winner and the online tool reported, that candidates actually copied the solutions from each other. Some of them were sitting in the same office room. Great start.

Interviews

After reviewing the CVs of those who didn't copy, we picked a couple of developers ranging from junior to intermediate to senior level to get a good skill-mix. One thing that was very obvious was that a lot of them have been working in legacy or maintenance projects as well as 3rd-party support and did not have experience with up-to-date technology. Moreover, they were very eager to present a lot of certificates which I completely ignored because I don't give a shit about paper and grades. Skills matter. Smart people matter.

The next thing to learn was that there is actually a very significant difference in ranking developers into junior, intermediate and senior level. In Germany, you would not call yourself a senior but apply for a position as senior. In most of the CVs we got, the candidates actually presented themselves as e.g. "Senior Developer". Second, candidates rated themselves as "senior" as soon as they've worked for 2-3 years in fixing bugs in a legacy application. Usually from my experience, a senior would have 10-15 years of professional experience that includes larger projects, green field at best. But we obviously didn't have that kind of candidates.

Personally, I think that this is actually caused by the way that developers in India are treated from western companies: Cheap exchangeable developers mostly doing the stuff that others don't want without ever meeting them in person or relying on specific developers. That pushes people to advertise themselves as senior with lots of useless certificates.

One thing I found very surprising in a positive way was the high number of female developers. In Germany, you'd usually have a low single digit percentage and it's very hard to find female developers even if you're looking for them.

The interviews were OKish as were the candidates. We picked some seniors which we internally handled as intermediates and a couple of intermediates that we treated as juniors. It was very remarkable that a lot of candidates were unable to answer some basic questions regarding programming languages and frameworks that they listed in their CVs as "very good knowledge". And we needed to reject some candidates because we simply could not understand their English pronunciation. You know, most Germans have bad pronunciation as well but we knew that we needed a lot of remote communication across teams, we planned to have a weekly stand-up with their team and we figured that this was going to be a mess if we couldn't understand them talking face-to-face already.

Coaching

Next step was to coach the hand-picked team. As mentioned before, we had two developers from our team plus the lead developer of our in-house framework. But first off, we tried to emphasize on very basic but yet important things:

  • Independence: They should feel as a respected team that can make their own decisions.
  • Confidence: They should be able to speak-up if they see anything thats wrong or in-effective. We tried to point out that we also make mistakes and they should feel free to mention that.

We explained them how our development process looks like, what systems we use. We explained the functional concepts and requirements. We went through code that we've written, we explained the importance of clean code, test-driven development, testing in general and code-quality. We even did coding-dojos together with them.

After three weeks, we had a pretty good feeling. It went much better than we expected. We liked the team and most importantly, we felt that we couldn't have done much more than what we did during those three weeks to make it a success.

Development

Back in Germany, we had their team lead working with us in Germany for some weeks to get him even more integrated.

The first weeks, we had a daily stand-up meeting via Skype together which went quite good. The time difference was just 3 1/2 hours (yes, three and-a-half!), every piece of code was reviewed by one of our developers. But that was very exhaustive and couldn't obviously go on like that forever. Then we switched to weekly meetings, reviews were mostly done by their team lead.

The End

We actually expected a high turnover of developers because we were told that as soon as one got a slightly better offer, she's gone. That did not happen. The team nearly was stable, just one developer left because of private reasons.

Nevertheless, after some months we've had the feeling that something was going not so well. We've tracked the hours spent by our team to review code and helping them as well as feature output and code quality. After some tracking, it was pretty clear that we've only got the output and quality we needed when having one full-time-equivalent on our side making sure that quality goals are met.

Then we dug a little deeper in it and did some statistics on git commits. It turned out that from those 8-9 developers we hired, at the end of the project only 3 of them were actually committing code. Although I never got an explanation, their managers were probably selling their employees to multiple projects. I've heard that this happens from people working there.

As you can imagine, that escalated quickly and we eventually stopped that cooperation. After taking some in-depth review of the code, we decided to rewrite most parts of the backend service as well as the mobile app.

It cost us nearly a million Euros to get both applications to a state that was acceptable to our customer. That did not include the actual personnel, coaching and travel costs.

It seems that costs scale better than people.

2018

(New) Manager: Jan, we need to think about getting an off-shore team in India because it scales better.

Jan: But, but, but...

What Was the Problem?

I am seriously asking myself, what was the problem. Could we have done anything differently to make that a success story?

Remote Work

Obviously, working from remote is - let's say - a bit harder than if everyone is sitting together in an office. But it is manageable. We actually forced their manager to pay some of our off-shore devs a taxi every morning and afternoon to get from another location to the office where the rest of the developers were located so that they at least did not have to work remotely. The application they needed to develop was green-field, almost no dependencies so that could work pretty much autonomously. We did not throw a 200 pages requirements document over the fence but worked in an agile way together with them.

Skill Level

Yes, the candidates did not all have experience in the frameworks we needed. They probably worked in boring legacy projects. But as @sivalabs pointed out correctly in his tweet, there are a lot of talented and smart developers in India. Their school and university systems focus much better on educating people to work in tech businesses than we do in Germany.

And usually, if one likes her job, is passionate about software development, learning new technologies is actually fun and does not take that much time.

Hierarchy

That did not turn out to be that much of a problem, at least not as we expected it. We did not needed to uses our mapping table very often.

However, employees being sold to multiple projects without telling us might be the result of their hierarchy.

Focus on Quality

They just don't know that they can say NO to some of the unreasonable demands from their bosses and delivering poor quality of work to meet deadline.

There were no fixed deadlines, at least not in out project. It really seems to me that they are drilled to deliver as feature as quickly as possible as soon it superficially looks complete. In a former project, I coined that as CDD (compiler-driven-design). It's ready when it compiles. This is definitely a significant issue if you need to have a team with equal rights and obligations that works together on a larger project.

In 2017, there was a wave of Indian candidates that applied for an internship regardless of the fact that they actually worked for a couple of years as profession developers in India. I asked one of them, we he actually came to Germany. He said, he hated the spirit, the ways software was developed in India. The lack of quality. He also came from Bangalore.

Value Individuals & Work-life Balance

When I first entered the building which was 10 stories high and full of cubicles, I thought I would not want to work here.

Most of them came pretty late (around 9 or 10am), mostly due to heavy traffic in the city. Some of them took a 1 hour (or more) ride each one-way. Then, they made long hours. 10-11 hours was pretty normal. However, they did a lot of social interaction, chatting in the kitchen etc. which they actually subtracted from their daily hours. So, I hardly found anyone that just did their 8 hours and went home.

So to sum that up. most developers spent a considerable longer time at work but their are - as far as I noticed - not valued as individuals but rather as exchangeable resources. That might lead to less commitment to their jobs.

Culture & Mentality

That's actually the toughest point for me to think about. I have the feeling that it might just be an incompatibility at a very basic human communications level. We tried to coach their team-lead to have a proxy that knows how we work and what we want. That person should than be able to translate that to his team in an appropriate way.

But it just did not work out.