Building better data analysts

Why Should We Move Back To Data IC Role After Being A Manager?

2024-11-14T00:00:00Z

This is Part 4 of a 4 part series, detailing the trials and tribulations of moving from being a data IC to data manager, why we might think about going back to Individual Contributor again, and what could go very wrong with that move.

In Part 4, we’ll look at:

What can go right with a move back to being a Data Individual Contributor from a management role.
The things you learned as a manager that can supercharge your IC skills.
And the things to consider when making that all-important final decision on whether to take the plunge.

What can go right

My goodness but the last part of the series was a bit doom and gloom, wasn’t it? Surely we’re not completely over the hill? Not a busted flush just yet, are we?

Not by a long chalk.

Show us what you’ve learned

Is it all pain and no gain? Of course not. You didn’t leave all of your hard won experience and skills at the door when you went over to ~~the Dark Side~~ the world of management in the first place.

And you won’t leave behind any of the new skills you’ve picked up over there if you move back.

You’ll need some time to get back up to speed on the technical side of things, the world waits for no-one in that regard. And the modern data stack is an unforgiving mistress if you take your eye off the ball for too long.

However, the superpowers you will have developed, maybe under the radar, even to yourself, will more than compensate. In fact, they can help propel you to new heights of possibility as an IC.

Taking Off The Shirt - Superman GIFfrom Superman GIFs

Prioritizing the really important work is one of the most important skills any IC can develop As a line manager, you’ll have had this as one of your main tasks, day in, day out. Why wouldn’t you bring those talents back over to the IC world?

Understanding the business domain.
Seeing the value of building a team ethos while working on the same level as your team-mates.
Spotting possibilities for cross-pollination and silo-busting with other teams outside of your function and org chart limits.

GOLD. DUST. Ignore these superpowers at your peril.

Winning friends and influencing people

You can be the best data analyst in the whole world. Stats genius. Coding legend. Visionary gift of second sight into the hearts, minds, and souls, of the customers who are the lifeblood of your business.

But if you’re an insufferable asshole who can’t get along with anyone, on your team, in your management, or throughout your org, then you’re stuffed.

The rest means absolutely nothing.

That doesn’t mean being likeable is the most important thing you can do to further your career. It definintely helps rather than hinders but you don’t want to turn into Captain Butt-Kiss to achieve it.

But…you remember the analogy of being the tennis ball that gets bashed between upper management and the working class when you are a junior/middle manager?

All of that experience of managing up to the top brass, managing sideways with your managerial peers, and managing down to your team members, really does stand you in good stead for becoming a better IC when you go back.

In fact, it’s vital.

Becoming partners, not servants

Building strong working bonds with your stakeholders, be they engineers, product managers, finance analysts, or whoever, is one of the most important aspects of being a successful data analyst IC.

It boils my piss to see so much of the “advice” from data influencers (whatever they are) being a shopping list of different technologies to learn.

Yes, it helps to be technically proficient. It helps a lot.

But it doesn’t mean a damn thing if you can’t work with your stakeholders.

Understand them and their role. Get deep into their business domain. Find their blockers. See where the overlap occurs between your previous world and where you are meeting them now.

That means a lot of:

Being attentive but not overbearing.
Being approachable but not bending over backwards for every little request.
Influencing without brow-beating.
Working together for the best possible end results, not just an excuse to show off some technical wizardry you are looking for a problem to use it on.

Am I an expert at all of these? Heck no. Go ask some of my previous and current work partners where I need to keep improving.

But work on it I will. For me, this is the best possible area to focus on if you want to maintain your IC career for the next couple of decades. No question about it.

“But I’ve been out of the game for a few years Coach, what could I possibly teach these young whippersnappers I’m working with now?”

Many of the benefits of moving back to being a data IC from management aren’t exactly tangible in nature. It’s about the skills and experiences you’ll have built up, even if you didn’t realise that was happening at the time.

It’s about working with people, whether you self-identify as a “people person” or not.

It’s about being able to take a mental model of the biggest picture, split it down into smaller ones, break those down further to manageable chunks, and then look for how to do it all better.

And it’s about communicating that to your team-mates, your ~~stakeholders~~ partners, and your leadership.

It’s about recognising how you do that, even if it’s not something you even realised that you do in the first place. And, most of all, it’s about helping those around you learn better ways to do those things you do so they can up their game too.

Learning, sharing, and making the whole environment even a little bit better. That’s what it’s all about.

And who better to provide that mentorship, to bring others along with you, than someone who’s seen it from the other side of the room as well?

Room for creativity

I’m not saying management isn’t a realm suitable for creative work. Getting people to do something that they don’t necessarily want to do for the greater good of the company can require the greatest creativity imaginable. No doubt about it.

But for me, it was never enough.

I get sad when I read a lot of data discourse online and it’s stuck in the “data vending machine” mindset. Someone in the company punches in an instruction and the data team spew out a report without any creative input to it. What a waste for all involved.

I’m not saying data analysts are always creators of great art, although there’s no denying the artistic talents of many data visualization experts.

We are largely creative thinkers though.

We need to see the business domain, understand how it fits together, and apply that to the multitude of data sources that exist, or could exist, to help us understand it all even better.

This is the part I love about the job. Providing the bills were covered, this is what would keep me doing this job until I’m ready to retire.

Identifying the important strands, finding the data, putting the model together then telling folks a story with what you’ve found. That’s what I like to do.

Being a data IC allows me to do that every day in life. Why wouldn’t I want to jump back over to this ladder and see how far I could take it this time around?

What should I do?

The late English philosopher Alan Watts did a very inspirational lecture on the subject of what would you do with your life if money was no object. Every time I’m in a professional rut, I put the Youtube video on and give myself a mental kick up the rear end.

Am I saying that I listen to Watts and think “if money was no object, I’d be a data analyst!”? Eh, no, not quite.

I love this job, don’t get me wrong, but I’d still swap careers with Stephen King, Slash, or Heung-min Son in an instant.

We’ve go to be realistic though.

We’re talking about choosing between being a data manager or a data analyst. Either way, the data world’s gain is fiction, rock music and football’s loss. Them’s the breaks kid.

It’s not quite a grand existential crisis question. But if we put aside a lot of the societal and financial reasons for choosing one path or the other, which do you want to do the most?

It’s a big decision but, in your heart of hearts, maybe Alan Watts’ advice can help get you over the line on what’s best for you right now.

Wrapping it up (for now)

We looked at many different reasons not to do it, and a good many that made my own decision up for me over five years ago. I haven’t regretted it for a second (disclaimer: that’s probably not true, the path of true commerce never runs smoothly but I don’t regret it at all right now and that’s enough for me.)

Your mileage may vary but the one aspect I always think about is the fact that life really is too short to purposely spend it in a miserable working state.

Consider that part as carefully as the, maybe temporary, loss of prestige, power and ladder-climbing possibilities if you do cross back over the aisle to the IC side.

What’s the worst that can happen?

And if it doesn’t work out then just go back to the first instalment in this career journey series and go be a data manager again. Sure it’ll be a cakewalk after you’ve been there and gone back again!

Good luck in your decisions and future careers whichever way you decide to go. If you have any questions or anecdotes of your own to tell, please get in touch. It’s good to talk and share these experiences with each other.

Photo by Levi Guzman on Unsplash

What Can Go Wrong When Moving Back To Data IC Role After Being A Manager?

2024-03-30T00:00:00Z

This is Part 3 of a 4 part series, detailing the trials and tribulations of moving from being a data IC to data manager and why we might think about going back to Individual Contributor again.

Quick Recap

In Part 1, we looked at the reason why we’d move over from being a data IC (in my case, an analyst) to becoming a manager of data analysts.
TL;DR glass ceiling, money, seemed like the right thing to do at the time.

In Part 2, we looked at why that move might have started to turn sour and make us cast wistful glances back across the aisle to our old IC stomping ground.
TL;DR tired of politics, missing doing what we actually enjoy, scared of messing up our careers.

What have I really learned?

Like anything in life, there are wins and losses, ups and downs. But if we put aside the hypotheticals and voices in our heads, how can it really work out in practice?

We have to be fully armed with the potential downsides. So let’s take a run through what can go wrong with the move back to IC from data manager first.

And then, in Part 4, we’ll do the positive ending to bring this all home.

What can go very right? What are the reasons we’d do it, even in the face of adversity and potential disaster? And, ultimately, why none of this should be a reason to lose sleep unnecessarily in the first place.

What can go wrong?

You don’t belong here

Ever get the feeling that you wandered into a room that you don’t belong in? Even worse, the looks on the faces of the people in the room seem to very clearly show that they know you don’t belong there either.

Whether the second part is actually true or not (if it is you’re really in the wrong room, life’s too short to be around assholes like that every day), you need to recognise it for what it is: the age-old plague of imposter syndrome.

That relentless annoying voice in your head that tells you that you aren’t actually good enough to have gotten this job and ANY SECOND NOW they’ll catch you on and give you the old Jazzy Jeff heave-ho out the door.

I won’t lie.

It’s damn hard going back to being an IC if you’ve let your skills gather a little dust over the past few years as a manager.

Even harder when you get into it and see a load of younger, better qualified, (seemingly) smarter people than you doing what you used to do back in the day. Only they’re doing it better.

Spoiler alert: they’re usually all feeling the same way. Tell the voice to shut up, you deserve this, you earned it. Don’t go sabotaging yourself and remember, if you really don’t deserve to be there on merit, they’ll catch you on pretty quickly anyway. So why worry?

Age is just a (big) number

Chris Rock does a great bit in his Bring The Pain routine where he talks about not wanting to be the old guy in the club.

Take it away Chris:

Every club you go into, there’s always some old guy. He ain’t really old, just a little too old to be in the club.

Like being a 40 year old starting back into being a data analyst IC at a hip young tech company and being told you meet the criteria for being an Under-Represented Minority.

Find me a white, straight, university educated, Western European man who squeaks into any other minority group in the world. I’ll wait.

Privilege doesn’t come any more privileged than the boxes I was able to tick but here I am as Chris Rock’s old guy in the club. Damn this natural process of ageing.

You need to be clear on how this is going to work in practice.

Your co-workers will be younger.
Your manager will be younger.
Your senior leadership will be younger.
The C-suite will be younger.

You’re going to have to print out a sheet that says “AGE IS JUST A NUMBER” and Blu Tac it on the wall in your office.

Hold on, did you just say “print out”? You own a printer? Are you from the 1970s? How old are you?

Ah damn it, busted again. Might as well have opened with the line “Back in my day…” 😦

Money matters

When the IC job market was red hot a couple of years ago, this mightn’t have been such an issue. Comp packages and bonuses were getting bigger to attract the top talent as companies went on hiring sprees and needed to compete to get their chosen hot-shots in the door.

You may not have noticed (ha!) but times have changed.

Layoffs in many sectors, not just the metaphorical bloodbath in tech, have seen a dramatic softening in this area as a glut of skilled ICs hit the job market, pushing wages down.

If your managerial experience is at a company that views management as a promotion then maybe you got a salary bump when you went in there.

Looking at a slightly more junior level over in IC World, as you try to get your foot back in the door in this constricted market, might mean a sizeable drop in salary.

Inflation is still on the march.
Bills are through the roof.
Mortgages and rents haven’t corrected.
Staple foods are now seen as luxuries.

Can you really afford to take a pay cut right now and risk never getting back to where you already are?

Head-butting the glass ceiling

It’s still a rare organization that truly runs parallel tracks for managers and ICs right up to the higher echelons of the company.

Even if they do, the air gets thinner and the opportunities fewer and fewer for progression once you drag yourself a level or two above the general population.

Maybe it’s an age thing again but where do you see yourself at age 50 or 60? Still setting up dashboard tiles in Looker or Tableau? Even if you are happy with that (and that’s perfectly fine, not hating on it at all), how does it look to hiring managers?

I saw a post on LinkedIn recently about a senior software engineer in their late 50s/early 60s who didn’t get an engineering job they’d applied for. The reason given was that they didn’t display enough drive in their career because they weren’t a CTO already.

WTF?

Either there’s a glass ceiling for ICs who don’t jump into management (which happens in many places), or every single IC should be in the C-suite by the time they reach 30 years of experience. Which is it likely to be?

If I jump back into being an IC, am I destroying my future career prospects? How many years can I expect to eke out of this move before getting slung on the scrap heap, either for being too ambitious or not ambitious enough?

Good questions all.

Who’s the boss?

Once you’ve been the one dishing out the orders and got your head thinking in that way, it can be difficult to go back to being the order-taker, not the order-giver. It really depends on your whole view of the management / IC relationship.

My personal philosophy on this is simple: management works for the ICs, not the other way around. It all comes back to that concept of Servant Leadership.

When The Rolling Stones go out on tour, Mick Jagger doesn’t work for the tour manager. The tour manager works to remove all possible roadblocks to enable Mick to take of business where it really matters: out on the stage.

They are a team. The Stones wouldn’t put on anywhere near as good a show without their great tour manager.

But without Mick, there wouldn’t be a show at all.

And no, that doesn’t mean I think I’m Mick Jagger. Of course not. I’m obviously Keith Richards…

TL;DR A good manager is a force multiplier for the positive impact of their team. But nothing happens at all without good ICs.

Getting the balance right between you and your new manager will be vital to making this transition back to IC work. If you can’t, or won’t, find the right working groove between you then you’re dead in the water.

It’s not all doom and gloom

Don’t be downhearted at this stage.

We had to cover off the downsides to make sure you were properly armed to make a big decision at the end of this. And you can’t do that without knowing both sides of the story.

In the final part of this series, we’ll look at what can go very right with making that jump back to being an IC, and what we need to consider to make the decision.

Check out There And Back Again Part 4: Why Should We Move Back To Data IC Role After Being A Manager?

Photo by Zbynek Burival on Unsplash

Can I Go Back To Being A Data IC After Being A Manager?

2024-03-29T00:00:00Z

This is Part 2 of a 4 part series, detailing the trials and tribulations of moving from being a data IC to data manager, and back to IC again.

Once upon a time, there was an analyst who became a manager

And they wrote an article about why they did it, and stuck with the decision, through thick and thin, the good times and the bad. And the ~~analyst~~ manager was happy…but not really.

It turned out that the career path that had been laid out in that move into management wasn’t all that it was cracked up to be.

And to be truthful, it hadn’t ever been sold as that in the first place. It was just something you were expected to do.

Want to keep “moving on up”? Get your foot on the first rung of that ladder and start climbing. Get on up there.

Moving up or holding on for grim death?

But what happens when you’ve done it for a few years and every day is becoming a grind?

What got you here is not what will get you to the next level…if you even want that. And the skills you mastered in your previous glorious Individual Contributor (IC) adventures will have started to dull and fade.

What if you could do a Cher and turn back time, casting off the shackles of your management prison and jumping back into being an IC?

Is that even an option? And, if so, what would the pitfalls and ramifications be to the world you’ve created for yourself?

You’re about to hear from someone who knows i.e. me. I did it. It’s been over five years since I made the full-time leap back into the wicked world of ICs.

And, now, it’s time to talk about it.

Why would you do it?

I quoted startup founder Rob Walling in the first instalment of this career journey saga, and this part sums it up perfectly:

“If you find yourself in management and wish you were coding, don’t wait until you’re completely burned out and bitter; do yourself (and your company) a favor – and do it now. Get yourself back into the code jockey seat.

There are many different reasons why you would reach tipping point on the decision though. Every person in this situation will be different but here are a few that I wrestled with that finally made the decision for me.

The eternal grind of office politics and people

Managing up AND managing down is tiring.

Especially in larger companies, I picture the back-and-forward between the upper exec and senior leadership levels and the rank-and-file workers on the ground as a tennis match.

As a junior/middle manager, you will find yourself in the unenviable position of always being the tennis ball.

You get whacked by one side as they send down orders from above to be enacted upon the proletariat.
And you get whacked back in the other direction when said ICs take umbrage at the directions they’re being dictated by from above.

Don’t be fooled. Shit always flows downhill.

But it sure does feel appealing to only have to deal with that slop bucket being poured over your head rather than having to be the pourer and the “pouree” at the same time.

What have you done for me lately?

As a manager, your main priority will be to grow the skills and opportunities of your team. That means you will primarily be responsible for growing other people’s careers, but not necessarily your own.

If you are very successful at this, they could use this coaching and talent management extravaganza to get good enough to jump ship and ditch you for a better offer elsewhere.

Then you have to pick up a rookie to start from scratch on, or a grizzled old veteran who thinks you’re a knob and maybe should have your job for themselves.

The rookie might think this too to be fair. It’s not mutually exclusive.

Meanwhile, your own technical skills have disappeared into the mists of time, as relevant to the modern data stack as an abacus and chalkboard.

And this is a successful scenario for you. Well done, you have truly become a Servant Leader.

Wanting to do the actual work

This is what Rob Walling was talking about. If you are doing the managing but secretly (or not so secretly) wishing you were coding up a storm instead, then the writing is on the wall.

Sarah Mei wrote a great Twitter thread, back in 2017, on all of this, and sums up this part particularly well:

“There are lots of people who do both well - but serially. Not simultaneously.”

Preach it Sarah.

When I was a manager, I usually kept the best technical projects for myself. Sorry not sorry.

This was both a managerial decision. (I felt I was the best player on the team to handle them properly.)

And a personal professional choice. (You could have me do the dirty parts of management but you would have to pry the interesting jobs I enjoyed doing from my cold dead hands.)

When it came time to sit and meditate over the “stay as a manager vs. return to being an IC” question, this was a big factor. The biggest actually if I’m being really honest.

A change is as good as a rest.

Say it after me: my mental health matters. Louder. Even louder, once again. Now shout it out, over and over and over again until your throat gets hoarse and raw and scratchy.

There is no company, corporation, organization or business on this planet that deserves you setting yourself on metaphorical fire for them. You are a cog in a wheel. A number on a spreadsheet. A needle in a haystack.

If you don’t believe me then wait until you get laid off for the first time. Or the second. Or the fourth (and counting, that game never ends…)

If the job of management is causing you so many literal headaches as to impact your health, then it might well be a good pointer that it’s time to go and do something different.

Maybe I was just a bad manager?

Maybe I was. My manager and our head of department didn’t think so. They told me the opposite regularly.

When I finally left the company, my most trusted team members told me I was a good manager. And none of those folks had any cause to lie to me. I wasn’t signing off any more performance reviews by that time, my “power” had been relinquished.

There would be a few (ahem) who might offer a different report but…you can’t please all of the people all of the time.

It just wasn’t a discipline that I envisaged myself ever wanting to get really really good at. I didn’t want to be a bad manager.

But I also didn’t care enough to be a really good manager either. Or at least not only a good manager of other people.

Isn’t it career suicide?

So once you’ve weighed up the factors that can push you towards a decision, what else can stop you calling time on this leg of your management voyage? Fear is usually top of the list.

Fear of the unknown.
Fear of public perception.
Fear of ridicule and scorn from your family and friends.
Fear of financial catastrophe.
Fear of messing up.

Fear. Fear. Fear. Fear. So much fear.

But how do we overcome it? By facing those fears and seeing if they are real and what we really have to fear from them.

Life’s too short.

What would you rather endure: a slow, painful death march towards retirement in a job you hate or a painful rebirth into a possible bright new future?

“Get busy living or get busy dying”, if we’re to quote the ever wonderful Shawshank Redemption.

Admittedly, this isn’t a binary choice. There isn’t a right answer here, both have their perils and pitfalls.

To keep the Shawshank vibe alive, both will probably involve crawling through many miles of tunnels full of pure poop that you’ll inhale, ingest, and have to swallow whether you like it or not.

And there’s no guarantee that ditching the current situation will allow you to exit into glorious freedom. We can’t all wind up drinking beer and renovating boats with Morgan Freeman on beautiful Mexican beaches. (Can we?)

But…if you’re hurting now, do you want to do another 20 or 30 years and regret not finding out if something better was really out there?

You can never go back.

Horseshit and nonsense. These doors swing both ways. Always have.

Maybe not at the company you’re currently with but, if they were so great, why are you considering this major career change again anyway?

I’ve put off writing about this whole leg of my career journey because the definitive essays on the subject have been written over the past few years by Honeycomb CTO Charity Majors.

I’ll link all four of Charity’s quintessential Engineer-Manager Pendulum series at the bottom of this post.

TL;DR she not only shows how it’s possible but also recommends it: not even once, but multiple times during a technical career.

Going back to IC is a demotion.

My original post on this is titled Becoming An Analytics Manager Isn’t A Promotion. It’s A Career Change.

I’m going to caveat that. It shouldn’t be seen as a promotion. That it is at ~~many~~ most companies is a damnable indictment of poor corporate and organizational culture.

This leads to multiple problems of hierarchical leadership conflicts, unaligned expectations between management and ICs, and, very damagingly, the following perception:

If management IS a promotion then going back to IC from management is a demotion. Why would I do that?

We need to break this taboo. We have to make people more comfortable jumping across from one boat to the other, and not stigmatising them because of it.

Smash the false hierarchy problem today and we’ll all benefit from that increased mobility longer term.

Do you really want to go on up the ladder?

Some people see a ladder and they have to climb it. Doesn’t matter where it goes, they take that first step and the feet and hands just keep on moving.

Now, taking a junior data manager job five years ago doesn’t mean that the only possible destination for you is the C suite. But where do you see yourself going with it?

Staying as a line manager forever?
Moving up another level to senior and becoming a manager of managers?
Up again to become a manager of managers of managers? Manager-ception.

When is enough going to be enough?

If the thought of that makes you vomit then let’s go back to an earlier question: would you rather be coding/analysing than managing? If so, why would you ever force yourself further and deeper into the underground cave system of management?

And if that upward trajectory does sound like your bag then…well, have at it, it takes all sorts in this life.

What’s Next

In the next instalment of this saga, we’ll take a run through some of the main things that can go wrong, and also some of the things that can go right, if we do decide to jump ship back to being an IC.

Check out There And Back Again Part 3: What Can Go Wrong When Moving Back To Data IC Role After Being A Manager?

Some Further Recommended Reading

The Charity Majors Collection
The Engineer-Manager Pendulum
Twin Anxieties of the Engineer-Manager Pendulum
Engineering Management - The Pendulum or The Ladder
If Management Isn’t A Promotion Then Engineering Isn’t A Demotion
The Official Authorized List of Legitimate Reasons For Deciding To Become A Manager

Main Photo by Rob Wicks on Unsplash

How To Build Your Data Science Career Change Action Plan

2024-01-23T00:00:00Z

Your Job’s A Joke. You’re Broke. Your Love Life’s DOA.

You’ve spent the last ten years grinding it out at a job that you don’t exactly swoon over when the alarm clock goes off. It’s starting to feel like life is passing you by.

And yet, you know there must be something out there that you can really get your teeth into. But which direction do you go to find it?

Looking for a new life…in all the right places

So you read some articles online about Data Science being the sexiest new career around. Well, since the last sexiest new career anyway.

And it starts to sound interesting. Who wouldn’t want to find out more and see if this whole data world could open up for them?

I’ll be honest now. Coming into the data field at age 30 or above without any prior experience or tech background can be tricky. To short-circuit the whole “how do I even get started?” problem I always suggest planning backwards.

What is Backwards Planning?

Backwards Planning is a great concept I learned from Amy Hoy and Alex Hillman from stackingthebricks.com. Essentially it means looking at where you want to get to as an end product (e.g. writing a book, building a data product…hooking a data analyst job), and working backwards from there.

For example:

What does the end result look like?
What’s essential and what’s nice to have?
What do I need to prepare up front?
What has to come first? Etc. etc.

All questions to take you backwards from your end goal to where you are now so you can fill in the blanks in between. (You can read more about using Backwards Planning in Amy and Alex’s book Just Fucking Ship).

When it comes to approaching a career change you need to look at getting that new data analyst job as your ultimate end goal. If you know your end goal then you can plan backwards to find the steps you need to take between there and now to get you to that place.

These five steps in fact.

The 5 Step Guide To Backwards Planning Your Career Change.

1) Research your targets

First up, go to a jobs site like Linkedin or Indeed and search for “junior data analyst” jobs in your hometown.
Go through the wordy job descriptions and make notes on what they are asking for.
Pay particular attention to the technical skills, education and soft skills sections.

When you’ve got enough data together, you can start to pick out the actual skills and requirements that will get you in the door at more of these jobs.

Got those skills already?

Great, you’re one step ahead. If not then it’s time to work out how to get them.

2) Technical Skills

I’ll go out on a limb here but you will probably see Excel and SQL pop up most often in Required Skills. These are the fundamentals of working in a BI or analytical role in the vast majority of companies, large and small.

If you don’t have these skills then you know where to start your learning. You won’t have to be the world’s best Excel jockey or SQL superstar either.

You’d be surprised how many people get tripped up at the first hurdle on these entry requirements. Make sure that isn’t you.

Excel and SQL really are two givens in terms of necessary knowledge in the broadest range of companies so it makes sense to pick those off first.

More bang for your buck.

At this level there’s no sense in spending a fortune on training when you have so many excellent courses available for free (including my own FREE SQL Crash Course).

I’ve put together a list of the best free and paid online courses for learning SQL from scratch. Check it out and save yourself a LOT of legwork: Best SQL Online Courses, Training and Tutorials.

Have a look on there and jump on in. I’m a big believer in getting the Pareto Principle and making sure you get the basics right.

If you can nail 80% of the most important SQL commands or Excel functions you’ll be streets ahead of the entry level competition.

Even those graduates from fancy colleges. In fact, ESPECIALLY those graduates from fancy colleges.

Don’t I need to know how to code from Day One?

You can get into a coding language like Python or R afterwards. Having a curiosity to dig into datasets, thinking of questions and then finding the answers from the data is far more important than listing 20 different programming languages at this stage of a career change.

Those of us who scan CVs and resumes when hiring analysts can tell immediately that you are a bullshit artist if you rhyme off a shopping list of programming languages and database technologies. Especially if you are straight out of university.

Better to know the important ones well.

3) Education

Do you need a 2 year college course or bootcamp in Analytics to learn Excel and SQL? No. Not at all.

Despite that, not having even a bachelor’s degree will tend to be a barrier to getting in the door of many organisations. I don’t agree that they SHOULD require it.

I’ve worked with plenty of excellent folks who don’t have them and it hasn’t stopped them being damn fine analysts but HR departments often disagree. If you don’t have one, search out the jobs that don’t ask for it and see what they want instead.

Or just apply anyway. Let them weed you out if they want but don’t rule yourself out unnecessarily.

Actually, STOP and write that last piece of advice down right now. It’s a big one.

In fact, that’s probably the most important piece of advice I can give for your job search. Companies employ masses of HR people who are only fit to scan for buzzwords on application forms. If they have a strict policy like only hiring graduates for entry level analyst jobs or only hiring from top level colleges then it’s their loss.

Angela Bassa is the Director of Data Science at iRobot. She covered this very subject in a Tweet thread about how she wouldn’t have passed the Master’s or higher graduate-level degree bar herself on a team that SHE set up. And she went to MIT!

It’s a ridiculous example of artificial gatekeeping. But in a crowded job market it happens.

Not all hiring departments are the same.

Some companies are a little more forward thinking however. They give their recruiting managers a little more leeway when it comes to who they bring in for an interview.

As more people change careers in their 30s, 40s and 50s than ever before, it makes sense to not rule out picking up a diamond that may have otherwise been discarded.

TOP TIP: Don’t limit yourself by deciding you aren’t fit to apply. That’s someone else’s job, not yours. Fortune favours the brave. If that’s you then the next step will be where you can really shine.

4) Soft Skills

Let’s say you’ve worked as a chef. If there is a more highly stressful, on-demand, pressurised working environment I’ve yet to see it. You should have learned plenty of skills running your kitchen that can transfer over to a BI analyst role.

See what the job listings ask for (e.g. “Ability to be self-managed, work independently as well as to collaborate within an agile, fast-paced, dynamic team environment.”).

Things to think about in your own background are:

Have you ever worked with other people?
Have you worked on something on your own?
Been under pressure on the job? How did you deal with it?
Was there something broken in the process you were meant to follow? How did you fix it?

These are the kind of open-ended questions that really show how you will deal with situations in the real world. It’s not all about the technical chops, especially for folks with some previous life experience in other arenas.

You mightn’t be able to quote the exact dictionary definition of what a SQL window function is but if you can show me you know how to Google things, I’ll respect that more.

Tell us a little about yourself.

Prepare some examples of when you have displayed these soft skills in your career and life to date. Don’t be afraid to use non-work related examples either.

I’ve had people use their experience in sports clubs or other organisations to display their soft skills at interview. I’m always happy to explore those as it shows they can transfer the skills beyond just the office work environment.

NOTE: I should say that I hate the connotations of these highly valuable skills being termed “soft skills”. It’s as if they are an afterthought or not quite as useful as the (presumably) “hard” technical skills of programming and working with databases.

Anyone who has ever had to manage a team of other grown-up human beings will know this is horse shit.

5) Show us what you got

When you’ve built up your skills a little, it’s time to show people what you can do.

Blog.

Write blog posts documenting how you approached a project. There is a common misconception that, seeing as everything in the world has already been written about, there is no point you doing the same thing.

That. Is. Bullshit.

Write about what you have learned regardless of how basic you think it is. Get used to communicating your process and what you’ve found.

HINT: these are REALLY important skills to have as a professional data analyst.

Analyze this, analyze that.

Or get a dataset, analyze it and present what you’ve found. (Just don’t use pie charts. Please.)

Interested in sports? You’ll find plenty of sports data online, jump right in.

Got a book on your coffee table full of the US Government’s notes on Italian-American gangsters of the mid-20th century? Source the data, cleanse it, wrangle it and analyze it (just like a project I’m currently in the middle of writing up).

Learn from the experts.

For your online portfolio, Google and read as much as you can from other data science folks who have posted theirs online. Unfortunately a lot of us aren’t able to discuss actual projects we’ve done on the job due to NDAs etc. but there are still a lot of great resources out there to pick over.

See what the format is like. How did their voice come over in the writing? How did they present it? Github/Jupyter Notebook/R Markdown?

It’s all research, all analysis. Even the searching and researching is building your data skills. Learn from everyone, pick the best elements and plunder all you can to learn more.

The Final Pep Talk.

It might look like a lot of work but this is a great field to get into. Career progression through the many different areas of data science is good and well compensated. Getting a foot in the door is the hard part but it’s well worth it and there is no better time to start than today.

Overall, I’d say just work hard on picking up just enough new skills to be able to demonstrate you can use them. Don’t wait until you consider yourself an expert to get applying for jobs.

The real learning, as in any field, is done on the job as you are adapting to real world situations. And, on that front, in the immortal words of The Carpenters, “we’ve only just begun…”

(Photo by rawpixel on Unsplash)

The Five Elements Of The Data Science Unicorn

2024-01-23T00:00:00Z

You pushing your Generalist agenda again?

I’ve written before that I believe that generalists are the real data science unicorns.

Especially when you are running a small company or early stage startup. Their mix of skills, and ability to roll with the punches, are invaluable as we get a never-ending tidal wave of rapidly changing data demands crashing over us.

A quick glance at job listings for data scientists immediately makes me balk though.

The never-ending shopping list of technologies.
The academic list of honours that would take anyone until they were in their thirties to complete.
The required list of side projects.
The open source contributions.
The burgeoning Github repository of polished data science portfolio projects.

Yes, we prefer generalists but how much do you really need to know, and have done, to qualify? Which skills are really the most important for our small company Data Science Unicorns and how do you know if you have what it takes?

The Five Elements Theory Of Small Team Data Science.

These are the five main elements I look for when chasing my own particular data science unicorn:

The Plumber

Having knowledge of setting up and looking after relational databases is vital. Doesn’t particularly matter whether it’s SQL Server, Oracle or PostgreSQL. The underlying concepts and skills can be ported from one particular technology to another and this is a strand you will see coming up in some of the subsequent skill-sets.

Knowledge of SQL is therefore #1 on my list of must-have skills regardless of the seniority of the role I am recruiting for.

Knowing how to take data out of a database, change it and put it back in somewhere else is so fundamental as to be a given. Long term if you find that the arena of ETL, data management, data cleansing and aggregation is your chosen home then you’ll be prime material for becoming a Data Engineer.

Without those basic skills however, you’d be no use to me. I need people who can get the data, tidy it up and make it presentable. Both for themselves, the other analysts on the team and for our customers.

If you find that you’re a “Senior Data Analyst” and you still have to sit around waiting for other people to hand you clean datasets to work on, you’re virtually useless to me.

The Detective

Inspector Clouseau. Lieutenant Columbo. Hercule Poirot. Sherlock Holmes. Jessica Fletcher. Nancy Drew.

What do they all have in common? A penchant for sticking their nose into things that often shouldn’t concern them which leads to them finding out the necessary details to bring a case (read: project) to a satisfactory conclusion.

We all need a touch of the data detective about us, especially those who err more on the analyst side of the business.

It’s not enough to just blindly trot out numbers in a presentation or results from a machine learning script. You have to want to question them and get under their skin.

Play good cop, bad cop. Even if it’s just by yourself.

Does this tell me what I want to hear?
What should I check to make sure that’s not a false positive?
Is the data quality good or just about good enough?

Wrangle the source data a bit more and see what the impact is. Take a data dump and run some analysis on it. Look for patterns.

This is often the unglamorous side of the job. But it’s absolutely essential to have some of these skills in your locker.

If you run a report that tells me we have an average Customer Lifetime Value ten times what I actually know it is, you won’t last long on my team. Knowing when something looks off is more important than having a photographic memory for every number that ever crosses your screen.

(Feed your inner Sherlock but if you find yourself in a series of American coastal towns and your close friends seem to keep getting bumped off just as you arrive, maybe lay off the Jessica Fletcher part a little.)

Mr. Robot

You don’t have to be a cyber-vigilante or underground ethical hacktivist (whatever that is) but you really should be coming in with some kind of coding background. Too many aspiring data scientists get caught up on the specific language they should know to make them more hire-able.

“Should I learn Python or R for data science?” Yes.

It doesn’t really matter which you do first though.

It’s more important to pick up the basics of programming and get those concepts into your head than worry about getting too deep into one particular language or another. If you know just enough Python, odds are you’ll pick up the main concepts of R quickly enough too. Or SAS. Or VBA. Or {insert archaic language here}.

Big corporations in particular are slow to move off their legacy tech stacks. You might want to be working on the bleeding edge all the time but sometimes you’ve just got to roll with the punches.

I know one very senior analyst with twenty odd years of VB development who didn’t let his lack of SAS knowledge stop him moving to that language when the call came. He just used his SQL knowledge to run PROC SQL queries until he learned more about Data Steps.

Be a coding MacGyver. Use whatever you have to hand and keep morphing when the situation demands it.

The Storyteller

This is how you deliver your story to senior management and really unlock the power of the analysis your team has produced. Mixing the personal presentation skills of a seasoned TED Talker with the data visualisation chops of a staff writer on The Pudding.

It’s these skills that knock the old “if a data analysis project falls over in an empty boardroom, does anyone hear it?” question for six.

You might be old school and go with Powerpoint decks and Excel generated charts. (Just not pie charts please, we all have our limits and standards). Maybe you’re more into matplotlib in Python or ggplot2 in R. Or even D3.

Most importantly, you’ll understand what is going on in the numbers behind the visuals and be able to communicate that to what is likely to be a non-technical audience. You have to get the story out and winning over the room is often the most difficult part.

Conquer that and you’ll have a string to your bow that translates VERY well to progressing up the old career ladder to management, and the Executive bathrooms above that.

The Double Agent

Maybe the most elusive element of our unicorn’s overall skill-set is the ability to play both sides.

Being on top of the technical side of the data science work, be it the data engineering, analysis or predictive modelling, is one thing.

Having enough of a finger on the business’s pulse to make it more than just another academic exercise with no real-world impact is quite another.

You’ll have to make both sides respect you as one of their own, while also appreciating that you carry enough kudos over from the “other side” to formulate and deliver actionable plans and projects.

So it’s not for the faint-hearted.

You can find yourself becoming the ping pong ball in the middle of inter-department politics. Batted from one side to the next and then back again with no real feeling of making any progress.

Dale Carnegie’s How To Win Friends And Influence People will be your best bedtime reading in that regard. You will need allies across the organisation to truly fulfil your mission while still maintaining enough deep knowledge of the technical side to not get left behind.

Speaking the dual languages of data-driven analyst insight and wily old gut feel business experience, she will be the linchpin on which all of the analytical hopes and dreams depend.

Bringing it all together.

And there we have it. Five elements. Five distinct skill-sets. Ideally spread in varying amounts around a small, tightly knit data team.

Finding all of them to a high level in one person is obviously nigh on impossible. If you do, you really will have found the mythical data science unicorn. Pay them extortionate amounts of money and ask them to recommend some like-minded friends.

If you are looking at it yourself as an aspiring, or current member, of the wider data science fraternity:

How do you think you personally measure up in each of those areas?
What could you do to beef up the areas you are weaker in?
How can you make the most of your strong points to get a bump up in your current job or a move to a new one?

And if you’re running a startup and are thinking it’s time to hire a data scientist but aren’t really sure what you are going to do with them - drop me a mail.

I’ll be glad to help point you in the right general direction.

(Main photo by Franck V. on Unsplash)

Fewer Dashboards, More Analysis

2024-01-23T00:00:00Z

It pains me to say it but Business Intelligence has gotten itself a bad rep. And we’re all partly to blame.

It’s pretty clear that to most people in the data science world, BI means building endless reports and dashboards - and that’s all it’s good for.

Take a look around your own workplace. I don’t care if it’s a bootstrapped start-up or a global corporate giant. Odds are that the virus-like spread of the BI dashboard has reached epidemic levels.

BI teams are stretched thin building more dashboards than the Tesla factory. And the requests just keep growing.

The business teams are demanding more and more and more. “I need a new sales report”, “I need another traffic report”, “I need another hundred metrics on this dashboard or else it’s useless and I can’t run my business and it’s all your fault and…and….and….”

Someone please wake me when the nightmare is over.

WE. NEED. TO STOP. FEEDING. THE. DASHBOARD. ADDICTION.

It’s not that I’m against providing valuable insight to management based on the data we have at our disposal. Quite the opposite.

I’ve spent most of my working life trying to deliver better insights through data-driven projects. I might get jaded at times when it feels like I’m banging my head against the metaphorical (and sometimes physical) brick wall. But I’m not suggesting we throw the baby out with the bathwater.

Gut feel (or what should more correctly be termed Subject Matter Expertise and Experience) alone doesn’t work either. We need a joined up approach that brings the two worlds together.

I want everyone to slow down on the “drive to dashboards”. Endless reports filled with countless metrics are not the best way to get full value out of either your data OR your data team.

Benn Stancil, the Chief Analyst at Mode, wrote an excellent article called Don’t Choose Dashboards Over Analysis. Benn says:

“Though it sounds counter-intuitive, more dashboards often make people less informed and less aligned…a company with hundreds of dashboards can’t focus on anything.”

And he’s dead right.

I see some dashboards so jam-packed with metrics it’s difficult to even work out what they contain, never mind what they are telling us.

The “More Data/More Dashboards” game-plan really only provides a comfort blanket for indecisive managers. They really want an inanimate object to blame if they make a decision that doesn’t quite pan out.

It’s a symptom of poor company culture that a scapegoat and sacrificial lamb have to be prepared well in advance of any business choices happening.

But that is often the choice junior and middle management have put in front of them. They carry all of the risk but little of the responsibility to really move the dial. So it’s easier to sit and play spreadsheet games with the moving numbers on the BI dashboard instead.

How does incessant dashboard building impact the data team?

My philosophy is that data analysts should be analysing data. Not blindly building more and more reports.

It’s our job to analyse and, most importantly, interpret what is happening in the business. Spotting trends and investigating the causes and possible drivers is where the real value is gleaned from having a strong data analyst team.

If all you have them doing is building mega-dashboards which mash up and squeeze in hundreds of different figures, you’re wasting their time. Every second is vital and the opportunity cost of that time sink is the inability to get focused thinking time to properly analyse the real data at their disposal.

What gets measured, gets managed.

But does it really? Basecamp CEO Jason Fried spoke about his views on this oft-quoted business mantra when he appeared on Tim Ferriss’s podcast (episode 329).

Jason suggested measuring less. He tests by removing elements of what he does measure then sees if it makes any difference to their business performance.

Measuring less didn’t have a negative impact on Basecamp’s success over the years so maybe Jason was really on to something. (If only the same could be said for other policies he and DHH subsequently embarked upon…)

I advise business people to really focus on no more than five key metrics if they really do have to measure their KPIs. I mentioned a senior exec in another article who was only interested in one key metric. At her level that was all of the management info she needed to see how her division was working. I balked at the time when I heard the story but understand it better now.

Less is more. It’s OK to not look at everything all of the time lest we get snowblind. Just like Benn Stancil says.

I’ve seen this from both sides.

I’ve been a data analyst for many years. And I’ve worked very closely with business stakeholders to try and understand exactly what they think they will get from their latest round of MI report requests.

Conversely, I’ve ran businesses where it’s easy to get caught up obsessing over small details. There is always the anxiety that you’re missing something important, if you aren’t proactively measuring EVERYTHING, every second of the day.

Real-time reporting feels like complete control but it comes at a price. You gain a close-up view but lose the big picture.

There has to be a trade-off.

As an analytics manager, it’s your job to talk the business side down from the idea that they NEED to see everything all at once and all on the one page.

They don’t. They need to focus on what really moves the dial for their area of the business.

As an analyst, you need the space and time to dig into the data and ask it better questions. You’re wasting everyone’s time if you’re on a constant treadmill to deliver more reports that will rarely get looked at. And no-one needs that realisation.

If you’re in the business owner’s chair you need to take a deep breath and take the advice I give when working as your data consultant. Focus. Don’t spread your attention too thin. And use the data to lead you to a place where it’s telling you a story, not just showing you more and more numbers on a screen.

Getting to that stage needs the ever useful help of the data detectives in your analyst team. But you’ll never know if you keep them tied up on those dashboards and comfort blankets all week instead.

Free them. Utilise them.

Or, you will inevitably lose the best of them.

The choice is yours.

(Photo by Marek Szturc on Unsplash)

Becoming An Analytics Manager Isn’t A Promotion. It’s A Career Change.

2024-01-23T00:00:00Z

So you want to be a rock n’ roll star?

Becoming a data scientist may be the modern tech version of becoming a rock n’ roll star but no-one really seems to be talking about what happens a few years further into your career.

Analysing big data sets. Building models. Connecting data pipelines. The challenges never end.

But what happens if it’s all starting to get a bit too routine and you start to hear whispers that you should want to move on to the next step on the career ladder?

The traditional move in tech has usually been to start eyeing up a move to management. And in the analytics world it’s no different.

Maybe your own manager has moved on leaving a vacancy. Or your company mentor (you do have a mentor, don’t you?) has suggested it’s time to push yourself on to the “next level”.

The only way is up. Or is it?

In many corporate structures, the data science career progression isn’t differentiated from that in the rest of the company. You’ll go through some process from junior analyst to analyst to senior analyst and then hit the management hurdle.

The big problem there is that moving up through the ranks in any technical role should not necessarily mean that management is the ultimate end point.

Once upon a time, I too made the move from senior data analyst to analytics manager, so trust me when I say:

Becoming an analytics manager is not a promotion.

It is a career change.

Pure and simple. Except moving from an analyst, or data scientist, technical lane straight into team management is not simple. Not even a little bit.

And this is what we’re going to explore.

Where to begin?

It took me over six years from entering the management world to really start to dig into what makes the jump so difficult for so many high achieving technical analysts.

At the root of so many of these people’s introductions to management is what is known as “the Peter Principle”. The underlying premise is that you are judged to be the best candidate for promotion to another role based on your high performance in your current role.

The problem with the Peter Principle is that your upward ascension in the company org chart will only ever stop when you get promoted into a role that you no longer excel at. In short, managers rise to the level of their incompetence. I can sense a million light bulbs going on in heads all over the world as I write that.

All of those totally incompetent middle managers you have encountered in your career suddenly shoot into your mind. And all those times you questioned how they got into that position suddenly get answered.

They were good at something else and, when they got into this job, their bosses found they weren’t as good at it and left them there. And now it all makes perfect sense.

But that won’t happen to us, will it?

Well, what does your personal experience tell you? Invariably…it does.

Why would you want to move into management in the first place?

Hitting the glass ceiling.

Maybe the reason you have decided to try your hand at management is that you are worried about hitting the glass ceiling as an analyst. There is no doubt that technical roles usually have a shelf-life when it comes to suitable rises in compensation. It’s a very rare company that allows it’s technical staff to maintain their individual contributor status and ride that pony all the way up the promotion highway to the boardroom.

If the only reason for moving into management from an analyst role is to get more money then you may need to re-think your strategy. For data scientists in large urban centres, it’s questionable whether their managers are making significantly more than they are in the first place.

Given the supposed shortage of quality, experienced analysts and data scientists, there is only one way the compensation for actually doing the do is going – and that’s up.

It’s OK to be average.

I may be alone in thinking this but there’s also nothing wrong with finding a role you enjoy and sticking with it. A quick look around any modern cubicle farm will show up plenty of analysts with 15 or 20 years on the clock that are more than happy in their comfortable rut.

You don’t have to be pulling up trees all the way through your career if the drive to do that isn’t in you. Sometimes just doing good (if not outstanding) work, getting pretty well paid and going home at 5pm is good enough.

If this sounds like you then know that there is plenty of room for you in the corporate analytics world. Don’t go making yourself miserable chasing a promotion to a job you may well hate.

No back room boys in the boardroom.

It has to be said that in many companies there is still a widespread (if covered-up) belief amongst senior execs that technical staff don’t have the potential to keep pushing on up, regardless of their performance as managers.

I’m minded of a story I was once told about a senior exec doing end of year performance reviews on his team. The department was rather varied in it’s remit, so it included marketing, comms and several other large teams, including analytics.

The analytics team had had a particularly fruitful year. High praise all round for meeting and exceeding business expectations and completing a multitude of successful projects. When it came to score the analytics managers however they all got bumped down a performance grade.

He felt that they were all “back room boys” and, as such, were little more than button pushers, regardless of how highly they were thought of throughout the company.

It’s a cautionary tale that getting onto the management ladder won’t magically give you access to the directors’ private bathroom or the top floor offices.

There are still a lot of high level execs in well-known companies who don’t understand nor trust what we do. Thankfully that’s changing but this anecdote shows there is still a long way to go. Just beware is all I am saying. The grass is not always greener.

Leave your current skills at the door. You won’t be needing them here.

Management is clearly not for everyone. It’s definitely not a natural extension to the analytical role you’ve been doing (and doing well) so far. Once you have made the jump, you’ll probably find the main problem with making the transition is having to learn a whole new skillset.

Analysts do not always have the most well developed inter-personal skills for a start. It’s a stereotype but experience tells me it’s somewhat rooted in reality (sorry everyone I’ve ever worked with – I’m putting myself at the front of this list!)

Learning how to conduct one-to-ones, performance reviews, manage other people’s workstack priorities and get dragged into upper level office politics are not what you’ve been used to. And it’s a steep learning curve for someone used to getting high grades at their own end of year reviews so far.

Playing an interchangeable role of peacemaker, shrink, priest, benevolent dictator and all of the other day-to-day roles a people manager has to employ can take it’s toll. Especially on someone more used to slapping on the noise-cancelling headphones and diving headlong into analysing a set of data.

Learning to let go.

Perhaps the hardest part of moving from an analyst role to analytics manager is the realisation that you aren’t supposed to be doing the analyst work any more. I found it extremely difficult to let this one go especially when a juicy interesting project came our way.

I like nothing better than rolling up my sleeves and turning data detective. Exploratory data analysis is my bag and when I had to do the scoping and requirements part then hand it off to others, I struggled.

If you find yourself wishing you were the one doing the work rather than managing others doing it, maybe it’s a sign that you’ve taken a wrong turn on the career path.

Drip founder Rob Walling sums it up very well:

“If you find yourself in management and wish you were coding, don’t wait until you’re completely burned out and bitter; do yourself (and your company) a favor – and do it now. Get yourself back into the code jockey seat. If your current company won’t work with you, there are companies out there who will.”

Life really is too short to do something you hate. And there is no shame in slipping back into your analyst role if you find the admin and people management too far from what you want to be doing.

So why did I move from quarterback to coach?

I moved into management for a few very specific reasons.

I had a second child on the way being the main one. Kids are expensive and there is no greater motivation to grab more money than having some small mouths to feed and clothe.

I had been a contractor and then consultant with the company for the previous five and a half years as well which played a large part in it. I knew the team I would be managing and I had (and still do) have an excellent relationship with my manager. There is nothing wrong with a little stability at certain times in your life and I figured our impending arrival heralded one of those times.

The fact that our consulting gig was ending and the offer of a full-time job was coming at just the right time may have also factored into it as well…

I certainly never wanted to maintain a steady charge up the ladder once I’d placed my foot on the first rung. Dealing with many different characters at senior levels and seeing how their world is so detached from the day-to-day operations of the business has only reinforced that.

The choice should be YOURS and yours alone.

Your motivations for considering a move into management will be your own but my ultimate advice is to make sure that you are really doing it for your own reasons. Not because you are being press ganged into a move you don’t want because it’s expected of you.

If the change of direction is being forced on you because you’ve been told it’ll reflect badly on you if you pass up the opportunity, you’ll want to think about what that tells you about the culture of the place you’re working in.

Whatever your personal decision is, just know that it doesn’t have to be forever. I know some data analysts that have made the jump to management and never looked back. They are adored by their teams and they have adapted themselves to suit the demands of the new role.

Others become stiff robots, unable to bring any empathy to their man-management style and end up making their analysts’ lives hell for no real end benefit.

For those who can’t give up the technical aspects of the job, they have two choices. Either admit it’s been a mistake and get back to what they were good at in the first place or carve out a compromise for themselves.

I always made sure I had some hands-on coding and analysis work for myself. It made the brain drain of bureaucracy, admin and politics slightly more bearable.

And of course I knew the work would be of a high quality. I mean, why else did I get promoted in the first place? 😀

Let’s hear your stories.

Drop me a note with some of your own tales of moving from an analyst role into management and how it worked out for you. Knowing where the pitfalls lie and the possible successes you can achieve can be really helpful to anyone coming behind.

Plus I’m nosey and like to hear other people’s stories.

Interested in the opposite move, from management back to IC?

This is Part 1 of a 4 part series, detailing:

the trials and tribulations of moving from being a data IC to data manager
why we might think about going back to Individual Contributor again
what could go very wrong with that move.
and, ultimately, what could make it a very good decision for us, and for those we work for.

(Main Photo by rawpixel on Unsplash)

Q&A with David Napoli, Healthcare Analytics & Data Viz Consultant (and Recovering Rocket Scientist)

2019-11-18T00:00:00Z

Interviewed on 18th November 2019

If you ever heard an exasperated teacher tell you the concept you are failing to grasp “isn’t rocket science”, you’d be forgiven for rolling your eyes at the trite cliché. Not so with today’s interviewee David Napoli. He actually knows exactly what rocket science is as a former aerospace engineer.

Now, as a data consultant and highly respected member of the teaching faculty at General Assembly, David uses all of his analytics experience to help create a new generation of data analysts and storytellers.

Welcome to Analysts Assemble, David Napoli.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

My data journey could probably be described as “unique”, having started nearly 25 years ago as an aerospace engineer – more specifically a propulsion subsystems engineer – performing modeling of second stage propulsion systems by coding in Fortran … with a little Excel modeling thrown in here and there.

I bounced from project to project for a number of years as funding was lost, and after the eighth time of that happening, I decided to give up on that dream (I had – and still have – an interest in the space field since I was 10) and began a search for a new career path.

Well, that was roughly 20 years ago, where I was fortunate enough to be offered the opportunity to make the leap from “rocket scientist” to Health Care Actuary … and that journey has since taken me through several other roles, such as Lead Decision Support Analyst, Lead Statistician, Research Manager, and more recently, Director of Business Intelligence.

As a Director, it was my responsibility to set the foundation for Data Literacy across the organization, as well as develop and implement the roadmap for Data Maturity for both my team and the organization as a whole to follow.

Besides implementing a Data Analytics Maturity Model for the organization to use to monitor and evaluate its analytic development over time, I also instituted a Business Intelligence Competency Center (BICC) and a Data Governance Committee (DGC) to support the data literacy efforts.

The BICC was established to provide a platform for analytic staff to convene and share knowledge and insights, as well as institute a formal setting to develop data lineage, data dictionaries, and policies and procedures to ensure repeatable processes were established.

The DGC then formalized oversight and direction at the executive level, and provided the pathway to establish formal data literacy trainings, which I developed and led, to ensure the organization would grow in how to use information, how to ask meaningful questions of data, how to read visualizations, and more.

I have since moved on from the Director role to a new stage of my career journey, which I am hopeful will continue for many years to come, and blossom for both myself and others … but we’ll get to that in the next question.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

For the past several years, I no longer have a typical day. Since I left my last full-time role, I have branched into part-time independent consulting and part-time teaching.

My consulting efforts span the entirety of the analytics space, from data management and governance – establishing ETL processes, MDM, data models, and warehouses – to developing statistical and predictive models – IBNR estimations and risk adjustment methods, for example – to creating visual exploratory analytics, dashboards, and more elaborate visual storytelling of programs and initiatives.

Where my true career passion lies, however, is in teaching.

I came across this career path somewhat serendipitously roughly four years ago, during essentially a hallway conversation at a consulting gig at the time. The individual I was conversing with was an occasional speaker at General Assembly, an organization focused on education and career transformation, specializing in the areas of UX & Design, Marketing, Career Development, Coding, and, of course, Data.

The local (Denver) organization was looking for instructors in the area of Data Analytics, as they wished to expand their offerings, and this individual said they would introduce me to an Instructor Manager there if I were interested.

While I had developed and hosted several “lunch and learn” sessions related to analytics over the years at several organizations, I had not given serious thought to teaching as a career pathway, despite how much I had enjoyed those previous brown bag sessions I hosted.

So, needless to say, I enthusiastically said yes.

I ended up speaking with the local Instructor Manager the following week, and was offered the opportunity to teach an evening “Introduction to Data Analytics” class a few weeks later … which went better than I could have ever anticipated. Being able to share my passion for data analytics and visualization with others, and see their own excitement in learning and growth in their analytics capabilities was beyond rewarding.

The feedback I received from that first class was overwhelmingly positive, which led to a second class, and then a third, and… well, now four years in, I have been selected as a member of General Assembly’s Distinguished Faculty program, where I lead the 10-Week Data Analytics course instruction, and also host classes in Data Visualization, Data Storytelling, as well as a concept I am currently developing called Data Dexterity.

As these teaching roles are only part-time, I do what I can to “fill in the gaps” with consulting efforts I am able to secure.

I have used a wide range of tools during my analytics career. My current courses focus on Excel, SQL, and Tableau. I have been an Excel user for almost 30 years, so I know that tool inside and out. SQL is not too far behind in my use, at around 25 years.

Tableau is much newer to me by comparison, having put that in place for an organization a bit over six years ago. I also use Power BI – having used this tool since its first incarnation called Power View when it was originally available in Excel 2010 – and have explored Qlik and Looker.

Coming from a statistical background (I have all but my dissertation in Health Services Research, with a focus on Biostatistics), I also work in SAS and R. Once I am able to carve out some time, Python is on my list to learn.

I am always keeping up on the latest analytics and visualization tools to be aware of and explore, and there is certainly no shortage of them!

You have an excellent daily analytics newsletter that rounds up a wide range of stories from around the data world.

You are also very active on Twitter. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I can not stress any more strongly how important it is to share with others your thoughts, ideas, and passions.

The whole concept of “if you and I exchange a dollar, we each have a dollar, but if you and I exchange an idea, we now both have two ideas” is absolutely true.

Establishing relationships with others, through sharing and learning together, begets further understanding of the topics at hand by both parties, fosters a deeper passion for learning and sharing, and often will spark interest and excitement in others who initially are only peripherally involved so they join in on the dialogue and continue their learning and further the knowledge sharing.

The number of valuable perspectives in the fields of data analytics and visualization is vast, and one of the hopes of my newsletter is to offer others a source which provides a steady stream of those different perspectives, providing a breadth and depth of knowledge and experiences to learn and be inspired from.

I will never stop being curious on what else there is out there being discussed, explored, evaluated, and communicated in data analytics and visualization.

What can we be doing better which could lead to better insights to help humanity?
What methods from other fields have I yet to be exposed to which could prove fruitful to efforts I am or will be undertaking?
What concepts and methods do others feel are foundational I should then incorporate in my teachings so others can grow above and beyond my own standing?

The list of such reasons could go on and on, but it is all focused on how much data analytics and visualization is a passion of mine, and I will do all I can to share my passion and understandings with others, with hopes of sparking a shared enthusiasm in others and finding those who wish to join me in this intellectual journey.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

As I had already been in management roles for 10+ years and chose to “move on”, I wish to continue my current path and further my efforts in teaching, and promoting data analytics, governance, visualization, ethics, and the human element.

I do this through my newsletter, giving workshops, speaking at events, and hopefully soon, establishing my own website (I’ve had the URL for around a year … I just need to figure out the setup and maintenance concerns and carve out the time to get it up and running) and finally begin writing a book.

As the concept of “uncertainty is information” has been a focal point of mine during my career the past 20 years, I had originally ruminated on putting together a book to explore and explain methods of visually displaying uncertainty in health care information.

Since that original concept came to life in my mind, I have reconsidered the topic and focus of the book, and while I still wish to include clarification around the methods of displaying uncertainty, I am now leaning towards a more broad publication, one in which I step through the tools and concepts which assisted me in growing and advancing from entry-level analyst to lead and beyond.

I hope others would find value in the details and learnings of such a journey.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

Anything I find valuable in data analytics and visualization spaces, I share through my newsletter as well as in the classes I teach. There are far, FAR too many to choose from to list here … if I attempted to cull my list down, I would be here until the Knicks are finally good again (two winning seasons since my kiddo was born … who is now a sophomore in high school).

I will mention one book which not many in the data analytics space may be aware of, and this is “The Book: Playing the Percentages in Baseball”, by Andrew Dolphin, Mitchel Lichtman, and Tom Tango.

Although the topic is obviously sabermetrics – one of my side analytical passions – the methods explored and explained in this book are widely applicable across industries and the analytical lessons are invaluable to all in analytical fields.

What is the number one piece of advice you give to aspiring data analysts?

Be intellectually curious and analytically skeptical … OK, I know, that’s two, but they are two sides of the same coin.

There are certainly more elements I would want to touch on – data ethics, data visualization, and data humanism immediately come to mind – but those are for another time and lengthy discussion.

Where can readers find you online?

You can find me on LinkedIn and Twitter, through my newsletter, and my General Assembly instructor page.

Q&A with Manpreet Singh, Analytics Lead and Forecasting Specialist

2019-11-10T00:00:00Z

Interviewed on 10th November 2019

Back in early 2012, the UK-based bank I was working for decided to begin outsourcing a significant portion of it’s technology and analytics capability to India. I was contracting with them as a senior data analyst under the umbrella of a BI and marketing strategy company I’d founded the year before with two other colleagues.

Seeing as the economy in the UK and Ireland was still feeling major repercussions from the Global Financial Crash in 2008, we were somewhat concerned about what that would mean for our own immediate futures.

We needn’t have worried. What we got instead over the years was an introduction to a long list of excellent analysts who helped us all grow as data professionals.

This week’s guest was one of the first of these new India-based analysts to join our team. And it’s been great for me to catch up with him and see what he’s been doing since we last worked together around six years ago.

Welcome to Analysts Assemble, Manpreet Singh.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

Always being curious about mathematical tools and the way they work during school days and later on wanting to dig deeper into its real life applications going forward, I got drawn towards studying Economics in my graduate and post-graduate level education. There I enjoyed econometrics, mathematical economics, and statistical analysis.

Although I graduated from one of the best places for economics in India (First Economic Research of Country), there were no immediate placements allowed (as the Institute wanted everyone to go for a PhD). I still got lucky to start as a research assistant in another great research institute that had state of the art facilities and computing labs to get me started.

Back in those days basic data management in SAS and knowing proc logistic could land you with a nice office job in MNCs ,so I focused on learning those few then niche skills and before I was an expert I already had a job with GENPACT working for premier GE clients.

I think a good data scientist will need to have 3 very basic skills:

First, he should know how to handle large volumes of data using SQL or another querying tool.
Second, he should be well versed with Statistical Techniques.
Thirdly, he should be able to present results with fine graphics.

These three basics of analytics have been constant throughout, its just that the tools are always evolving and changing with time.

Back in early 2000’s it was SAS, VBA and SQL. Today there are open source tools like PYTHON, R (Opensource), Cloud, Tableau, Qlikview etc. for visualization.

At Google we have all sorts of inhouse tools like Google Trix and Google Docs. Google has it’s own version of every tool you can think of.

Having traveled this long journey of data analytics, I have worked across domains and for some of the best of businesses. Companies ranging from GE, RBS, Whole Foods to Google, and across domains including Industrial Manufacturing Analytics, Retail, Digital Marketing and BFSI.

This has helped me to envision how data flows from a ground level sensor to a SCADA system in manufacturing. Or from a POS transaction to weekly targeted marketing campaign. It shows how these databases can be married to create a single source of truth to answer all customer questions in every business.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

I am heading India Analytics for a global major in Renewables Energy headquartered in Singapore. My role is a rich mix of Vendor Management, Strategic Decision making and delivering on top leadership expectations (Global CTO & India Head).

My work week starts with taking a stock of things on timelines and then working with the vendor partners throughout the week to deliver on committed timelines. That means over other routine things keeping a watch on sudden surprises coming our way that may impact the team works and keeping them in check.

I’m mostly interacting with my team at remote site locations: data scientists in Singapore and some remote vendor teams in India working on various new, newer and upcoming tools and techniques.

These include Python, to cloud to IOT data handling from remote wind turbines, solar panals and generators in thermal plants.

You’ve been active sharing your progress on LinkedIn. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

LinkedIn can be leveraged to share high level details ensuring confidential client and business information is not compromised.

The right people can always track you and understand that you are the right fit for their requirements if you share information smartly.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

After a level (10-15 yrs) you are bound to get into senior/key roles as the organizations can always find resources to do your role at 1/10th the cost. Also the experience you have gathered over years helps you to do justice to the strategic roles you get into.

However, you need to keep pace with the changing technology in industry and always keep yourself abreast with the latest.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

What is the number one piece of advice you give to aspiring data analysts?

Everything is open source so there is no secret.

The key is to stick with something over time and trying to do it again and again till you master it. It takes about 10,000 hours to master a skill and takes about 10 years of time to put that kind of effort.

There is nothing like right or wrong idea in technology and business, just focus on the present and do the things right and optimally. There are times when things go south when you need to do even better, that is how you get there eventually.

Where can readers find you online?

Yes, I have my web footprint. I run my page called Hadooping on Facebook with around 1000+ followers and I also keep posting on LinkedIn at times.

Q&A with Wesley Engers, Data Science, Math and Statistical Consultant

2019-10-19T00:00:00Z

Interviewed on 19th October 2019

It’s easy to get drawn into the misapprehension that all good analytics work is being done by in-house teams at large companies like Google and Facebook.

There is a wide world of opportunities out there for freelancers and consultants to take their data skills out on the road and bring value to a wide range of clients. Which is what today’s guest, Wesley Engers, has been doing.

Wesley is a long time top-rated data science consultant on Upwork, a teacher and tutor on maths and stats and now runs his own analytics consultancy, Mentor Analytics.

I find out more about Wesley’s journey along this career path and his tips for success for those interested in following a freelance data career.

Welcome to Analysts Assemble, Wesley Engers.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

I’ve always been mathematically and analytically minded. I’m very curious about the world and finding ways to systematically analyze and understand it.

I studied math in college and for my graduate studies but my focus was always on applied math. Hence my studies also focused on finance and physics and learning the math was really a key step in helping understand the world around me.

After graduate school I went to work at Symantec (Fortune 500 in cybersecurity) and did Six Sigma, statistics, and analytics work for them. I started part-time freelancing in graduate school and have continue ever since.

Doing freelance work part-time really allowed me to springboard into it full-time once I’d built up the requisite income from freelancing to allow me to quit my full-time job.

During the first few years out of grad school, I really focused on building up my skills both through my full-time job and on my own (through online Courses at Coursera for example).

These days I freelance full-time and often taking on new projects with new clients forces you to learn new techniques and approaches to their particular problems.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

I typically work from my laptop at my desk at home. These past few years I’ve mostly been a lone wolf but am really looking forward to working with more teammates at Mentor Analytics.

I don’t think I have a typical day but often I will sleep in until 9 or 10am and get a bit of consulting work done in the late morning/early afternoon.

Then, go to lunch, probably tutor some math/stats, go to the gym, get dinner, and possibly put in another hour or 2 of work around 10pm. Often I try to put in 2-4 hours of work per day.

My primary language is R but also do a fair bit of output in Excel as that is a more business friendly program. Many people who are not data scientist/analysts or generally people with less technical knowledge will work in Excel.

Hence, it is important to understand the language of business and be able to communicate with clients effectively.

I love R for it’s open source nature and the ability to find libraries on whatever algorithm I need for a current project.

I found you via an interview you did on the Fusion Analytics World blog and you’ve also been featured as a top-rated consultant on Upwork.

How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think it depends on the individual and their goals. I think you can have a very successful data scientist/data professional that doesn’t publicly share their work.

Now, they probably will need visibility within their organization if they want career advancement, but they don’t need to share with the broader public.

I do believe there are benefits to sharing your knowledge and abilities as teaching is one of the best ways to learn. Additionally, you get some exposure and a chance that others in need of your skills will find out about you. So, I’d says it’s helpful but not essential.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role now that you are running an analytics agency at Mentor Analytics?

I mostly see myself moving more towards a management type role although I’m sure I’ll still have heavy involvement with the actual analytics at Mentor Analytics.

For tackling bigger data problems having a team of specialists is needed. There is too much for any one person to know all the technical details to being able to bring together a team is key for doing an end-to-end data analytics pipeline.

End-to-end in this case would be from gathering and collecting data, to data processing/cleaning, to the actual analytic models and algorithms, and finally the reports/dashboards/tools that give the results and interpretations.

I’ll be looking to manage a team that can do all of those steps effectively.

Freelancing in any field brings its own set of hurdles to overcome before you get sitting down to do the actual work. From marketing to book-keeping to basic office admin. It’s not all coding and data viz, that’s for sure.

Would you recommend the freelance data science route to people considering it as a positive career move? How can they set themselves up for success?

There are definitely a lot of other aspects besides coding and the actual data science work that goes on when you are a freelancer. Freelancing can be great for some people and not so great for others.

You definitely need to find methods that keep you disciplined and on track to meeting your goals and deadlines. That’s very difficult for some people.

I personally, have no plans to change career paths and intend to make freelancing/self-employment my long-term career path. I enjoy the flexibility that is brings me and how creative I can be with all the new client projects.

Risk management is often key in getting set up for success. If someone is currently working a full-time job and looking to get into freelancing I would suggest starting part-time and getting some experience. It takes some time to build up a good clientele and find your particular niche.

Since Data Science is such a broad field figuring out where you can provide the most value and be most efficient with your time is going to be key. For example, I focus on data analytics and business decision making.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

I personally got started using Coursera Specializations to really get a handle on the coding side of things. I completed 2 specializations (Johns Hopkins 10 courses in Data Science) and University of Washington (4 Courses in Data Science).

Overall, I’ve also completed around 30 courses on Coursera related to Data Science and Statistics.

There are many online classes from Coursera, Udemy, Udacity, Edx, etc. that can help get you started.

What is the number one piece of advice you give to aspiring data scientists?

Never stop learning.

Data science is a big and constantly evolving field so you need to keep up with all the developments that you can. By constantly learning you’ll not only develop key knowledge but also show that you take initiative and are highly curious.

Curiosity and persistence are going to be essential for any data project that you take on.

Where can readers find you online?

LinkedIn Profile: https://www.linkedin.com/in/wengers/
Upwork Profile: https://www.upwork.com/fl/wesleyengers
I’ll occasionally answer some questions on Quora: https://www.quora.com/profile/Wesley-Engers
And of course, my website: https://www.mentor-analytics.com/

WTF Is Data Quality?

2019-10-16T00:00:00Z

Setting the scene.

Your business data doesn’t live on Quality Street, does it? It lives somewhere a bit skeezier.

Somewhere a little rougher around the edges. The kind of place you drive through faster at night and don’t stop at traffic lights.

Your data lives in Garbage Grove. Rubbish Road. Dumpster Drive. (OK, you get the picture).

Yet every day your business just pumps more and more of it in there. It’s not a data lake or even a data swamp.

It’s a data sewer.

And the stench of it is coming back to haunt you and your business every time you try to use it. Paint quite a picture don’t I?

Boring dry definition alert.

Anyone interested in how Wikipedia describes data quality? Didn’t think so.

But we’re going to check it out anyway and see how it ties in with what we think we know.

Wikipedia says:

“Data quality refers to the state of qualitative or quantitative pieces of information.”

What does that even mean?

Let’s try again. Wikipedia also says:

“The state of completeness, conformity, consistency, timeliness, duplication, integrity, and accuracy that makes data appropriate for a specific use.”

(They lifted that one from the Government of British Columbia).

It is a bit of a mouthful but it does make a little more practical sense. Each element of the sentence can actually be broken down to make a judgement call on whether the data sitting in front of us gets a tick or a big cross beside it for that part.

Let’s ask the data management professionals.

DAMA UK, the organisation for data management professionals, produced a white paper on the six key dimensions of data quality.

(Their paper quotes Socrates who said, “The beginning of wisdom is the definition of terms.” Now if that isn’t a mission statement for my “WTF Is…” series then I don’t know what is!)

DAMA put this paper together so it would operate as a checklist for data professionals when assessing the level of data quality in their own organisation.

Let’s look at each one of DAMA’s six dimensions in turn.

Completeness.

The potential of stored data against “100% complete”.

It does allow for a judgement call on what “100% complete” means though. Are all data elements mandatory or critical to our data collection and business requirements?

If the purpose of our CRM is to store customer details for future marketing but we don’t collect their email address, would we count that data as complete? What if we collected first name and surname but not middle name?

Uniqueness.

This means nothing is recorded in our database more than once based upon how it is identified.

I spent FAR too many years dealing with the repercussions of customer databases that contained multiple customer-level records for the same person. Each person should be in your Customer table once.

Any duplication of customer records (with different first names / customer identifiers etc.) means LOW data quality.

Timeliness.

Is the information from our data available when it is needed?

If a customer changes their address in one of your stores but the change isn’t made until several days later on the IT system, you are likely to fail this test.

Even more so if that customer gets a marketing letter sent to their old address in the meantime.

Not everything needs to be available for real-time reporting however. Sometimes an overnight batch update will be sufficient to meet the timeliness criteria for a particular data field.

(In business analytics, there are VERY few occasions when you really need real-time data - this is a hill I am prepared to die on.)

Validity.

Data are valid if it conforms to the syntax of its definition.

Huh? In other words, make sure it fits the type and format of what you expect it to.

A date can’t be on the 35th of any month. A phone number shouldn’t have characters in it (555-BETTER-CALL-SAUL doesn’t count). If your credit card number should have 16 digits then a 14 digit number is invalid.

Accuracy.

The degree to which data correctly describes the “real world” object or event being described.

This is closely tied in with the Validity assessment.

If a phone number is not in a valid format then the data can never be accurate. If the phone number is in a valid format but one digit is incorrect then it meets the Valid criteria but still fails the Accuracy test.

Consistency.

Are there differences in the information shown by the same data point on different systems or data tables across your organisation?

Is Alan’s date of birth different on the Customer table than it is on the Staff table? Have you closed bank branches but your weekly sales report still shows new sales against that closed location?

Both can’t be right. You just failed your Consistency test.

Do we need to have all of these to make our data quality “good”?

No, it doesn’t really work like that. And it’s always dependent on the requirements set out by your business.

As we have seen above, data can pass on one test yet fail another or fail on several and pass another. You could have a customer’s email address record which is consistent across multiple tables but still fails both the Validity and Accuracy tests (e.g. alan@hylands - it has to have a dot and a domain extension at the end to be valid for a start.)

DAMA also make the fair point that data may pass all six dimension tests with flying colours but still fail a basic usability test.

If the data is all in English and your business is based in rural China then it’s still bad data quality regardless of how consistent, complete, accurate etc. etc. it may be.

So if data quality is essentially subjective, how do we know what high quality data looks like?

DAMA’s six tests can certainly help you begin to judge the quality of your data. But we have to take things out of the theoretical realm to really find out how much quality there is lurking beneath the hood of our data systems.

If you are building a data report or analysis, can you trust that the data you are using is both complete and correct? Only one way to find out. Build some reports and start to see what it looks like.

A worked example.

I gave an example above of a bank closing local branches but still seeing those branches appear in their weekly sales reporting for years afterwards. We’ll call this a “hypothetical” example.

There may be nothing wrong with the source system or the data entry of the sales clerks who input the customer details. It might legitimately be a quirk of the way each individual customer is tagged to a certain branch location when they become a customer. And that’s fine.

But how does that impact our data quality score when this branch location is used for sales reporting further down the line?

What are the implications of our low data quality?

It can erode confidence in the overall business intelligence being produced.
It can negatively impact the sales figures and potential bonuses for the branch staff who actually sold the new product.
And neither of those leads to positive results for your business.

Low data quality leads to multiple versions of the truth, conflicting reporting and increased support time and expense for the data factory who have to go around cleaning up the different messes.

And if it starts to impact the quality of reporting to government or regulatory bodies then it can become VERY costly indeed.

It sounds like good data quality is nearly impossible to achieve in the real world.

Simple response is: yes, it is very difficult to achieve especially across a large organisation.

It has to become part of a full life-cycle quality control approach from the entire business to have any chance of becoming successful in the longer term.

From ensuring user interfaces don’t allow staff or customers to input garbage to quality control measures and reporting being implemented at each stage of data production and storage. This can become VERY expensive and onerous VERY quickly.

Beware the bureaucrats.

Even worse, in some large corporates, is when an entire bureaucratic industry of pen-pushing is allowed to creep in under the well-meaning guise of “Data Management and Governance”.

There are countless examples of these departments being completely filled with jobsworths who have no idea about the actual data they are meant to be protecting and nurturing.

Instead they get caught up tinkering with data quality measurement schemes to use as sticks to beat people with and documenting data lineage that is nothing more than a paper-filling exercise.

Ultimately the data is no cleaner or of a higher quality after all of this but it certainly fills your day with meetings and talking shops if that’s your thing…

How can I improve my data quality?

I have foreshadowed above (in a very subtle fashion) what I don’t think you should do. But that will come as little surprise. I’m no fan of meaningless bureaucracy in any form.

To get it right, you will have to start off with as simple a plan as possible.

Audit your existing data and see where you stand on the DAMA 6 principles. You can then start to see where your main problem areas are.

For example:

Planning to implement an email marketing campaign but find your customer details are spread across three systems that don’t talk to each other? Consolidate, cleanse, de-dupe and match your customer records into one master Customer database.
Ran an analysis to find your customer base’s product penetration levels are well below industry norms? Maybe you have customers sitting under multiple customer identifiers which again need to be matched and consolidated under one record.
Found thousands of your business customers are tagged as sole traders and are availing of sole trader discounts but have the phrases Ltd or Limited in their company name? Run a simple query to identify them by name and get the tagging re-calibrated.

None of these solutions are rocket science but they are daily data quality occurrences for tens of thousands of businesses of all sizes all over the world.

I’ve dealt personally with all of them. You don’t have to go looking too hard to find them, that’s for sure.

How can it go so wrong?

It’s not always someone’s fault. Sometimes the best of intentions to fix the situation can backfire and cause more data quality issues down the line.

Let’s say a bricks and mortar business collects customer data when they are buying a new consumer good in-store. And the business now wants to collect email addresses for future (cost-effective) marketing campaigns.

So they get their IT team to change input validation rules and enforce an email address being entered for each new record on the sales system.

Not every customer wants to give over their email address. Some don’t even have one.

So the customer support staff on the frontline have to input a dummy email address for every one they don’t legitimately have. This wreaks absolute havoc in terms of email bouncebacks and poor marketing reach metrics for the Marketing team further down the line.

These decisions, day in, day out, have real, long-term implications for the quality of your business data. That’s why it needs a Total Quality Management approach to the full life-cycle of such decision making.

And it needs proper data people to be involved at all stages. People who understand the consequences of getting dirty, poor quality data through the pipes that can cost millions to clean up.

What should we do today to help improve the quality of our business data?

Sadly, it’s often easier (and quicker) to mess up a perfectly good quality data system than it is to tidy up a bad one. Such is life.

But that doesn’t mean we throw our hands up and just let our (data) world burn.

Taking simple steps to audit, monitor, clean and revitalise our data can have almost miraculous-looking results if we concentrate our focus.

Basic action plan.

Get the right senior people onboard in your business to own the process and outcomes.
Get an experienced data professional to guide you through the data quality audit and cleansing process.
Don’t get gummed up in paper-chasing exercises that lead nowhere.

Just take action.

Do it now. Keep doing it. Then do it some more. Then rest a little. Then do it again.

Then it’s job done.

Only kidding, this job is never done. Not until the next time. And the time after that. And…

Happy cleaning!

Get it right and your analysts (and your bottom line) will appreciate it come next reporting time. Don’t bother and you only have yourself to blame.

Q&A with Oscar Baruffa, Sustainability Professional and Data Specialist

2019-10-14T00:00:00Z

Interviewed on 14th October 2019

My wife thinks I spend too much time on Twitter. I tell her that it’s not all football gossip and comedy memes but it’s a hard sell sometimes. What I mainly use Twitter for though is keeping up to date with what other data professionals are doing in their day-to-day work.

And that’s where I happened upon my latest interview guest. Oscar Baruffa is a Data Specialist at a non-profit based in Utrecht in The Netherlands. When he started this new job back in June 2019, Oscar tweeted a great stream of daily updates on his progress.

So I caught up with him this week to get a little more background and see how he approaches his own data career.

Welcome to Analysts Assemble, Oscar Baruffa.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

All my roles have had a strong element of interpreting data and making commercial decisions based on that information. I started out doing Mechanical Engineering, moving onto sustainability auditing, then running a national recycling programme through to my current role as a Data Specialist.

I realized the other day that my two most transferable skills are:

Interpreting and communicating data
Project management.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

At the moment I work 4 days in the office and 1 day remote in a medium-sized company. In my previous role I was fully remote for many years.

I am sort-of a lone data person in that I have a data colleague who is only part-time on my team and part-time on another.

My aim is to use R a lot more for some of the work that I do, and some will indeed need it especially for wrangling many spreadsheets. I use Excel a whole lot and some Tableau.

Tableau is actually great for doing “live” analysis with a non-technical audience because they can follow along and see how you’re building the analysis and ask questions on the fly.

It’s quite fun to do 😃.

You are very active on Twitter. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think you can have a long and rewarding career without any social media presence or sharing what you’ve learnt.

The benefit I see is that it certainly enriches my daily work by being aware of what current thinking and trends are out there and I learn quite a lot on a daily basis. I think it is perhaps most useful to learn and share publicly when you are a solo data professional and don’t have peers to bounce ideas off of.

LinkedIn is really good for keeping tabs on where people are moving to throughout their career. I think its worth keeping a decent profile maintained for that purpose.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I’m focusing on building my technical skill as I have spent quite a few years moving more to management-based role. I’d say probably 90% of my time in recent years has been management-focused.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

There’s quite a few but I think a commencement speech by Neil Gaiman some years back, titled “Make Good Art”, is really fantastic for anyone creating anything. Especially if you feel that there are already tonnes of blog posts/tutorials/books on a subject and that you have nothing to add.

It’s a really good perspective on why to create things and which projects and roles to choose in life.

What is the number one piece of advice you give to aspiring data scientists?

Remember that data is a tool for decision making, but don’t rely on it as the sole means to make decisions. When interacting with others (clients, customers, execs, peers), they will have a lot of considerations (data points 😃 ) that are hidden from you.

Your data and insights are a piece of the puzzle.

Where can readers find you online?

The best places these days are:

Twitter - @oscarbaruffa for more general conversation: https://twitter.com/OscarBaruffa
Newsletter for updates about my side projects in R: https://tinyletter.com/OscarBaruffa - the latest project is a book about Twitter for R programmers 😄
LinkedIn if you want to connect and keep in touch that way: https://linkedin.com/in/oscarbaruffa/
Youtube - my YouTube channel is called Other People’s RStats
My blog - OscarBaruffa.com

Comparing Two Values With A Dumbbell Plot In Python, Pandas and Matplotlib

2019-10-08T00:00:00Z

What is a Dumbbell Plot?

A Dumbbell Plot is a variation on the Lollipop chart and is often used as an alternative to the traditional clustered bar chart.

It’s made up of dot plots with two or more grouped series of data. The distance between the dots illustrates the difference between your two data points.

When would we use a Dumbbell Plot?

Simple answer is: when you get bored of looking at the same old clustered/grouped bar charts. I’m far from a visual artist when it comes to data visualization but even a dyed in the wool coder like me knows that you have to mix things up every now and again.

Real world examples of when to use the Dumbbell Plot are:

Comparing expected sales for a particular quarter against actual sales.
Checking event ticket bookings versus total capacity.

Let’s see an example.

I’m going to use Python to work up an example of a Dumbbell Plot. We’ll use the Pandas and Matplotlib libraries to help build up our dataframe and the Dumbbell Chart.

What’s our business problem?

Football fans tie themselves up in knots over their club’s league position at the end of a season. Judging whether a certain league position is “deserved” is very much a matter of subjective opinion. One measure we can use to work out an expected league position for each club is based on their annual wage bill.

Unlike American sports, there is no salary cap in football. Clubs can pay whatever they can afford to their playing staff. And we are going to use the English Premier League wage bill rankings to judge whether a certain club has been under or over-achieving for the past twenty years.

I am a Tottenham Hotpsur fan so we’ll use Tottenham as our example subject.

Where do we get the data?

The Telegraph and Guardian newspapers have ran annual articles rounding up the financial affairs of English Premier League teams. I used these as a secondary source while they often take their numbers straight from the clubs’ own reported accounts or the annual Deloitte Football Finance report.

What does the data look like?

We need a grouped series and two (or more) values to compare to each other for our Dumbbell Plot. Our series will be the season (named SEASON, in the format like 20002001 for the 2000/2001 season). And our two values will be a wage rankings position (WAGEPOS) and a league finish position (LGEPOS).

Let’s see some code.

We’ll be using the pandas library to create our dataframe to work from. And the matplotlib library from pyplot to chart the Dumbbell Plot.

import pandas as pd
import matplotlib.pyplot as plt

I saved the data off in a .csv file and we can read that into a dataframe using pandas .

df = pd.read_csv("thfcwageslgepos.csv")
df

Which gives us this dataframe:

We need to sort the values by SEASON to get them in the right order.

ordered_df = df.sort_values(by='SEASON')
my_range=range(1,len(df.index)+1)

The Python Graph Gallery has an excellent tutorial on this which I originally worked off to get the plot looking correct. I’ve changed some of the colours, columns and labels to suit our own data.

plt.hlines(y=my_range, xmin=ordered_df['WAGESPOS'], xmax=ordered_df['LGEPOS'], color='grey', alpha=0.4)
plt.scatter(ordered_df['WAGESPOS'], my_range, color='navy', alpha=1, label='WAGESPOS')
plt.scatter(ordered_df['LGEPOS'], my_range, color='gold', alpha=0.8 , label='LGEPOS')
plt.legend()

# Add title and axis names
plt.yticks(my_range, ordered_df['SEASON'])
plt.title("Comparison of the Wages Position and League Position", loc='left')
plt.xlabel('Value of the Positions')
plt.ylabel('Season')

And that’s it. This is what my Dumbbell Plot looks like:

What does the chart tell us?

Let’s put our business analytics hats on for a second.

We started out wanting to know if Tottenham had been punching above their wage bill weight with regards to their league finishes. And it’s pretty clear that they have.

From our Dumbbell Plot we want to see the GOLD dots further to the left of the NAVY BLUE dots. And that has been happening pretty consistently from at least the 2008/2009 season onwards.

The navy blue dots for Wage Ranking have been incredibly steady in sixth position for the best part of the last decade. This means that any league finish over or above 6th should be looked at as a win for Tottenham fans.

Are there other ways to chart this type of data?

Yes indeed there are. As I said at the top, we could use the more traditional clustered bar chart approach. Or even a line chart to show the same data.

This is a quick clustered bar chart I did with the same data in Excel:

Having the desired position (at least in league position terms) being value=1 makes aligning this chart a little more difficult. As a reader, it doesn’t convey the message to me as quickly as the Dumbbell Plot and that’s what really matters.

I like the Dumbbell Plot better as it displays the differences more clearly between expected and actual values. It’s well worth trying out on your business data to see how it looks for your own use case.

And of course it brightens up a slide deck of the same old bar charts if you fancy a change.

WTF Is Business Analytics?

2019-10-05T00:00:00Z

Tired of hearing how data science is the sexiest job of the 21st century? Yeah, me too.

What most of those puff-piece business mag filler articles didn’t point out was that analytics has been used to help manage business performance and strategy for decades.

So relentlessly tuning your AI bot model is great but have you ever gotten deep into predicting what your next quarter sales will look like based on changing market conditions? Getting inside your customers’ heads and knowing what they will do before they even do it?

That’s business analytics at work - right there in the wild.

Sounds like another buzzword. What exactly is Business Analytics?

Let’s try some dry and dusty official definitions of business analytics on for size.

Wikipedia:

“Business analytics (BA) refers to the skills, technologies, practices for continuous iterative exploration and investigation of past business performance to gain insight and drive business planning.”

Techopedia:

“Business analytics (BA) refers to all the methods and techniques that are used by an organization to measure performance. Business analytics are made up of statistical methods that can be applied to a specific project, process or product.”

Gotta love overly stuffy business-speak don’t you?

So what does that really mean?

It means using statistics on data from your business to make your business perform better. From identifying weak points in your existing processes to spotting trends and patterns in the data that help build plans for future strategy decisions.

Sounds a bit like what we call Business Intelligence?

Business Analytics is indeed related to Business Intelligence. Some sources call it a sub-set of BI and I tend to agree.

For me, traditional Business Intelligence is seen very much as backwards-facing. Regular and ad-hoc reporting on what has happened, what is happening now and where it happened in your business.

Business Analytics on the other hand looks more at why things happened and what may happen next in the future.

“BI” reporting will come at you in various forms such as basic queries and reports, dashboards, scorecards and the types of Excel-based pie charts we all know and love. It’s an integral part of bringing a data-informed ethos to your business.

I will not be dumping on the reporting aspect whatsoever as answering the question “how are we doing?” is vital at all times. It’s not the be-all and end-all for Business Intelligence though.

That’s where the Business Analytics section jumps in to bring the big value. Knowing what has happened is great. Knowing what to do next is the real challenge.

“I don’t need numbers to tell me what to do. I’ve been in this industry long enough to trust my gut.”

There has to be a middle ground between handing over the keys to the whole show to a bunch of algorithms and automated AI Skynet-like bots and ignoring all data to go totally down the gut-feel management road.

Subject matter expertise is one of the most overlooked aspects when young analysts look to move into pushing a data-driven agenda.

I prefer the phrase “data-informed” rather than “data-driven”.

It allows the business expertise built up over many years to help guide and augment the analytics outputs and direction of investigation.

Hand in hand, they really do become a power couple for the ages.

So how do we approach Business Analytics in our business?

There are several important elements to a Business Analytics strategy.

Data Collection, Cleansing and Aggregation

Without plenty of good quality (i.e. not absolute dirt) data your efforts will all be for nothing. From your in-house transaction logs data to augmented third party data, it all plays a part and can help fill in the gaps that will help you uncover what’s really happening under the hood of your business.

Get this step wrong though and you might as well write off the rest of your analytics work. Don’t underestimate this point or do so at your peril.

Data Mining

Put your machines to work and let them loose on your vast supply of data. Here we use statistical techniques such as clustering to segment our customer base, classification to bring groups of transaction types together or regression to work out how past performance might indicate future patterns of behaviour.

Text Mining

We can use techniques like Natural Language Processing (NLP) to gauge the sentiment and tone of our free-text data. Sentiment analysis on customer complaints, social media posts and app store reviews is a direct route into dragging value of these rich data sources.

Not everything sits in a nice, clean table in the data warehouse. Don’t waste the real gold hiding in the rivers of mud that are Facebook, Twitter and Instagram.

Forecasting

Just because something happened before doesn’t necessarily mean it’ll happen again. But there is still a hell of a lot of value in anticipating future events based on what your data says has happened in the past.

Energy companies have used forecasting to prepare for surges in demand at the mid-episode break of very popular TV soap operas (caused by viewers putting the kettle on). Ice cream sales at the beach go up in summer months and down in the winter.

I’ve been analysing some local police data on reported crimes this week. There is a clear seasonal bump in the number of crimes associated with anti-social behaviour over the summer months.

Insights like this can help the management prepare their resource plans for these times of increased demand. Remember, it doesn’t have to be a crystal ball to improve your odds of success.

Predictive Models

Forecasting helps us prepare for overall changes in trends. Predictive scoring models help us drill down to an individual customer or component and predict how they will act in future.

Risk-scoring customers who apply for a personal loan from their bank is an obvious example. Customer A has a poor credit record and patchy employment history. Customer B has an exemplary credit history and is a tenured university professor.

Risk modelling would consider Customer B a safer option and Customer A might be turned down altogether or given a higher interest rate to compensate for their riskier profile.

Optimization

When I studied Management as part of my Accounting degree, they were big into case studies about Kanban and Just In Time manufacturing techniques. Optimizing the flow of supplies into the production process at just the right time when they are needed is an art form.

This can apply to white-collar service industries just as much as car manufacturers and logistics companies. Using business analytics we can build models that tell us the optimal stock levels for certain products months in advance of them needing to be ordered. Or a Next Best Action suggestion for cross-selling other products to our existing customers.

Uber’s surge pricing is a good example (for Uber, if not the customer) as it takes demand into account when ramping up price to maximize the transaction value. Even if it’s distinctly unethical in my book.

Data Storytelling

Just like our data collection and cleansing phase, the whole thing is for nothing unless you get your storytelling just right.

50 page slide decks with the Encyclopedia Britannica squeezed onto them in tiny 5pt font.
Pie charts.
Donut charts.
You name it, it’s been used (badly) to convey a business story to the people at the top making the decisions.

And in many cases, failing to hit the mark.

Modern data viz tools like Tableau and Power BI give a lot more scope to generate eye catching, interactive, graphical content. But it still comes down to digging through the haystack to find the one needle of data-informed truth that will move the business on in the right way.

The ability to do that is rarer than rocking horse shit so consider it a proper superpower.

Business Analytics sounds like magic. Are there any downsides?

Business + Humans + Computers + Data. See any problem areas there?

In my experience, the biggest issue with getting full value out of your Business Analytics strategy is lack of real buy-in from the senior decision makers. They’ll talk a good game in front of people. They’ll maybe sponsor an analytics project or two.

But the will to really break with decades of past behaviour and let the data inform their own decision making is clearly a mental step too far.

How do we solve this?

Good question. Get senior allies onside as early in the process as possible. In large corporate environments, you are unlikely to get too far without a senior champion on the board or driving the sea-change in attitude that you will need for ultimate success.

In small to medium sized businesses, you need the person who writes the cheques to be the champion. Simple as that.

What about cost - both in investment and people?

You have to be committed to the long haul. By all means, start with a small Proof of Concept project (which is my chosen starting point actually) to build up confidence. But know that over time, this will grow and you need the infrastructure in place to support that over the long term.

That doesn’t mean fire a tonne of money at it, bring in every vendor going and expect overnight results. It means having a key advisor there at the start and at each stage of growth to ensure the roadmap is planned out and being followed.

Keeping that champion in high places happy and on-board is vital as well.

Implement from the ground up and across the business.

Building a successful Business Analytics function in your business means embedding it across the whole of the business. You can’t just have a data nerd sitting in a cupboard somewhere churning out reports and models. If you can’t get the business folk on the frontline to actually use any of it, you may as well not have bothered.

And if it turns into a turf war of ownership between IT and the analytics crew then it’s game over immediately.

It needs buy-in and cross-departmental collaboration to really work and that’s not going to be easy. But my word, it’s worth it if you get the thing right.

Sounds like a lot of hard work. Do we really need to do Business Analytics?

Business Analytics makes your business Harder, Better, Faster, Stronger (in the words of Daft Punk).

If you don’t do it then your competitors will. And even their half-assed implementation might be enough to knock you out of the market.

Do you really want to risk that by sticking to your gut feel/back of a cigarette packet “data” philosophy? Of course not.

Get on the Business Analytics bandwagon now. You won’t regret it.

Q&A with Alli Torban, Data Visualization Designer and host of the Data Viz Today podcast

2019-10-05T00:00:00Z

Interviewed on 5th October 2019

Seeing as this week’s Analyst in the spotlight has spent a considerable portion of her career working in both the FBI headquarters and the Pentagon, I’m going to be extra nice to her. Friends in high places and all that…

These days, Alli Torban is a highly regarded data visualization expert and host of the popular podcast, Data Viz Today. Alli takes us through her background as a government systems analyst and how her journey into motherhood also took her into the wild world of data visualization.

Welcome to Analysts Assemble, Alli Torban.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

I have a background in math and GIS, and I spent the first seven years of my career as a software tester and systems analyst for various government clients like the FBI and The Joint Chiefs of Staff in Washington, D.C.

I stumbled upon the world of data visualization while being a stay-at-home mom. I wanted to use my time off from my career as a chance to pivot into something that I was more passionate about, and data viz turned out to be it!

The snag was that I didn’t know the first thing about the data viz design process. So I decided that the best way to learn would be to start a podcast that documents the small but important decisions that go into the amazing visualization we see around us. That’s how Data Viz Today was born!

After nine months of producing the show and building a portfolio, I was hired as a data visualization designer by the American Enterprise Institute, which is a research think tank. The podcast is now 1.5 years old, and I’m still using it to constantly improve my craft!

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

I have a very unique schedule! My daughters are still very young, so I work part-time. Two days a week I commute into the office in Washington, D.C. to have meetings with scholars who want to create visualizations for their reports.

The other days, I work from home in the Virginia suburbs creating those visualizations. My primary tools right now are Tableau, R, Adobe Illustrator, and Excel. I’m embedded on the design team, which is only a handful of people.

You’re active on Twitter, host a very successful podcast with Data Viz Today and have created your own data viz courses. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think it’s really important to create and share for many reasons:

By sharing your work, you encourage others to engage with you. You can get invaluable feedback and make connections with other people in the field. It’s never too early to start building your network! You need connections and leads when you’re looking for a job.
Speaking of a job, creating and sharing your work allows prospective employers to see more about your personality and how you solve problems and express yourself.
By articulating your thought process and what you’ve learned, it forces you to really understand it. If you can’t explain something simply, then you probably don’t understand it well enough. When I put together a podcast episode about a specific topic, I have to go deep and anticipate all the questions my listener might have since it’s a one-way conversation. It’s an astounding learning tool!

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I really want to work on defining my unique design style. I hear from designers that your style chooses you, so I guess I just need to be patient and keep experimenting!

I also have “learn d3” on my to-list. I bought Amelia Wattenberger’s new book, and now I just need to make time to go through it!

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

I wish I had started learning data viz by working through Scott Berinato’s Good Charts Workbook. No matter what phase you’re in, you’ll find it helpful.

He patiently explains various concepts (like choosing color), works through a few examples, and then allows you to try. There’s nothing better than hands-on practice!

And if you like to practice using interesting datasets, I’d highly recommend subscribing to Jeremy Singer-Vine’s Data is Plural newsletter, which he sends out weekly with a handful of curious datasets that he’s come across.

TWhat is the number one piece of advice you give to aspiring data analysts?

The advice that I wish I had heard (and listened to!) sooner would be to just relax.

Your viz doesn’t need to do everything. It can’t be everything to everyone.

It needs to be focused on communicating one main goal. That doesn’t mean that it can’t be complex; it just needs to be focused.

Where can readers find you online?

You can find the Data Viz Today podcast on your favorite podcast app or my website! I’m also very active on Twitter 😃

Q&A with Michael Kaminsky, Writer, Data Strategist and Entrepreneur

2019-09-28T00:00:00Z

Interviewed on 28th September 2019

It’s hard to stand out as an online content creator these days. The web is awash with influencers, thought leaders, content marketers and what we used to just call bloggers. So it takes something pretty special to grab my attention.

But that is what happened when I read an article called The Analytics Engineer on the Locally Optimistic blog. The writer, Michael Kaminsky, nailed the job description at the intersection of data engineering, data analysis and data scientist which is something I’ve been trying to do myself for quite a while. And it’s no mean feat.

I liked his take so much I immediately emailed Michael to see if he’d do a Q&A for me. And, I’m very pleased to say, here it is.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

I got into data through economics and econometrics – I was lucky enough to have the opportunity to do research with Kerry Smith at Arizona State University as an undergrad where I learned how to use the statistical software package Stata to do econometrics research (I even was able to co-author a paper as a result of some of that research).

After undergrad, I joined a boutique economics consulting firm where I wrote a lot of SAS code to do research for pharmaceutical companies – that was a nice blend of academic research and industry. After a while, I realized that I really enjoyed the more technical aspects of the work.

I wanted to focus on developing more software engineering skills and becoming a “data scientist” (which, at the time, was still a very new term).

During your analytics career, how did a typical day look for you? Which tools and languages did you use? Big team / small team / lone wolf? Remote / office-based / co-working space?

I’ve had the pleasure of getting to do a lot of everything. There have been months of writing nothing but SAS code and building excel models, months of just R, and months of just python.

My favorite toolset is my Vim + Tmux setup which I use as my core IDE for whatever I’m working on – normally some mix of SQL, Python, and R.

I’ve generally always worked on small teams (3-6 people) as part of a larger organization. Historically always in-person, though last year I moved to Mexico City and have been working on building two startups from there, completely remotely.

Now you have transcended into a startup founder role with two startups, Bolster and Recast. How does that differ on a daily operational level and how do you find the skills you nurtured in analytics have crossed over to your new role?

To be honest, it’s hard to underscore the difference in the type of work that I’m doing.

Bolster is a digitally native brand that helps people with long-term care planning and involves almost exclusively marketing and operational tasks, and Recast is a data science product for marketers (imagine a new-and-improved media-mix-model).

What I’ve learned is that starting a new company takes way more skill in marketing and sales than I ever would have imagined. And sales is hard.

I spend a lot of time just talking to people trying to learn how they think about a product or a problem so that I can figure out how to position the product in a way that’s most appealing – I would have never guessed how much time I’d have to spend on marketing and sales when I was first nursing the idea of starting a startup a year ago.

The most important skills that cross over to building a startup are the critical thinking and numeracy skills I developed as a data scientist. When you’re starting something new you have to be super-disciplined when it comes to making decisions and evaluating the strategic opportunities in front of you.

One thing I’d like to expand on (and it might just be because it also applies to me at the minute) - how do you find splitting your attention between two startups when all of the conventional wisdom is to Focus Focus Focus?

I don’t have a great answer, and I actually have a lot of self-doubt about this. On the one hand, having more irons in the fire seems like a reasonable way to decrease risk and increase the chance of a “hit”.

However, as I’ve gotten deeper into these projects I’ve started brushing up against the point where I truly have too much work to do and the divided attention really does cost me something (and closes off some opportunities).

When I was just getting started, I had plenty of time (and a lot of the work was waiting on things to happen so even with both projects I had some light days / weeks). But now I’ve got a more-than-full plate and am having to make real trade-offs about where to spend my time.

I expect that if / when one of the projects gains significant traction I’ll have to put the other on the back-burner (or kill it altogether) – it remains to be seen whether or not I’ll have the discipline to do that when the time comes!

I’ll have a better answer for you on whether or not this strategy was a good one after another year and we’ll see where I am 😃

You are a writer and blogger and it was your article “The Analytics Engineer” that caught my eye when I first read it.

How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think writing regularly is one of the best activities you can perform to boost your career. Most importantly – to write well, you have to think well. The act of writing helps me clarify my thoughts on a topic, and personally I’ve observed that writing more has helped me clarify my thinking generally, not just when I have my hands on the keyboard.

The practice of clarifying your thoughts for a blog post or article has carry-over effects that make your thoughts clearer in other domains. I cannot encourage everyone enough to write more!

The second benefit is the professional network you can develop through sharing your writings – from this Q&A to the entire locally optimistic community, I feel super lucky to have gotten to meet so many super talented analytics professionals.

I’m 100% positive that my next full-time job will come through the locally optimistic community, and I wouldn’t have that if I wasn’t regularly sharing my thoughts through the blog and our Slack channel.

Where do you feel your own career will go next? Can you ever see a point where you will move back into analytics full-time?

Or do those analytical skills now just become one of the fundamental building blocks of making you a better overall entrepreneur?

Gosh I wish I knew the answer to this.

I really like being an entrepreneur, but I miss “earning money” and “not working every weekend”. The truth is that I can totally see myself going back to a full-time analytics role if none of these entrepreneurial projects work out – however, I believe that the last year has been such an incredible learning experience for me that I’ll be a much stronger analytics leader if and when I join another organization as an FTE.

The lessons I’ve learned about business strategy, product-building, and sales will surely be valuable in whatever organization I join next.

Your own career path has taken you through non-profits, fast-growing B2C businesses like Harry’s Grooming and now your own startups.

Do you think that made you a better analytics all-rounder rather than, say, sticking in one industry and diving deep into that specialisation?

I think it all depends on what you want to get out of your career. I really treasure variety and take a lot of pride in using expertise from other disciplines to bring unique insights into whatever I’m working on.

While this works super well for me, I know that I’m not as skilled in one discipline as lots of other people are – there are lots of things I can’t do (and professional trajectories that are closed to me) since I’ve taken this path.

I’m never going to be a great machine learning or algorithms engineer, and I’d be wayyy behind if I wanted to get back into academic-style research.

However, I think my broad knowledge base and general interest in pretty much everything is one of my super-powers, so I’ve leaned into it.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

Some things I do that I think more people should do:

Keep a diary and write in it every day. You’re your own best coach if you take the time to actually analyze your life!
Read a lot. Like … a lot a lot. I read the New Yorker cover-to-cover every week plus 30-50 books a year. Plus a steady diet of blogs and news sources everyday. That type of broad-based knowledge is invaluable.
Inbox zero every day
To-do list zero every day

What is the number one piece of advice you give to aspiring data professionals?

“Work to create as much value as you possibly can.”

There are a few parts of that sentence that people need to answer for themselves, but the core of the sentiment is super important.

You need to figure out what “value” means in your particular organization (or to yourself!). You need to figure out how you can actually create that value.

And, most importantly, you have to go out and actually do the work!

Lots of data professionals are really interested in “research” and “working on interesting problems” – the problem is that “doing research” doesn’t necessarily create value!

If you want to find success in whatever organization you’re in, a single-minded focus on how your efforts can create more value is the best way to do it.

Where can readers find you online?

Email is the best way to get a hold of me: kaminsky.michael@gmail.com

And I’m pretty active on the locallyoptimistic slack channel – learn how to get an invite here.

Q&A with... Ludovic Tavernier, Data Consultant and Data Visualization Lead

2019-09-21T00:00:00Z

Interviewed on 21st September 2019

Analysts Assemble is back for another round of interviews with data professionals from across the wide data spectrum. This time we have French data visualization lead Ludovic Tavernier in the hotseat.

I’ve been looking forward to finishing this one for about six months. Our original conversation got cut short with
the arrival of Ludovic’s first child earlier this year - which was a more than acceptable excuse to put an interview on hold!

Let’s hear from Ludovic now on how he got into data, where he developed his skillset and how he thinks
other data folks should be working to push their own skills on.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

Hi, I’m Ludovic Tavernier and I’m a Data Consultant and Data Visualization Lead in a French data consultancy agency named Valoway.

My work consists in supporting clients in their technical data needs and leveraging the value behind their data legacy. I am also involved in my company growth participating in recruitment, sales and pre-sales activities.

I began my data journey in my engineering school with a Masters focused on Business Intelligence and data mining. My first internship, more than 7 years ago, was mainly about benchmarking two growing softwares: Qlikview and Tableau Software.

After that, I joined a data consultancy firm in Paris where I learned more about building and maintaining data environment for the following years.

In 2017, I decided to come back to the tool I tried a few years back: Tableau Software. Aside of my daily work routine, I learned more about Tableau and data visualization.

I discovered the Tableau community and took part of some dataviz challenges. It ended by bringing my first professional Tableau-related projects and by participating in the Tableau Software IronViz Finale in 2018!

(Weather Memories built in 20 minutes during Tableau IronViz Contest)

Now, I continue my work in traditional data environments but with much more focus on data visualization. I teach and explain basic concepts of it and keep learning at the same time!

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

My day work is like data visualization: it depends!

I work at my client’s office most of the week. But more and more I can perform my consultant job remotely. This is a game changer, it avoids heavy transport durations and lets me focus on the job to be done (and more).

I tend to be a technical guy, but I enjoy teaching and supporting my clients in their data journey. Especially when it comes to Tableau or Data Visualization.

I have used a good variety of tools, from databases like Oracle, MySQL, Teradata, SAP Hana to reporting softwares like Qlikview, Power BI, Toucan Toco and, of course, Tableau Software.

I also used some ETL with a good understanding of SQL language.

You’ve been building up a good following through your blog and social media. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

Thank you! I can be shy about what I am doing, less about what I am learning. I would say people do not need to share everything publicly from the beginning.

If you know about the Dunning-Kruger effect, the Mount Stupid is a dangerous place for every data professional to be because we feel confident enough to teach what we “poorly” know. And if we have a voice in social media, it can be disastrous for our readers.

But it is a good moment to share what we do, to receive feedbacks and to go deep into the valley of despair. I personally began to teach Tableau and to give feedback during my journey in the slope of enlightenment. I think it is the good moment to do it.

I do hope to reach the plateau of sustainability in the data visualization field during my lifetime, to feel confident enough to share every type of content.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I first want to improve my data visualization skills, learn new languages and tools to perform better in that field (like d3.js or Adobe Illustrator). I am sure it will lead to new great experiences.

Both building my technical skills and moving to a management-based role are attractive though. And I do think the two are correlated. Keeping practicing is crucial as the technical data landscape is always and quickly changing.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

I encourage participating in challenges, for me it was #makeovermonday or #SWDchallenge. I encourage welcoming feedbacks from the community you are in (for me it was the Tableau community).

And my best-kept-secret is people in general. I try to follow as many as possible to keep getting inspired in my field.

What is the number one piece of advice you give to aspiring data professionals?

Practice! Everything is about practicing and courageously confronting the science you want to master. Oh and keep in mind there is always better than us!

Where can readers find you online?

You can find me on Twitter at @ltavernier7, on LinkedIn, on Tableau Public or my blog.

Data-Driven Tech Needs An Ethical Revolution

2019-09-12T00:00:00Z

Ethics.

What does that word mean to you?

Doing the “right” thing? Doing “no evil” to other people?

Even within these vague notions of ethical behaviour, there are massive gaps for nuance on how to accomplish them.

Wikipedia says:

“Ethics addresses the issues of what is right, what is just and what is fair.”

Which again gives us the what but not the how.

These aren’t new concepts.

Ethical questions crop up daily, not just in data and technology, but in society in general.

So how did we, collectively, allow ethics to become the forgetten element of our data and technology revolution?

Fake news, bias in Artifical Intelligence algorithms and widespread privacy breaches mean that ethics is becoming an increasingly large area of concern for data professionals.

But shouldn’t it always have been like that? (I hear a quiet voice naively say.)

We’d all like to say yes. That it’s always been top of mind. But the truth is somewhat different.

And we all know it.

Bringing privacy concerns to the masses.

Netflix’s documentary film The Great Hack brings the story of what went on behind the scenes in the Cambridge Analytica / Facebook data scandal to a more widespread audience. It’s well worth a watch.

But it doesn’t steady the nerves to come out the other side and know deep down that it barely scratches the surface. Not even close, especially when it comes to dark money and the influence that has had on Western politics and discourse in society over the past decade.

We have all played our part.

We have been complicit in handing over large swathes of our personal data and, in most cases, not even expecting any compensation for it. Someone has been getting rich off of it however - a lot of someones - and they have displayed absolutely no qualms whatsoever about doing it.

The list of wrongdoing goes on and on:

“Free” social media sites sucking up your every online thought and feeling.
Tailored online advertising that follows you across the web from site to site.
Third party tracker software that lets companies do that.
Wearable devices that tout on you to your employer and insurance companies to help them weasel out of their obligations to you.
Period tracker apps that hand your personal data over to Facebook.

And those are just the tip of the iceberg.

Two things are at the root of all of it - data and technology.

So on one side, we (the private citizens) have willingly given up our privacy and data at a faster rate than anyone could imagine or fathom.

And on the other side, we (the technologists of all shapes and stripes) have helped scoop it all up and turn into weapons of mass civil destruction.

All in pursuit of the mighty dollar.

It’s been said that the greatest minds of our generation have been wasted on building new and ever more insidious techniques for the proliferation of online advertising technology. And what a sad thought that is.

For them and for the rest of us.

The detrimental race to the absolute bottom has been nothing short of disastrous for users and consumers alike. Just ask anyone who has used an internet browser over the past ten years.

How many would tell you the experience has gotten better as the technology has advanced? None bar the argumentative trolls would be my guess.

Online frackers.

Professor David Carroll, one of the main protagonists in The Great Hack, wrote an article in 2015 where he likened adtech to fracking wells:

“The trackers that pepper every page start to look a lot like fracking wells. They tunnel deep into a place that they probably shouldn’t go, poisoning the well as they dig deeper toward personalization and retargeting. Consumers know they’re being stalked and most hate it.”

It’s little surprise to find that Professor Carroll was the man who pursued Cambridge Analytica with so much vigour to see how much of his own personal data they held.

It’s not just being used to fuel consumer spending.

It’s bad enough when the data being harvested is used to try and sell us more stuff we probably don’t need.

But even more so when it comes to using that data for voter profiling and deliberately using it as a base to attack democratic structures across the globe.

Yes, advertising has always been a dirty business you say. It was psychological warfare on an intimate, personal level long before David Ogilvy turned it into an art form. So what’s different these days?

The technological advances of the past decade have moved things to a previously unprecedented level. If unchecked, we could be looking back on these days as glorious halcyon days in terms of personal privacy in five years, never mind another ten.

We have to stop that downward spiral.

Have the tech unicorns done this to us on purpose?

I refuse to believe that the leaders of this technology revolution set out to degrade personal privacy and standards of general decency on purpose.

Can we honestly say Mark Zuckerberg set up Facebook with the grand plan of turning into a modern, real-life Bond villain?

In his Harvard dorm room, he couldn’t have foreseen how every decision he took along the way would erode our civil liberties. But surely he sees it now.

The real power move would be in holding up his hands and saying “we’ve went too far here, far too far, it’s time to rein in back in”. But I’m seeing little of that in his recent appearances (and non-appearances) in front of a growing collection of different countries elected representatives.

Jack Dorsey’s unwillingness to counter the issue of Nazis on Twitter is the same. It’s not that Jack lives in a bubble (although listening to his infamous podcast interview about his daily routine might suggest otherwise).

He knows the problem is there and has clearly decided not to address it. It’s right in front of his face. There is no moral obstacle to hide behind.

The people have to stand up for what is right.

Do we instead have to look to the designers, developers and product managers in companies like Facebook and Twitter? Is it enough to hope they suddenly take appraisal of what they are part of and say “no, time to stop the train, I want off”?

At what point do they do a Mitchell & Webb, look around the open office at HQ and say “are we the baddies?”

Or do they just quiet that voice in their head and tell themselves it’s not them. It’s “someone else” who should be making that decision, someone far above their pay-grade. A suit somewhere high up the chain of command who is getting paid much more money to make those calls. They just work there after all.

It comes down to your own personal ethics.

It’s a matter of personal taste but how bad does the stench have to get before you can no longer hold your nose and ignore the smell? It comes back to ethics. Your own personal ethics. What you are prepared to do for a few quid and what are you prepared to accept in the pursuit of that.

It’s not an easy question for anyone. I know.

I’ve been there.

I worked for over twelve years for a major high street bank. And barely a week of that time went by without another story in the press about how agents and staff members at that bank had conspired to hoodwink and screw over customers.

Normal Joe and Jane Public. Small business owners. Larger business owners. Young people. Older people. The vulnerable and the successful. There was little distinction made when it came to finding potential targets.

When things were really bad I took to telling people I was a human trafficker and drug dealer because it was less embarrassing to my kids than telling them the truth. I exaggerate slightly but I did use that line a few times on taxi drivers when we were having a chat about it.

You choose what you will accept - and what you won’t.

So why did I stick it out so long? Because I made damn sure that anything I was a part of was not ultimately detrimental to our customers. I wore the uniform (so to speak) but I was damned if I was going to ever take a hand in anything like that myself.

I was lucky. I worked for a manager who acted with integrity. He respected that I would not sit idly by and keep my mouth shut when things were being misrepresented to our team. But the continual tales of what had happened elsewhere in the organisation eventually took their toll.

I won’t let my conscience allow my children to starve for the sake of principles but what kind of father would I be to blindly ignore what is going on around me? And worse, willingly partake in it.

And for what? A paycheque. An annual bonus. A new car in the drive-way and another foreign holiday.

It comes down to where do YOU personally draw the line?

Data is the new oil.

Which means it’s incredibly valuable and VERY dangerous to handle.

I was a data analyst. Yes, I became a manager and then a head of unit but at heart I’ve always been a data analyst.

I had access to TONS of personal data about all of our customers. Where they lived. What they bought. Where they went. The who, what, when and how of their daily lives.

We could have used that many times to build data models that skirted the boundaries of what was legal and ethical. But we all have to have our lines in the sand. Regardless of where we sit in the company hierarchy.

Fighting the good fight for an ethical corporate culture.

The major problem arises when the ethical breaches and general lack of willingness to pursue ethical business practices come from the very top. We hear a lot of talk these days about “corporate culture” but it’s here that the real standards are set.

It’s easy for an executive to make a public statement about how the culture is being changed and all of the misdeeds are in the past.

It’s difficult for employees, be they data professionals, developers or warehouse workers, to take that seriously when they know the person saying it was one of the culprits of the real ethical indiscretion.

Don’t treat your staff like mushrooms.

We know. And we know YOU know we know. So stop the charade. For everyone’s sake.

If you keep asking me to do unethical practices as an employee then I WILL NOT WORK WITH YOU.

I am but one small instrument. But all of us together demanding the same standards - now that is something to be reckoned with.

An integral component of everything you do.

Ethics MUST be embedded in our product development and interactions with customers at all stages. It can’t be taught separately in colleges - like it’s something distinct from the technical learning. It’s an intrinsic part of it.

That’s how we build a better future on strong ethical ground and put this current dumpster fire behind us.

Just because we can, doesn’t mean we should.

As a data professional I want to help stem the tide of privacy violation. I still believe in the power of data and technology and the positive impact they can bring to society - as a whole and to the individual.

We will never achieve that if we set our morals and ethics on the shelf when we sit down to start coding. But it’s going to need a concerted effort from all data professionals to speak up when things are being done that harm us as a society.

We can’t rely on those “upstairs” to make those decisions for us. That’s on us. Each one of us. And it will be difficult. But we can do it.

Oaths are worthless and talk is cheap.

The days of “move fast and break things” need to end. It’s time to go back to “don’t be evil” above all else. It doesn’t need a Hippocratic Oath for the data industry.

It just needs people to think about something more than themselves. Future generations will judge us badly if we don’t.

The time to make real positive changes is now.

(Photos by elCarito, Zbynek Burival, Pepi Stojanovski on Unsplash)

Presenting...SQL Crash Course

2019-09-09T00:00:00Z

My tune hasn’t changed over the years on just how important SQL is as a game-changer for careers in data. In fact, I believe it is THE most fundamental skill you must have to get started as a data analyst.

I don’t want anyone to be scared off from learning SQL by long, boring, complicated courses. So I’ve put together a free SQL crash course to try and keep folks in the SQL game.

What does it cover?

It’s (currently) 35 bite-sized lessons plus interactive quiz questions and covers all of the SQL basics to get people up and running and querying database tables. From basic SELECT queries to JOINs, CASE statements, dealing with NULL values and more.

Who is this FREE SQL Crash Course for?

You don’t have any SQL experience. All you’re getting is tumbleweed when you apply for data jobs that ask for SQL knowledge. It’s starting to look like you’ll never get your foot in the door.
You’ve been working with data and spreadsheets for years in your current job. Excel was your gateway drug. A macro here, a formula there. But no-one takes you seriously when you want to make the move into an actual BI analyst role.
You’ve taken Data Analytics classes in college and filled your resume with buzzwords aplenty. Spark, Hadoop, Python, R. But hiring managers won’t even consider you for entry level data analyst jobs. They see your face drop into a sad frown when they mention your lack of SQL knowledge.
You hear recruiters say “every analyst should know the basics of SQL” but you don’t even know where to begin.

What do you need to know?

You don’t need any experience of SQL or coding for this course. We take it slowly and build the solid groundwork you’ll need to get up to speed with SQL. Then you can practice at your own pace with some interactive quizzes with real data and hands-on SQL code.

Check it out at SQLCrashCourse.com and let me know how you get on.

10 Ways To Get Active In The Data Community

2019-08-31T00:00:00Z

Remote working has been one of the major highlights of the technology revolution of the past decade.

Taking the dog for a mid-day walk. Picking your kids up from school. Hitting the gym at lunchtime for a serious biceps session (what, just me?)

Can’t do those sitting in an open plan office with 200 other people trying (and failing) to focus on their SQL code.

But it comes with downsides too.

Working from your low cost of living, no commute, small town means you are miles away from a major city and the in-person tech communities that live there.

It can get lonely. You can feel like you’re being left behind and working in a virtual vacuum.

This can be tough at any stage of your career but especially in the formative early years when it’s even more important to soak up knowledge from co-workers a little further down the track.

Even if you live and work in the city, you might find your rather introverted nature is bringing up anxiety on getting involved with other data professionals outside of your own day-to-day work. So how do we overcome these issues and get more active in the data community?

10 Ways To Get Active In The Data Community.

Here are 10 ways for you to get up and running and find your data tribe. Online or offline, we have plenty of options to experiment with:

1. Get involved in online forums.

Doesn’t matter where in the world you live, fire up your browser and you’re instantly in a massively crowded room full of people looking for the same thing you are - to be part of a data community.

My personal favourite online forum is Reddit. It gets a bad rep sometimes because of it’s vast size and the breadth of topics it covers across hundreds of sub-Reddits. Don’t let that put you off.

When it comes to data, I stick to a few well-moderated and (largely) friendly subs, /r/businessintelligence, /r/datascience and /r/dataanalysis.

You can niche down further depending on your own tastes with forums specifically for SQL, Python, PowerBI, Tableau - you name it, Reddit has a large crowd of people ready and willing to talk about all aspects of it.

But to get the most out of your online data community forum experience, don’t limit yourself to just reading the other posts. That’s a great place to start and feel your way into the internal culture of how other forum members talk to each other.

But to really get into the community, you need to chip in on discussions. Add your own take on what’s being discussed, even if you think someone else has already said it. There is always real value in bringing your own experience to the table.

2. Twitter.

I’ve found the data community on Twitter to be the most welcoming, helpful and (generally) least toxic of all the social networks. Find the right people to start following and then go down the rabbit hole of seeing who interacts with them and who you want to keep up to date with.

There is so much excellent content being shared it can be difficult to keep up so try not to get swamped. As with forums, you’ll get more value out of overcoming the fear of joining in by actually tweeting replies and adding your own take on discussions.

It’s a community so there will be the odd knob. Block and mute are your friends. To paraphrase Van Wilder Party Liaison, arguing on the internet is like a rocking chair. It gives you something to do but it doesn’t get you anywhere.

3. Facebook Groups.

I have major trust issues with Facebook as an organisation. But it’s hard to argue with the number of specialised communities that it houses in Facebook Groups. Whether you are into Power BI, Tableau, different cloud providers, data science, data analysis, you name it, there will be a ton of Groups sitting waiting to receive you.

That doesn’t mean that all are created equally though so it will take some trial and error to find ones that suit you. The more controlled nature of FB Groups in terms of sheer volume of members means it can be easier to get your voice heard but you do have the downside of less regular discussions coming up from the smaller pot of people. Maybe that suits you better than a wide open space like Twitter or Reddit though.

4. LinkedIn.

It’s the number one business-related social network on the planet. There is a vibrant (and talkative) data community discussing all manner of data-related topics on a daily basis.

Everyone who is anyone is on there including all major recruiters for top companies all over the world. But sometimes the feeling that it’s all a bunch of sleazy salesmen selling to other sleazy salesmen can be hard to shift.

I read my LinkedIn news feed some days and feel like I need to have a bath. And then a shower just to make sure. It makes me feel dirty. Saying that, there are some great data folks on there providing daily valuable content.

The comment sections can be hit-and-miss. It’s easy to get lost in the throng of self-promoting big talkers who are always eager to push their gatekeeping agendas. But get the right people to follow and you can learn a lot.

Once you’ve got your sea legs with commenting on Reddit and Twitter, you’ll find the comment and Tweet format aren’t quite enough for all you have to say on a subject. That’s the time to start your own blog.

Even if you are posting articles to the sound of silence for a while until you build up a readership, it’s still the most valuable development programme you can put yourself through.

Own your platform. Self-host using Wordpress or another static site generator and just get writing. It improves your process of thinking through problems and is a great way to show potential employers that you can 1) write and 2) deliver those thoughts in a clear and succinct manner.

Get enough people reading your posts and start a regular email newsletter list as well. This is hands-down the best advice I can give to any data professional. Bar none.

6. Post articles on Dev.to and Medium.

When you’ve written your articles then you can cross-post them for further distribution on content aggregators like Dev.to and Medium.

I prefer Dev myself as the community there is smaller and a little more cosy than in other places. Don’t just drive-by post and then run off. Get involved the comments on yours and other posts and become a proper part of the community.

That’s what it’s all about after all. Offering up your own articles and thoughts is just your table stakes. The real value comes after that by helping others and joining in discussions on what they have to say about what you have to say.

Medium is bigger but they have a paywall and I don’t think it improves the service for readers or writers. Plenty of data people disagree so it’s purely a matter of personal taste.

7. Local meetups.

“Just a small town girl living in a lonely world”.

Journey had it right but it doesn’t have to stay that way for you in your hometown forever. If you have a large town or small city nearby, look on meetup.com for any data-related local meetups, make the effort to visit and meet some real human flesh people.

If you don’t find any then start your own. Even if it’s a handful of people from your town sitting around one small table in a coffee shop, it’s a start. And you might be surprised how many others like you are out there just waiting for something to materialise.

If you are on the introvert end of the scale (like me), you might even be better off organising your own as it gives you a job to do and something to focus on rather than sitting like a non-speaking extra at the side of the room. Just a thought.

Following local data community people on Twitter might give you an insight into where and when they go to meetups and a quick DM can be a great icebreaker.

8. Conferences.

The souped up version of local meetups. Hundreds (maybe thousands) of people milling around. Salespeople on booths trying to grab your interest. Too many talks to see in a day. And ticket prices that can run into thousands.

None of these are selling points to me. I don’t like big crowds for a start. Conferences can be incredibly useful though if you take a strategic plan of action to them.

Get your company to pay for it. Always.
Have a very targeted pre-plan for which talks you want to see. Don’t just wing it on the day, it rarely works out well.
If there are other people you’ve been in contact with on social media or forums (you have been interacting haven’t you?) then pre-arrange to meet up with them for a coffee in-person if it suits them and they have time.

Probably best not to wander around until you spot them and leap out from behind a booth to tell them you’re their number one fan. Personal boundaries and all that.

No-one wants to be friends with Annie Wilkes no matter how much they enjoy their ego being stroked.

The good thing is there has never been so much fantastic free data and analytics content available on the internet. The bad thing is…there has never been so much free data and analytics content available on the internet.

How to sift through it all to separate the wheat from the chaff? You can’t read or listen to everything or else there would be no time to do the actual analysing. So signing up to some of the data community leaders’ newsletters is a good start.

Some like Avinash Kaushik’s Occam’s Razor are excellent as learning pieces in and of themselves. Others are a great way to get curated links to interesting content from a wide range of sources, some you may never have seen or heard of on your travels.

Interacting with those running the newsletters can be as simple as replying to their email, most are more than happy to discuss what they’ve been sending out. Plus you can also use those articles you have been writing for your own blog and pitch them as potential content for the newsletter in future.

Win-win scenario for all involved.

10. Build and launch a data product.

There are two simple steps to generating some interest around yourself and the things you are doing in the data community:

Make something.
Tell people about it.

Doesn’t get much simpler than that (in theory at least). But what do you build that hasn’t been done before?

For starters, who said it never had to be done before? Get a data source, could be open data or even scraped from the web. Transform it, display it and put it somewhere people can see it. Job done.

Could be an interactive dashboard. A niche job board scraped from various other sites. Or convert a spreadsheet with some measures about different cities for travellers into a fully fledged public-facing product like Pieter Levels did with Nomad List. I don’t think Pieter classes himself as any kind of data scientist but that doesn’t matter.

Data is the glue that holds together all areas of business. Ideas are all around you.

And once you’ve built v1.0, get back onto your blog to write about it. Then tell people in the forums and social media groups that you’ve been befriending and see if it starts any new conversations. And who knows where that might lead for you?

Now it’s over to you.

I hope these have given you some ideas for how to move out of your comfort zone in getting further involved in the data community. There are so many options available, both on our doorsteps and across the globe, but they’ll only become valuable if you make the effort to join in and contribute.

Creation, not just mindless consumption, is the key and that advice crosses over to more facets of our lives than just data. Good luck in your adventures!

(Photo by Perry Grone on Unsplash)

What Should I Do When My Data Science Project Gets Canned?

2019-08-20T00:00:00Z

All that work down the tubes.

So your data science project that you worked on for six months got canned and you don’t know what to do. There are two distinct phases to recovering from such a mortal blow:

Anger comes first.

Gnash your teeth. Weep. Loudly. And in a sobbing fashion. Lie on the ground and kick your feet like a toddler in a supermarket that has been told to put the packet of sweeties back on the shelf.

Acceptance and personal growth.

Pick yourself up off the ground. Dust yourself down. Put on your Big Person Pants and get right back in the game. Because tomorrow is another day.

But why did your project get binned when you worked so hard on it?

The odds were against you from the start. A recent article in Venturebeat says that 87% of all data science projects never get put into production. You don’t have to be a mathematician to work out that in reality that means about 1 in every 10 DS projects gets green lighted while the rest get flushed. Not exactly encouraging when you frame it like that, is it?

So what stops the rest making it beyond the Proof of Concept phase?

Crappy Model.

Sometimes your model just plain sucks. Sorry, them’s just cold hard facts. Maybe the data scientists involved just weren’t experienced (or good) enough to get the most out of the data they had in front of them.

Could be there was a lack of business knowledge amongst the DS team that made them ask the wrong question. It happens. Again, inexperience or not collaborating with the right subject matter experts at the project scoping stage means you’ve basically wasted your time. Wave it goodbye.

Crappy Data.

Or the data they had to stitch together into a Frankenstein’s Monster-style creation from disparate spreadsheets, handwritten notes and whispered conversations under bathroom stalls wasn’t quite good enough to wring anything meaningful back out of it.

I should have “You Work With What You’ve Got” tattooed on my arm after the amount of times I’ve used it to account for less than desirable data situations over the years.

Frightened Management.

It’s one of life’s mockeries that the businesses with most money available to spend on data science teams are inevitably the ones who have the most risk-averse senior management with the most stringent regulators breathing down their necks. Think banking, insurance and healthcare for obvious examples.

All rotten with money and all (quite rightfully) shit scared of the heavy hand of regulatory authority coming down on them. In the choice between drawing a line through your project or getting hauled in front of a regulatory panel, there’s only ever going to be one winner. Sorry champ.

Shiny New Object Chasing Management.

A particular favourite of mine. They follow the trends in business magazines just long enough to kick projects off when the buzzwords are hot but not long enough to see them through to a proper conclusion.

Brought in to staff a Big Data team in Hadoop last year? Sorry pal, budget removed. We’re all doing AI this season, hadn’t you heard?

Office Politics.

“Why did Jane get budget for her project and I didn’t get any for mine?” “Why did Tom get ten more staff and I had a headcount revision?” etc. etc. The average corporate boardroom has more to offer David Attenborough in terms of animalistic territory marking behaviour than the Amazon rainforest.

Back-stabbing. Talking behind backs. Power struggles. Coups. Counter-coups. It makes Game of Thrones look like a quiet family sitcom. With the inevitable result that someone wins their Game and someone loses. And looks like this time it was your project. Them’s the breaks kid.

Communication Breakdown.

This happens a few levels down from the boardroom bickering Cold War-esque shenanigans we just covered. This is shop floor, inter-department, inter-team rivalry and it’s just as cut-throat as the war the suits upstairs get into. Data Engineering don’t talk to Data Science. Data Science don’t talk to Data Management. Data Management don’t talk to Business Intelligence.

And no-one talks to I.T. because they are a bunch of basement dwelling neckbeards with no standards of personal hygiene or inter-personal skills. Or so you heard Steve from Data Engineering say in the queue for coffee in the canteen.

Either way, we need a well oiled machine working in unison. Instead we’ve got a bunch of unicyclists all off doing their own thing with no co-operation towards the bigger picture. Is it any wonder that even projects that get into production don’t end up delivering any of the benefits they initially promised? Hell no.

What can we do to increase our project’s chance of success?

Winning the rollover lottery jackpot has a probability per ticket somewhere in the tens of millions to one. But still people buy them. If our chance of data science project “success” is around 13% then why wouldn’t we try to increase our chances of being on that side of the equation?

Looking at the reasons for failure above, we see a few we can’t quite control at the operational level. Execs gonna exec until the cows come home so we’ll ignore that one. Hoping we’ve not got one of those business magazine, hype-chasing senior managers is another one for the wishlist but hard to control in real-life.

Instead try these three points to increase your project’s chance of making it to the holy land of production deployment:

Pick the right problem.

Banging your head off a brick wall form the get-go won’t get you anywhere. You have to be realistic about what you are trying to do and how it will ultimately be implemented by the business. If it’s a long shot right at the start, it’ll make it a lot harder to keep bouncing it over the inevitable hurdles you’ll face to get it live further down the road.

Start small, stay simple.

Ensuring you don’t over-complicate the problem (just because you CAN doesn’t necessarily mean you SHOULD) is the main slogan to stick over your workstation. Simple doesn’t mean basic. It means easier to explain, easier to sell to your management and their management and easier to implement. There are no prizes for taking the hardest road possible just to prove how smart you are. It’s a self defeating policy right out of the blocks. Don’t do it.

Get the right team together to tackle the problem at hand.

Teamwork makes the dream work. Uuggh, I feel dirty even typing that. But it’s true. Office politics and inter-team rivalries will always exist but try and work through them for the greater good. It would amaze absolutely no-one who has ever worked in a corporate environment that the biggest obstacles to success are to be found within your own organization. The “competition” don’t even come into it. Work to overcome that in your project and you’ll have a much higher chance of success.

And it’ll still probably not make the grade.

I don’t want to put a downer on your mood but the odds will still be against you. A 1 in 10 strike rate should be expected in an experimental area like data science where not every idea is going to survive the Proof of Concept stage.

I’m seeing more and more despondency from young analysts and data scientists though when their long-term projects get shit-canned for whatever reason. I’ve been there myself and it’s one of the major learning experiences, even further on in our data careers, that not everything is going to go into production. It’s not called Data “Science” for nothing.

Educating the bosses.

Another major problem seems to be a misunderstanding at executive level that every project should be a winner or else the whole discipline is a waste of time. Which ultimately boils down to a lack of basic data literacy at senior levels. This is key to the success of any data-driven strategy and is often overlooked.

As modern data professionals we need to figure out how to educate our business leaders and bring them along with us as data equals. If we don’t we’ll see more and more projects fail and the attention will turn elsewhere for the next magic bullet for business success. And that would be a missed opportunity for all of us: nerds, suits and pen-pushers alike.

(Photos by Paweł Czerwiński and Raychan on Unsplash)

Best SQL Online Courses, Training, Resources and Tutorials

2019-08-11T00:00:00Z

What is SQL?

SQL is Structured Query Language. It lets you communicate with relational databases: getting data out, putting data in, changing data, removing data. And much more.

Do I really need to learn SQL to be a data analyst?

I don’t make the following statement lightly: SQL is a superpower for data analysts. If you have any intention of making a successful career in data, your first action should be to learn the basics of SQL.

It will stand by you for the rest of your career in a way that many other tech skills will not. In the words of the famous footwear brand with the swoosh: Just Do It.

Where do I begin learning SQL?

Start with the basics.

SQL is one of those deceptive masters in that it’s pretty easy to learn enough to get you up and running but it can take a lifetime to fully conquer it’s advanced intricacies. Don’t let that discourage you though, getting started and hooked is the easy part.

I’m going to build up a list of the best online SQL courses, training, resources and tutorials. I’ll keep adding to it over time so feel free to drop me a note (alan AT simpleanalytical DOT com) if your favourite resource is missing.

Best Free Beginner SQL Tutorials.

SQL Crash Course
(Disclaimer: I wrote the tutorials on this one. 30+ bite-sized SQL lessons to get you up and running. Interactive quiz questions and the lot…)
SQLZoo
SQL Bolt
W3 Schools SQL
SQL Teaching
SQL Course
SQL Fun Time
Select Star SQL
PostgreSQL Exercises

Best Paid SQL Online Courses.

Udemy.

Coursera.

Others.

SQL Essential Training - from Linkedin Learning (formerly Lynda.com)
Learn SQL - from Codecademy.
Mastery With SQL - from Neil Sainsbury.

Best Online SQL Resources and Articles.

Helen Anderson writes about SQL and databases on her own blog and on Dev.to. (She also did a Q&A with me here about her career and data journey so far.)
Randy Au is a regular contributor to Towards Data Science and font of all knowledge on small team data science skills including SQL. (Randy also did a Q&A with me here last year.)
Katie Kodes is a top class Salesforce and Python developer who also shares some excellent resources on SQL. Check out Katie’s Q&A here for more background on her data career and skills.

Are MOOCs Credible Enough To Get You A Data Science Job?

2019-04-24T00:00:00Z

Barely a week goes by without another set of battle lines being drawn in the war of future education. Fast moving disciplines (or collections of disciplines) like Data Science are quite rightly at the forefront of that war.

What got us a job in the data field five years ago is unlikely to be enough to get us a similar job today. That goes double if we haven’t kept our skills fresh and our eyes on developments across the whole DS spectrum.

That leaves those of us currently working as seasoned data professionals on the hamster wheel of continuous skill development, both inside and, more likely, outside of our working day. Beyond the burden of finding the time, energy and direction to pursue these new skills, we also have the choice of how how to get this education.

Never mind R vs Python or AWS vs Azure, our first choice has to be the method of learning: old school (in-person, trusted institutions) vs. new school (the potential Wild West of MOOCs).

The case for the defence.

Data Science Consultant Nic Ryan nailed his colours to the mast with a blog post on his Data Friends blog. Nic credits MOOCs with getting him interested in data science in the first place and getting him to where he is today. (If anyone isn’t familiar with Nic and his work, check out his appearance on Kirill Eremenko’s SuperDataScience podcast).

Nic’s background is in actuarial science. When he wanted to move into data science he realised he had the two choices above. Going the traditional route of back into studying full-time would have meant giving up the sole income his family was operating on (which wasn’t an option for obvious reasons) or use MOOCs to build his skills through online learning.

He went with MOOCs as they offered a more cost-effective and higher quality learning path than going back to university would have been able to give him.

It’s not just the ticket price of tuition fees, it’s the opportunity cost of dropping out of employment for a couple of years to pursue an academic qualification that may not give you the actual skills you will need at the end of it. The piece of paper you get is nice but it’s not much use on the first day on the job. Practical, up-to-date skills based learning in a MOOC is the opposite.

I’m a big believer in Lifelong Learning. Whether it be reading books and articles, taking courses or attending meetups and talks, there is something to be said for soaking up the value of other people’s knowledge and experience, regardless of what age you are. There has never been a better time for all of the world’s knowledge to be available to those of us lucky enough to have an internet connection and a connected device.

It’s all there for us. MOOCs are the large scale embodiment of that available knowledge-base so why wouldn’t we utilise them to their full potential?

What are the downsides of using MOOCs?

I’ve written before that the biggest problem with MOOCs is that we don’t actually use them to their full potential.

Anecdotally, it’s accepted that little more than 2% of MOOCs started are ever properly finished. I know I’m guilty of leaving a few in the 98% part of that. Self-managing your education means not having the same level of accountability over completing projects, lessons and tasks as you would have with actual tutors and deadlines to meet.

Without that accountability it also means that it’s somewhat easier for unscrupulous characters to claim that they’ve completed many MOOCs when they may not have quite made it to the end of the course (or anywhere near it in fact). That severely reduces faith in equating MOOCs with traditional training courses or learning qualifications in the eyes of many employers.

It’s hard to see how that can be overcome even with a growing number of MOOCs offering certificates and accreditation like nanodegrees for actual completion. The value just isn’t there amongst most HR departments when it comes to viewing these as actual credentials. And that’s a major problem when you don’t have a degree or Master’s underpinning your MOOC education.

Doing it for ourselves.

That being said, MOOCs do give us an opportunity to build and hone new skills that we may not get in our day-to-day jobs. Tech platforms and infrastructure in companies can make it difficult to implement every cutting edge technology we read about in Towards Data Science or KD Nuggets. Signing up for a MOOC on Coursera, Udemy or LinkedIn Learning can help provide us with the necessary entry into a new area which we wouldn’t otherwise have been able to get.

If MOOCs are still in the infancy of their credibility when it comes to the data science job search then isn’t it enough that we use them for ourselves and our own learning rather than to impress a potential employer? I think so. If resting on your technical laurels is good enough for you then I would question why you picked a field like data science to work in in the first place.

Of course, new technical skills only really get properly included in your personal toolset when you get to use them regularly in your job. That applies just as much to traditional in-person course learning as it does to MOOCs so it would be unfair to use this against the online learning options alone.

So will my list of DS MOOCs get me my dream job?

You’ll see I’m back and forward on the question of just how valuable MOOCs are when it comes to getting people into a data science job. Maybe the credibility issue will be worked out over time as we see more and more people taking advantage of the lower cost and higher quality offerings from MOOC providers.

As the data science field matures and some of the data scientists of today become the hiring managers of tomorrow, we may see a move away from reliance on often outdated university courses and teachings anyway. Until then it’s hard to see how we can get away from advising new entrants to get at least a Bachelor’s degree to base their future learning on.

On a personal level, I don’t see any downside to pursuing further education at any stage of your career. In a rapidly changing world like data science, MOOCs can keep you closer to the cutting edge at a fraction of the cost and under the tutelage of some of the world’s finest DS minds.

So why would anyone turn that down?

You do have to actually do the work though and push on through to complete the courses.

Even better is to take those new skills and put them into practice on a personal project you can point future employers to on your own personal site, Github or Kaggle. Learn for yourself but don’t be afraid to let people know what you’ve learnt.

There are no prizes in this game for hiding your light under a bushel. If you have the skills on the job, I doubt anyone will ultimately care where you picked them up, just that you have them to call upon when needed.

(Photo by Raj Eiamworakul on Unsplash)

Q&A with Martin Weiss, Analytical Consultant and Project Manager

2019-04-23T00:00:00Z

Interviewed on 23rd April 2019

We all know about Germany’s world-renowned reputation for efficiency and technological innovation. It’s no surprise then to find a tech superpower like Google using Hamburg as one of their main European bases. Based there is this week’s guest on Analysts Assemble, Analytical Consultant and Project Manager Martin Weiss. We dig into Martin’s path into the data world and see how he uses his marketing and business prowess, along with his technical skills, to help Google’s advertising clients.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

My name is Martin Weiss and I am an Analytical Consultant at Google in Hamburg, Germany. My role is a mix between a classic marketing/sales analyst and an internal consultant or project manager. As such I work on internal projects as well as directly with Google’s advertising clients helping them with custom data analysis as well as building tools and dashboards for them. Another major part of the job is internal and external data education teaching data savviness to all stakeholders.

I would say my data journey is quite typical. It started during the last year of my studies (Business Administration and Economics) when I worked as Business Analyst for now one of the most successful German ecommerce startup. I got hooked on the online marketing and entrepreneurship world and loved the open non-hierarchical startup and tech industry culture.

After graduation I wanted to stay in that culture but also wanted to start at a larger company for my first real role. That’s when I applied at Google. At least in Europe the entry job there is usually as Google Ads consultant for small businesses and startups. The position allowed me to build a really strong background in PPC and online marketing overall as well as to get insights into literally hundreds of client businesses and their needs.

During my time in that role I always had some projects with a more analytical focus. This helped me to build an internal portfolio of projects which I used to transfer into my current position as an Analytical Consultant.

As a side project I also work on AnalyticalMarketer.io, my personal blog writing beginner level technical guides and career advice for marketing analytics.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

I work in a 8 member team at the German Google headquarter office. Our team is part of a larger organization inside Google and acts as internal consultants for the teams managing Google’s advertising clients. As such our projects are incredibly diverse and we can choose projects depending on our personal strengths and interests. I am very happy and thankful to be on that team as it allows me to build up the technical and other skills I am interested in.

We have an internal BI suite, which we use for building dashboards, data pipelines etc. For languages SQL is my daily bread. I recently picked up some JavaScript, which I use for building some simple apps and tools.

Since our projects are so diverse, it’s hard to say what a typical day looks like, but it could be something like this: First thing in the morning I check my emails and get up-to-date on anything, that happened after I left the day before.

Next I re-prioritize my projects and to-dos for the day from that input. Afterwards I start working on a dashboard I have to build for one of our clients only to realize that the SQL query I started running the night before had some errors in it and has to be re-run. Which means I can’t work on the dashboard until late afternoon.

Instead I work on a search query forecasting tool and spend half of my time there googling simple stuff as I am a complete JavaScript rookie. My afternoon starts with a quick email check and a meeting on how we can include machine learning applications better into our day-to-day job.

Afterwards I work on a client insights workshop I am going to facilitate next week before I can finally finish that dashboard from the morning.

You’ve been building up a good following through some excellent guides and tutorials on your blog. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think other professionals are doing this already a lot more. It seem data professionals are a little bit behind on this. I am not sure why though. Maybe because there are not as many freelancers in this area? I am also not sure how important it is. However what I do know is that there is definitely no harm to it and that it makes a lot of things in your career easier.

Being publicly known will open a lot of new doors for you. No matter, if you are employed and especially not if you are self-employed. As I am just getting started with my blog I am not sure how beneficial it really will be in the end. But hopefully it will provide a valid second professional pillar for me at some point.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

Surprisingly probably neither. I am not really seeking to move into a corporate management role. And even though I definitely want to keep improving my technical skills I don’t want to become a data engineer.

I believe that our world and especially marketing is becoming more and more data driven. And data savviness is becoming a must have skill. As such I want to move more into educational roles teaching others data skills. A common challenge I see among marketers and businesses is that they feel helpless looking at all their data. They know, where to (technically) get the data from, but have problems drawing insights from it and translating it into next steps.

That’s the angle I want to take, teaching them how to use that data for actionable insights and for planning steps to grow their business. My current vehicle for that is AnalyticalMarketer.io, but I am thinking to put more focus on it in my day job as well.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

I have been very lucky, since Google has so many internal resources and courses to learn. So that’s where I usually go, when I want to learn something completely new. Externally (even though technically he also works at Google 😉 ) kaushik.net is probably my number one source for new analytical content.

As for “best-kept-secrets” I would say don’t be shy to google something! I guess for 95% of the technical challenges you are facing everyday somebody else had the same problem or something similar. Searching for your problem online will at least give you starting points to try out.

What is the number one piece of advice you give to aspiring data analysts?

Focus on analysis and insights rather than reporting. A mistake I see a lot of analysts make is to only focus on the reporting side of the job. So e.g. building dashboards or “data dumping” onto slides without any deeper analysis. But that’s where the real value of an analyst sits. Not only providing the data by rather drawing actionable insights and recommending next steps based on that data.

I think this will become more important in the future as on the one hand more and more data will be available and on the other hand a lot of the more technical aspects of the job will become automated (e.g. pulling the data). As such in order to provide value as analyst you will also have to know the business side quite well and provide actionable insights by combining the data and business world.

So in order to stand out be sure to be awesome not only at drawing actionable insights from data but also at visualizing and communicating recommendations to other stakeholders.

Where can readers find you online?

AnalyticalMarketer.io for all my content and consulting services, @the_mmw on Twitter.

Q&A with Christian Young, Senior Data Analyst / Team Lead at Xero

2019-04-23T00:00:00Z

Interviewed on 23rd April 2019

New Zealand may be small in terms of population but their burgeoning tech scene is punching well above it’s weight when it comes to excellence in data and analytics.

This week’s Analysts Assemble guest is Christian Young , a Wellington based senior analyst and team lead at accounting software company Xero. Christian takes us through his own route from privacy law and records management to managing his own BI reporting team.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

Hello, my name is Christian, UK born but have lived in New Zealand since I was 7.

My journey to data is quite atypical and often a story I tell to people who worry that they didn’t study the right thing etc. So I graduated university with a Bachelor or Arts in both Psychology and Linguistics (both were my major). In fact I had done every class needed to go on to apply for my PhD in Psychology. However I decided that wasn’t quite for me and graduated instead of applying for the Doctorate.

My first job out of Uni was a privacy focused role where I had to read child abuse and redact out the stuff that couldn’t be released to the client. This gave me a really good foundation in privacy law.

A few years later I ended up as a Records Manager, responsible for advising how New Zealand’s biggest and oldest government agency should store and dispose of it’s official records. It was in this job I self taught myself some SQL to get data out of the back end of the software we used.

I then got my first analyst job but in an Ops team. I used SQL to do all sorts of things, assess problems, assist marketing, automate admin tasks, find missing or bad data. It was a great job in terms of exposure. I learned multiple languages, different types of databases (Oracle, TSQL, Postgres) , and how to make and schedule procedures.

I also ended up as senior analyst in that team, I had to manage the team and their workload as well as juggle ~100 stakeholders. I became a SME for many of our systems and often consulted on projects and I invested a lot of time into automating and documenting what we did so we could free up time to do more. Which is how I ended up at Xero.

I was hired to take over a team, which I have now just done, after spending six months learning the ropes. The team is responsible for providing one-off reporting to our clients and I have an exciting, if not stressful, year or so ahead of me.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

My typical day involves monitoring our ticketing system for new work, meeting with clients, and cutting code.

I have a team of four, though currently we are a team of three and none of us are in the same city. However our wider data team are numerous so I’m not quite a lone wolf, there are always members of my team around (like the famous Helen Anderson.) My other two colleagues are in 2 different offices, so there is a heavy use of Slack and video calls to bridge that gap.

We are also an international company with substantial offices in Asia, America, Australia, UK, and New Zealand so we can be talking to clients at all hours. Though I do want to stress that my team works NZ hours, and I’m militant about work life balance so the only time a member of my team should be working odd hours is to talk to client in a different timezone and then they go home early / come in late the next day.

In terms of tools and technology. We have a Redshift data warehouse and we surface the information for our clients into MicroStrategy, our reporting layer.

The tool I spend all day using and cannot live without is Visual Studio Code. I cut all of my code here and have spent considerable amounts of time setting up extensions that speed up my workflow.

You’ve recently joined the ranks of data bloggers on Dev.to. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think it’s extremely important for anyone in the data field to find a way of making information transfer second nature.

Whether it be blogging, vlogging, conference talks, commenting on articles, or solving people’s questions on stack overflow it’s vitally important.

Because if we are being brutally honest no-one in BI has got it figured out, like at all.

No one team in the world has nailed data quality AND data governance AND their data pipeline AND their data storage solution AND their reporting layer AND their customer satisfaction. That scenario doesn’t exist and, if by some miracle it did exist, all it would take to undo it all would be one new feature, one migration, a single change in company direction.

BI is always a game of catch up and while we are all a striving towards the same thing at the high level, the devil is in the details. Lots of problem solving is involved and while no two scenarios will be identical there is enough similarities that its very worthwhile sharing what you did so others can benefit.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I’m not overly interested in the technical. I love what I know currently but doing things like AWS certs and development doesn’t quite hold my interest so I believe my future is in the other direction.

It does seem clear to me that Data Governance is a potential field that’s about to boom. With the big names in IT constantly shitting the bed in terms of customer data I think that’s a space I could move towards.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

Other than Google / Stack Overflow or ASK TOM (for Oracle only), I really don’t. I’m constantly on the look out for a reliable blog / Reddit / website / person but I’ve not really come across any that I have found endless value in.

Like most, I’m self taught which came from persistent googling and about 500 Stack Overflow tabs open.

So yeah, many many thanks to anyone who has contributed to a SQL problem on Stack Overflow!!!

What is the number one piece of advice you give to aspiring data professionals?

My advice for ANY aspiring professionals is to first understand the importance of risk management and documentation. Once you have those down pat and are practising them in your day to day job you will quickly start to understand how to get things done. It’s very easy to get lost in the business as usual aspect of your job, but if you can’t see the forest from the trees you aren’t going to push the needle very far, and understanding risk management is a great place to start.

Where can readers find you online?

Ronsoak on Medium.

Ronsoak on Twitter.

Ronsoak on Dev.to.

Ronsoak on Amazon at some point in the future (I’m writing sci-fi).

**Bonus points if you know the reference.**

Should I Bother Learning SAS Instead Of Python or R?

2019-04-19T00:00:00Z

Everyone love shiny new objects.

And data professionals are just as apt to “play the Magpie” as any other group of techies.

We’ve looked at the downsides of execs and senior managers getting blinded by the flashy new lights. But it can be just as distracting to the data analyst themselves when it comes to choosing which programming language to learn.

You’d have to have been living under a rock for the past five years to have missed the explosion in popularity of both Python and R for data analysis. Jeff Hale’s recent analysis of data scientist job postings on KD Nuggets displayed perfectly just how important these two languages are for any aspiring data pro on the job hunt.

Python appeared in over 70% of the job adverts he analysed and R appeared in over 60%. To spread your net it looks like a no-brainer to go either one way or the other in these directions.

But what about SAS? Is it really the dinosaur everyone says it is?

Or is there still value in adding this veteran of the data analytics scene to your portfolio?

SAS appeared in only around 25% of Jeff’s analysed job vacancies so based on that it’s already a dead duck. I’ve got some personal experience to the contrary however. I’ve been using SAS in a professional analytics setting for almost ten years so it’s served me very well.

SAS was used extensively throughout the whole of the banking group I worked for. From SAS Enterprise Miner to mine customer service calls to SAS Visual Analytics to give internal business users Self-Service BI and analysis options.

Some hands-on personal anecdata.

I used Base SAS myself for exploratory data analysis and a range of other analysis jobs. It gets a lot of hate these days but I think that’s due more to it’s proprietary licensing and cost compared to R and Python.

It’s incredibly powerful if used correctly but easy to pick up for anyone with any previous coding experience. Even being able to use your SQL knowledge to run Proc SQL statements until you get up to speed with the SAS data step is really useful.

I don’t like to see people choosing tech or languages like they choose a sports team and only sticking with one option. Having some SAS knowledge along with Python or R, or at least being able to take what you know in one and transform that to another, is a real superpower.

So if it’s not dead, where could we end up using SAS in our data jobs?

It’s well accepted that SAS is big in organizations with plenty of money to spend on licenses. It’s traditional powerbase has always been in industries such as banking and financial services, telecommunications, healthcare and in academia.

If you are looking for a data scientist job in the startup world, it’s unlikely that you’ll be handed a Base SAS license as your primary tool to work from. If your local analytics job market is filled with big corporate players in the aforementioned industries however, you would be well placed to pick up some SAS skills and ride that horse as long as it lasts.

It’s easy to think that every organization operates on the bleeding edge of new technology but that just isn’t so when it comes to large corporates. They can be slow to shift and have long drawn out procurement cycles.

If you are coming in to twenty years worth of legacy SAS infrastructure, you can be damn sure you’ll still be using SAS in that workplace in ten years time at least.

So are you saying I should definitely learn SAS then?

As always, analyse the job adverts in your target market and industry. See what they are asking for. Even better, ask their current analysts what they are working on and with and see if SAS pops up in that list. I’m a firm believer in bringing a broad toolkit to work with you rather than going very deep and narrow on one particular skill or technology.

For me, SAS has been a cornerstone of much of what I’ve built and delivered over the past ten years. And that’s pretty good going for a language I’d never even heard of until I started in that company.

Q&A with Zach Renwick, Business Intelligence Developer

2019-04-14T00:00:00Z

Interviewed on 14th April 2019

After speaking to this week’s guest on Analysts Assemble I’m more interested than ever in finding out just how many data professionals start their journey by studying Accounting at university before jacking it in as a career choice when the bright lights and high life of the financial services data world come calling. (Just like I did).

My guest’s name is Zach Renwick and he is a Canadian BI developer based in Vancouver, British Columbia. Zach tells us how he got his start in BI, pivoted away from his planned career choice and is working towards his goal of making a better world through data and analytics.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

In my first job out of university, I was working in accounting at a construction company. I found it crazy how much work revolved around copying and pasting data, and many of our reports were error-prone due to their manual nature. I knew there had to be a better way, and after doing some research I started using Microsoft’s self-service BI tools. Learning about PowerPivot and Power Query in Excel helped me automate most of my regular reporting, and I was eager to learn more.

Around this time I abandoned my plan to become a designated accountant in order to focus on analytics. I enrolled in a data analytics program instead of the Canadian CPA program. From the start of my very first course (Relational Databases and SQL), I could tell this path was a better fit for me.

Eventually I secured a job as a BI analyst, and was later promoted to a BI developer role in the same company. Along the way I kept developing my technical skills by reading books and technical blogs, and following what others were doing on Twitter.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

I work on a BI team with two other BI developers, a DBA, a business analyst who plans our work, and a QA analyst who tests. Most of the time we commute to the office, but a few times a month we will work from home. On a typical day I may work on the following:

creating/maintaining SSRS and Power BI reports that are integrated into our company’s web application
mentoring internal BI analysts on SQL queries/ad-hoc reports
helping plan future report development
monitoring ETL processes
investigating data quality issues
keeping up with the latest BI/analytics updates by reading a few blog posts.

We use the Microsoft BI stack (SSRS, Power BI, SSAS, SSIS) and spend a lot of time with SQL everyday.

You’ve been building up a good following through your interaction with the data community on Twitter. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I’ve found it extremely rewarding to engage with the data community online and in person. I’ve attended SQL Saturday and Power BI user group meetups, and I’m always impressed by how willing people are to share their ideas and experiences with each other.

There’s also a wealth of information on being shared on Twitter centred around how we can make better use of our data. I love keeping up with the the latest analytics and tech updates, and Twitter is often the first place these announcements are made.

I’ve recently started sharing some of my own experiences and learnings as a data professional and I’ve been surprised by how much feedback I’ve received from others – both inside and outside my organization. You don’t always get to implement your ideas into production, so showcasing analytics Proof Of Concepts (POC’s) in blog posts is another way to get your ideas noticed internally.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

For now, I want to keep developing my data engineering/ETL technical skills. Eventually, I would love to lead a reporting/insights/analytics team. Becoming a BI developer has allowed me to dive deep into the back-end side of things (DevOps, Database Administration, and Cloud Architecture), which is incredibly important for delivering full-stack data solutions.

In the future, I’d like to switch gears to focus more on working directly with business leaders to deliver insights and encourage better use of the existing tools we have.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

None of my resources are very “secretive”, or unknown. I found blogs like SQL BI and Simple Analytical (the previous name for Analysts Assemble) really helpful when I started learning about analytics and dimensional modelling. I also purchased a Data Camp subscription which was very useful for practising SQL before I actually started working in business intelligence.

What is the number one piece of advice you give to aspiring data professionals?

It’s one thing to read about analytics projects online, it’s another thing to actually implement them. You should practice recreating what other’s have demonstrated in your spare time. Try building your own databases, using different analytical techniques in SQL, R or Python, and trying the latest technology you’ve read about.

If you share your experience, even better, as you’ll have a portfolio of work to show potential employers and clients!

Where can readers find you online?

I’m active on Twitter and recently started writing articles on LinkedIn.

Q&A with Katie Kodes, Salesforce and Python Integrations Programmer

2019-04-07T00:00:00Z

Interviewed on 7th April 2019

My favourite finding in doing this Analysts Assemble series has been just how humble and genuinely helpful the data science community is. And this week’s guest is certainly no exception.

I’d first read Katie’s excellent article on SQL Joins on Dev.to and when fellow Dev blogger Helen Anderson recommended I speak to her for this series, I jumped at the opportunity.

Katie takes us through her introduction to the world of programming, getting involved with the Salesforce eco-system and how she recommends people lift themselves, and others, up through building their overall skill-sets:

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

Let’s get a few things out of the way. I’m white. I’m able-bodied. I speak English natively. I have no immigration worries. My childhood environments were safe and educational. I’m university-educated and had professional computer programmers as adult role models growing up.

As a teenager, my family paid for the roof over my head, not the other way around. That enabled me to look at summer job descriptions first, wages second. The job I had as the Great Recession of 2007/2008 began remained unscathed and provided my career incredible refuge.

I call myself a “late bloomer” into programming, but the soil I was planted in is a huge part of my story!

But jumping ahead to my first full-time jobs, I started learning a lot about databases through data entry roles that I applied for as a “fast & accurate typist.” At one job, people got so tired of writing query reports for me that they taught me how to build my own.

That’s how I learned about joining tables and avoiding “cross-joins.” Programming was much more engaging at the office than it had ever been in my few attempts to study it in a classroom.

I only recently thought about it, but the fact that all the programmers who delegated work and trained me were women — and that all their bosses were female programmers/architects — and that they weren’t all white or computer scientists — might have normalized programming as a career path in ways that being friends with hobbyist teenage boy coders previously hadn’t.

Eventually, when I felt “stuck”, I took night classes to formalize my studies, which advanced my foundation for further learning enormously.

Landing a database query reporting job on a team that had just purchased Salesforce made me an #AccidentalAdmin. Trying to automate away #WhyAdminsDrink problems turned me into an #Admineloper as I learned to write Salesforce Apex triggers and Python scripts.

As I’ve gone deeper into “Extract, Transform, Load” (ETL) integrations, I’ve returned to my traditional database roots and started working with a team of Oracle and SQL Server developers, DBAs, and integration specialists. We’re inventing my job description as needs arise, but so far it’s going to involve:

keeping “runbooks” of things that need to be done when we upgrade a database
helping with deduplication (which came immediately after I offloaded my old deduplication responsibilities to someone else — you just can’t run from that job, can you?)
working with 3rd-party vendors to troubleshoot and fix their integration products when upgrades make them fail in production
coding!

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Office based/remote?

I try to exercise as often as I can, because computer work is brutal to your hands, eyes, and back. Recently, I’ve been able to use a pool and discovered treading water is great for your upper body AND timing meetup talk ideas. (Does anything speed up 30 minutes better than needing 40?)

As to the rest of the workday, I was lone-wolf as the “computer person” on a team of end users until joining my current team, which is small and office-based. I’ve veered toward the “integrations programming” side of data, not “analytics” or “statistics” or “visualization.” None of my coding projects typically take more than a week, and a lot of my work involves supporting them afterwards. Here are a few examples of requests I see:

– “Hey Katie, I loaded bad data into database A, and now the daily sync to database B you wrote filled database B with the bad data. What’s the proper sequence of cleanup tasks, and can you help me perform them in bulk?”

– “Hey Katie, we stopped requiring a snail-mail address on our ‘request info’ form, so the automation you wrote us to remind us to send everyone a brochure is often irrelevant. Can you change it to only create a reminder if they asked for paper mail?”

Tools:

– I use point-and-click tools with Salesforce as much as I can, but occasionally write Python scripts to clean data on my hard drive or over an API, and I write Salesforce database triggers in their Java-like language called “Apex.”

– With Oracle and SQL Server databases, most of what I do is in SQL and PL/SQL or T-SQL.

– Enterprise-level ETL tools for scheduling inter-database communications often involve what I like to call “point-and-click programming.”

Integration work is pretty “MacGyver”-ey. You access data with the tools your company invested in, and you cobble solutions together as effectively and efficiently as they allow.

You’ve had a great response to your own blog and your articles on Dev.to. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I have priceless colleagues who don’t seem to share what they know far beyond the water cooler.

But don’t be afraid to scratch that itch if you have it, any time you have an “aha!” or “???” moment.

As Jessica Kerr and Julia Evans pointed out in episode 16 of the “Greater than Code” podcast, people who just learned a topic often explain it best to people who are trying to learn it.

Meetups (including in-company user groups) and local conferences are a very forgiving setting for stumbling through public speaking. Blogs and Twitter are great ways to practice writing concisely.

You can always start a new blog as you “level up” your writing and speaking experience to consolidate your “best-of” moments and move on from your “newbie” days.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

At the moment, technical skills interest me far more than managing people, as much as I love being around them.

But never say never!

If you had a list of “best-kept-secrets” (websites, books, coaches) you’d recommend, which would you recommend?

Alan, you gave me so much decision anxiety when you asked this question. I got stuck for days until I had added all the books at my office desk to the “resources” section of my website so they’d know I love them, too.

Other people will cover the big ones like FreeCodeCamp and Dev.to, so let me share two unusual dusty old books:

“Ones and Zeros : Understanding Boolean Algebra, Digital Circuits, and the Logic of Sets” was a random library find that absorbed me nightly. I’ve always loved conditional logic, but this book really threw down the gauntlet and asked me, “Oh, so you think you can AND/OR?”
“Introduction to Unix and Shell Programming” (Venkateshmurthy). This is published in India and can be hard to find in some countries, but it’s worth the effort. After many failed efforts, this was the friendly, thorough explanation that finally taught me Unix.

What is the number one piece of advice you give to aspiring data scientists?

One? Just one? Can I pick two? Please?

1. Master the art of concisely reframing your “shortcomings” so you’re in charge of the narrative. I hate to make a “case study” out of another person’s struggle rather than my own, but I think an issue that foreign students sometimes face makes a great example.

Friends I’ve known often struggled to find work permitted by their student visas, either because companies “don’t offer internships” (in American English, that has a pretty narrow connotation in the world of short-term work) or presume it costs money to hire visa-holders (true with “work visas,” but free of charge for the limited work permissions included in a “student visa”).

I’ve suggested working the following into cover letters and networking encounters: “My student visa allows me to work in the US for a few years at no cost to employers, so I’m looking for short-term opportunities during summers and after I graduate to take advantage of that.”

Humans, including employers, seem to make up reasons they can’t do things out of fear of the unknown. Apply your “explain it like I’m five” and “elevator speech” skills to whatever you’re terrified someone will “judge” you about. Offer your story first and don’t leave people gaps to fill with their own imaginations!

2. Ask yourself what you love about data.

– Storytelling?

– Asking questions and exploring the unknown?

– Math and statistics?

– Programming?

Even if you’re just interested for the money (that’s okay — everyone has to eat!), what aspect of “data” scares you the least to try to make a living with? What seems to suit you?

Specialize in that as you start, to quickly ramp up into an employable niche.

“Know what you don’t know,” though, and stay big-picture informed about what “data scientists” with other specialties are doing as time marches on. It’ll help you decide what new developments in your specialty are really relevant, and it’ll help you decide whether you’re interested in adding a new specialty.

Where can readers find you online?

My personal site – Katie Kodes

Dev.to

Twitter – @katiekodes

Q&A with Robert Crocker, Tableau and Data Visualization Consultant

2019-03-31T00:00:00Z

Interviewed on 31st March 2019

Analysing data has come a long way from the days of studying pie charts and tables of figures in spreadsheets. Publications like The Pudding and the New York Times have helped take data visualization to a new level and it’s in this world that data viz consultant Robert Crocker excels.

A globe-trotting digital nomad, Robert uses his skills in Tableau to bring real insight and business wins to some of the world’s largest companies. Let’s hear from Robert and see where his journey into the world of data visualization consulting began:

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

I started out as a Supply Chain Analyst. I transformed more data with MS Access than I care to admit. Thankfully the team I was working with allowed me to be the first from the team to display our supply chain simulation data with Tableau. After doing that for some time I moved into a proper IT role to learn how to build data pipelines with SQL to feed my dashboards.

After that work I moved out to the Bay Area to dive deeper into the technical side of things with D3 and Javascript. Professionally, I was a consultant for Slalom where my job was primarily designing Tableau dashboards for our client base in that local market. Now I am a nomad freelancing remotely for clients like IBM, Nike, Paul-Mitchell and all sorts of others.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf?

I’m using Tableau as an individual freelancer, which means my roles is different for each client. For some, the job description is as specific as building a single chart, while others have me making all their strategic decisions around the Tableau platform while also developing their dashboards.

You’ve built up a good following through your own blog and newsletter. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

It depends on what you want, but I do think a body of work demonstrates a level of dedication to the craft that clearly sets candidates apart. It also says a lot about your interest in data visualization and/or analytics, which is extremely important as you move throughout your career.

So I’d tell anyone to look for areas that interest them and see if there isn’t a way to publicly demonstrate these interest. Bonus points if the work you share can be made useful for others, such as tutorials.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I honestly don’t want a management role. I love building things. Tableau has actually expanded the breadth of their tool quite a bit lately which has had me dive a little more deeply into Tableau itself. I love D3 and JS, so those will always be fun hobbies.

I want to keep my SQL strong and might build a few data pipelines with Airbnb’s open source tool Airflow just so I can learn more about what it takes to engineer components I have not had to handle myself while continuing to leverage the skills I already have.

So more learning and hopefully stability in my clients base. I might need to build up some sellable assets to generate passive income as well, but stable clients first.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

MOOCs are everything, but the amount, of courses, you can take is ridiculous! That said, I do think you should always be learning so enroll in at least a course at a time and give yourself an hour a day M-F to consistently chip away at the course material.

What is the number one piece of advice you give to aspiring data scientists?

Learn to love the journey. If you aim to learn the first principles behind the applications(abstractions) we use to be productive you’ll be able to make better, more thoughtful decisions as a data scientist.

Where can readers find you online?

I post on vizsimply.com every once in a while, but I’d love to see you say hi on Twitter @robcrock or LinkedIn.

Q&A with Justin Butlion, Entrepreneur and Analytics Specialist

2019-03-24T00:00:00Z

Interviewed on 24th March 2019

In this edition of the Analysts Assemble interview series, I speak to South African digital nomad and analytics specialist Justin Butlion.

I started reading articles last year on Justin’s blog ProjectBI and was immediately struck by how practical and knowledgeable his advice and insight was. Not just in the theory of data and business intelligence but where it really matters – in the real business world.

We’ll find out more about Justin’s background, why he decided to take his BI consultancy services out on the road and where he plans to take his business and data career next.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

My name is Justin Butlion and I’m 34 years old. I was born and raised in Port Elizabeth, South Africa. At the age of 19 I immigrated with my family to Israel where I live for 14 years before becoming a digital nomad last April.

I’ve always been a problem solver, critical thinker and interested in computers. These interests got me interested in entrepreneurship and the online world. After a failed shot at starting my own startup I got extremely lucky to join an incredible company called Yotpo.

I was the 12th employee at the company and spent 4.5 years in a number of roles in the company. By the time I left Yotpo had grown to hundreds of employees and is today considered one of Israel’s best startups.

My data journey started at Yotpo. In the early days I was constantly looking at the data and the management took note. After 2 years working as a marketer and product manager I was given the opportunity of leading a new team in the company called the Business Performance Team.

I took the job and over the next 2 years my team and I built out Yotpo’s internal business intelligence infrastructure and analyst team. This period was a crash course in everything BI for me. I soaked up everything like a sponge and saw the power of BI and how a data-driven mindset can help a company in all areas.

After 4.5 years at Yotpo I decided it was time to say goodbye and start a new chapter. I believed strongly that other companies could benefit from my skills and decided to become an independent consultant. I’ve been doing that for 18 months now and have worked with 15 different companies around the world.

2018 was a great year for the business but I wasn’t 100% committed to the idea of building out a large operation. This year is different. In January I decided to go all in on growing my business and brand (ProjectBI). I’ve started scaling my operations both in terms of number of clients and resources. I recently signed my first sub-contractor and assistant and I’m aggressively chasing new clients.

My goal for 2019 is to bring in 5X what I did last year. It’s a big goal but I’d rather aim high and miss than aim low and hit.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

My typical day starts with a few hours of work in my apartment. I’m currently in Medellin, Colombia and all my clients are in Israel or Europe so the time difference is not in my favor. I therefore start the day answering emails, going over Slack and jumping into calls.

By around lunchtime I’ll leave the apartment and head to gym and then the co-working space. I’m currently working out of Selina which is a great space.

All my work is done remotely using your standard set of tools like Slack and Gmail. I use Asana for task management and Slack for communicating with my team.

The work itself is quite varied but there is a heavy focus currently on Tableau. I also write a lot of SQL and spend a lot of time in tools like Google Analytics, Firebase, BigQuery, Intercom and CRM systems.

I started my career following people like Seth Godin and Gary Vaynerchuk so I’ve been thinking about personal branding for a long time. I also started out as a marketer and know how powerful a quality blog and other web assets can be to a business.

I know it’s not for everyone but I’d highly recommend that anyone that works online consider investing in a long-term strategy which aims to build their personal brand. There are many ways to do this, writing content for a blog is just one way.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

For now I’m focused on building out ProjectBI. I see a growing trend towards being “data-driven” and I’m well positioned to take advantage of this wave. I have a lot of ideas for ProjectBI and excited to see where I can take things.

I’m extremely hands on as a consultant which forces me to constantly learn new things. I hope to structure my schedule in the future to allow for more structured learning time.

There is no way I can scale ProjectBI without transitioning from 100% hands on to a more balanced split between hands on and manager. For now I’m doing both but this isn’t optimal and I’m working on making the transition.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

I was very fortunate to get paired up with an amazing mentor when I became the Director of Business Performance at Yotpo. This individual worked with me for 2 years and was an incredible resource of knowledge in all aspects of the job. If you can find someone that has done the job before and done it well that can train and support you it will make all the difference.

Honestly I don’t have a list of resources which immediately comes to mind. My secret is I like to learn and I’m determined to solve the puzzle. One good example that I can share involves me being hired by a client that used Firebase. I had never worked with this platform before but I didn’t let that get me down.

I sat on my own dime for many hours working out how to use the platform and today I consider myself an expert on Firebase’s data capabilities.

I think it’s all mental in the end of the day. It’s a matter of being resourceful, determined, patient and hard working.

What is the number one piece of advice you give to aspiring data scientists?

Be true to yourself. I know that sounds very “new age” and philosophical but I really believe it. You need to know yourself very well and build your life around your strengths, interests, and preferences. This might mean working for a small startup because you like the high-paced environment, or focusing on building out your own training site because you like the idea of doing your own thing.

Where can readers find you online?

I can be found on LinkedIn, Twitter, Nomadlist and on the ProjectBI blog.

Q&A with Jesse Pisel, Geo-Data Scientist

2019-03-17T00:00:00Z

Interviewed on 17th March 2019

One of my favourite parts of doing the Analysts Assemble series is getting to find out more about data worlds that I would never ordinarily know exist. And this interview with Jesse Pisel is an excellent example of that.

Jesse analyzes data in the geology and geospatial tech world and caught my eye with a series of geology-based data visualizations on Twitter. Let’s get into Jesse’s story in a little more detail.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

My background is in quantitative stratigraphy and geology, so I learned a lot of the basics of data and analysis while doing my PhD at the Colorado School of Mines. I first got into data using R and Mathematica for my dissertation, and started learning Python as well. As I continued to learn more about data analysis I really got into Python as my language of choice. This was in 2015 as more and more data science oriented packages were coming out for Python.

When TensorFlow came out I knew that I had picked a good language to focus on, and have continued down the Python path. I have worked on a variety of different datasets over the past few years that include everything from spatial distributions of ancient river channels, to timeseries forecasting of oil and gas prices. The variety is great because it gives me a chance to learn new tools and apply different things I have learned to new datasets.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

A typical day for me depends on where I am at in a project. If I am starting a new project, or if a coworker comes to me with some questions on a dataset, I could spend a majority of my day doing exploratory data analysis. If we have a funding proposal coming due soon or if I am wrapping up a project I tend to do a lot of technical writing and documenting. It’s kind of funny looking at my GitHub page, I can tell when I am wrapping up a project because there are fewer commits and a lot less activity as I spend most of my time writing up documents.

Most days though, I spend a lot of time visualizing and understanding the data I am working with. The Seaborn package in Python is great for this because you can quickly visualize distributions and relationships in the data. One key thing that I have been working on lately is organizing workflows and making sure I comment code so that I can come back to the project and understand what I did.

Jupyter notebooks are great for adding markdown comments and creating a linear workflow that is easy to follow from cell to cell. I highly recommend Jupyter for collaborating and sharing information, you can run Julia, Python, or R in them which can lead to some pretty great integrations between the languages.

As far as work environment goes, I work as a lone wolf in our department. We have other analysts, but they focus mostly on geospatial data in GIS. Despite this they are great to chat with about different ideas and make sure I am following solid logic for analysis as well. We are all co-located in one building which is great to be able to walk over and chat with people, but we still use a lot of remote tools (chat, docs, version control, etc.).

I like that I can work on shapefiles in geopandas in Python and do full custom spatial analysis before sending them to colleagues working in GIS for online maps, presentations, and reports. One last tool that I have been using quite a bit lately is a package called Verde that is used for gridding data in Python. It’s great for making maps and interpreting subsurface and geophysical data (gravity, magnetics, etc.).

You’re building up a good following through tweets about your work on Twitter. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

At first I wasn’t sure how to really use Twitter besides keeping up with what was going on in the world of machine learning and geology. After seeing some of the people I follow tweeting about their blog posts, I figured that would be a great way to create content that I was interested in, teach myself new skills, and start a conversation with anyone interested in similar topics.

I think it’s key for data professionals to share different things they are learning even if they think it’s been done already. So many times I run across an analysis or a way of thinking from a different field that I can directly apply to geologic problems.

One of my favorites came from a discussion with an economic data analyst, where we came up with the idea to use cost functions to model connectivity between oil and gas reservoirs. So it’s great for everyone to share what they are learning because you never know who might be working on a similar problem in a different field.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I really enjoy the technical side of things, so I want to stay on that side of the business. I think the next area that I really want to move into is deploying solutions at scale. So I guess it’s moving from more of a science role into a bit of an engineering role.

Building a solid foundation for deploying at scale completes the data life cycle from collection to processing to large scale inference.

Understanding all the different parts of the data life cycle is key to designing systems that work efficiently. But who knows, I could be happy in a management type role, as there is definitely places where machine learning could make managing a team significantly easier.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

For geologists, geophysicists, and anyone interested in subsurface data I have to recommend the Software Underground. It’s a great community of scientists and engineers discussing everything from Python and machine learning to rock types and hackathons. There is a slack channel, a GitHub, and a bunch of other resources at https://softwareunderground.org/

Another great place to start is contributing to open-source projects. It’s pretty intimidating at first to contribute to production level code, but everyone is really nice and offer a lot of help on writing efficient code and documenting it.

It’s amazing the amount of work that maintainers put into open-source packages, and they need as much help as they can get. So it’s win-win, you learn something, and you get to contribute to something you use on a daily basis.

What is the number one piece of advice you give to aspiring data scientists?

Understand your data. Knowing how it was collected, who collected it, when it was collected, how it was processed, and what it’s supposed to “look like” goes a long way to interpreting results, finding bias, and solving problems.

Where can readers find you online?

I am on Twitter @geologyjesse and on LinkedIn as Jesse Pisel. On most social platforms (Slack, etc.) I am @jessepisel

Q&A with Samir Abid, Professional Sports Analytics Consultant

2019-03-10T00:00:00Z

Interviewed on 10th March 2019

This edition of Analysts Assemble takes us into the exciting world of professional sports analytics.

Analytics consultant Samir Abid tells us about his journey from engineering to analytics and how his experience of these worlds has put him in pole position to help professional sports teams with their insights.

We also hear how Samir is using the knowledge he has built up over the years to help others work better with their own data.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

My name is Samir Abid and I run a sports analytics consultancy called Pace Insights. We work primarily with British based Olympic and Professional sports teams to help them make their data useful (harder than it sounds!)

My background is actually motorsports and automotive engineering. Specifically “vehicle dynamics” which involves the design and optimisation of suspension and steering systems.

Computer Aided Design (CAD) and Computer Aided Engineering (CAE) systems were fairly new when I first started in the industry. The established guys were not so interested but I was. I’d built and upgraded a couple of PC’s whilst at Uni (as you did that then!) and always found them easy to use. I was always helping my Mum how to use her computer and also others at Uni with theirs.

Maybe I was also a bit lazy in that if I set up the computer right, it could do the work for me – less hassle and more accuracy. It didn’t bother other people so much but I always got frustrated having to repeat the same thing twice. Still does actually!

A car suspension system is surprisingly complicated. It has non-linear inter-dependencies and means you can quickly get lost trying to design and develop them. Some clever software guys had built these “multi-body-dynamics” tools which used maths to enable us to simulate the whole system.

These were physical models using lots of simultaneous equations. They were pretty crude and slow to use but I got into it and enjoyed running the models at work. To speed things up I then learnt about using stats models to create response surfaces using Monte Carlo and other similar types of analysis. We’d create a model off a model (eek!) but they proved really useful – I was even able to publish them into Excel with some slider bars so the other Engineers in the team could use them.

That was my first experience of developing “data products” for others to use and the lessons (nee mistakes!) I made then stick with me through to today. I just kept at it until they had something useful. It was really rewarding.

I was actually lucky enough to work on the new Aston Martin DB9. We did this from a clean sheet of paper (quite rare these days). Despite being a high-end brand, they didn’t actually have a lot of cash. In fact, where we’d normally have 3 prototype phases, they could only afford 2.

This meant the team were relying even more than ever on the simulation results from my models to specify and design everything from the suspension arms, the brakes, the steering rack, even where to position the engine. When it came out, Top Gear loved the car (and the handling!) so much they created a whole new section on their “cool wall” to put the car in! Good times.

A good friend of mine from Formula 1 had then moved to UK Sport. He asked me to help with a few projects in about 2010. It became clear that sports were getting more technical and starting to use data more on their programmes but didn’t really have the skills or experience. I’ve been helping them get more value from their data, reports and processes ever since.

What’s a typical day look like for you in your current data role? Which tools and languages do you use? Big team/small team/lone wolf? Remote/office based/co-working space?

As a consultancy the nature of the work varies wildly. We’re based out of an office and workshop in Leamington Spa. This is a great central location for most of our customer base. Over the years the team has grown and shrunk, peaking at 8. These days I tend to front the work and call on a select group of proven freelancers when needed.

It is probably not that cool but the majority of our work has been developing “apps” in Excel over the years. This was purposefully done as all our customers have it, it makes a great prototyping environment, can be programmed (with VBA), requires little/no “environmental” maintenance and can easily handle most of the data types and data volumes our customers have.

I have a thing for making reports and charts look good. This is possible in Excel (one customer actually didn’t realise they were using Excel on one training session!) but clearly Excel does have significant limitations.

We provide a “full-stack” solution for our customers. When Excel hasn’t been appropriate we’ve used Python (lots) and built a few Django apps with PostgresSQL databases. More recently we’ve done a lot more in Google Sheets with Appscript / Javascript as well as Power BI – which is my goto BI tool at the moment, not least because of DAX and Power Query and the constant evolution to support R and Python scripts.

I’ve even been able to try some “live” dashboard development on-the-fly in front of a room full of grumpy coaches! That clearly breaks every 101 rule about how to avoid looking like you’ve no idea what you’re doing but in fact the risk came off much better than I expected. The rabbit was pulled from the hat. Cue much debate about the consequences of what they were seeing rather than trying to comprehend what they were seeing (success!).

I highly recommend trying it sometime.

The blogging is relatively new thing. Writing, unlike maths and models, doesn’t come easy. I’ve set up the side project Your Data Driven where I am hoping to share some of my experiences over the years in the hope of helping others.

One thing I am finding is that it really makes you think about assumptions and jargon. I really want the content to be valuable and “actionable” rather than hand-wavey-bs and I think that is why I find it so difficult!

For anyone to do that well will certainly help their career because, whilst the headline is data, it is really all about people. Being able to communicate your ideas clearly is so important and often not done that well. Hopefully I will get better by doing more of it.

Where do you see your own data career going next? Building on your technical skills or moving into a more management-based role?

I really enjoy the consultancy and I work with a fascinating industry. Adopting value-based pricing has really helped square away some of my own issues about doing consulting (i.e. the obsession with buying / selling time, which has no correlation to outcome success and penalises people like me who like to work really fast). The roller coaster of feast or famine does get a bit much after a while.

Of all the things I’ve done over the years, helping others to do their work better / easier has given me the most pleasure. In the future I’d like to work out more ways I can help them away from implementation of bespoke apps.

Recently I’ve done more strategic consulting for a couple of customers and thoroughly enjoyed that. I’d also like to see where I can take the blog – if people are interested in what I’ve got to share then that could lead to creating some small products which would be a good challenge.

I’ve been successfully building them on a bespoke basis so I’d like to see if I can make something to successfully solve an issue everyone is facing.

If you had a list of “best-kept-secrets” (websites, books, coaches) that have helped you, which would you recommend?

I like the work of Edward Tufte. There are a few good “Data science in Excel” books which I think are worth a read – they build ML models from scratch using sheet formulas alone!

Other than that, the best people who ever helped me are the people I’ve worked with – in terms of their feedback, ideas and often crazy ways of using my work (like printing out an Excel dashboard I designed onto a 4 metre wall in the boardroom. True!)

What is the number one piece of advice you give to aspiring data scientists?

Don’t assume you know the problem you’re being asked to solve. I’ve a blog article coming out on this soon but I’ve distilled this down into the “3x knows.” These are “I don’t”, “I think I” and “I do”.

Most people start at “I think I know” and it just ends in disaster more times than not. Start with “I don’t” and get the person you’re doing the work for (your “customer” i.e. the person who will judge the value of the output) to tell you in laymen’s terms what they are expecting.

Repeat this back to them in your own words. Get them to tell you again. Then get them to email you a brief. Then check again!

It is a pain but it is better to have this exchange at the beginning rather than at the end …

Where can readers find you online?

Please sign up to the newsletter at www.yourdatadriven.com if you want to hear me share some thoughts each week (there are other benefits of course!)

If you’re a pro sports team, contact me at www.paceinsights.com.

We have the usual twitter (@paceinsights @yourdatadriven @samirabid but I must confess to being a bit of a Luddite when it comes to them.

Also I’m on LinkedIn – Samir Abid.

Q&A with Randy Au, Quantitative UX Researcher

2019-03-03T00:00:00Z

Interviewed on 3rd March 2019

This Analysts Assemble interview is with self-confessed data nerd Randy Au, a Quantitative UX Researcher at Google Cloud.

Randy’s career has taken him through several big names in the tech world but it was his article in Towards Data Science, called Succeeding As A Data Scientist At Small Companies / Startups, that got my attention.

As a fellow data generalist, I wanted to dig a little deeper into Randy’s backstory and see what his path to bringing those general data science skills to Google was like.

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

It’s a weird path. Undergrad majors were Business Administration and Philosophy and I minored in some applied Math. MS was in Communications (the social science) where I did some computational linguistics and learned a bunch about social science methodology and philosophy of science. After that, I did a stint at an office design firm analyzing survey data and automating reports, a very brief stint in an ad-tech firm, then almost 6 years at Meetup the social network (analyst), 2 years at Bitly the link shortener (analyst), 1 year at Primary a kid’s clothing manufacturer/e-commerce as data engineer, before winding up in Google Cloud as a Quantitative UX Researcher.

About the only common thread through all this was I work as an internal support consultant. I didn’t make data products that users directly interacted with, so I was more of a “force multiplier” essentially. I make teams better =). All through the time, I’ve worked with every single team that existed, from engineering and product development to customer service to legal and finance. Math and data and programming is useful everywhere.

These days I use the exact same data science skills I have, and apply it to understanding users as a UXR. I still deal with cross functional teams, tons of separate systems up and down the tech stack, and lots of interesting data questions.

What’s a typical day look like for you in your current data role? (Which tools and languages do you use? Big team/small team/lone wolf? Office-based or remote?)

Typical day is still chaotic as usual =) There’s often 3-5 projects/tasks running in parallel, lots of meetings to help set metrics, get alignment, report results. There’s a team of 7 other qualitative researchers, and I’m the only quant in the local group, so it’s a lot of the “building data-driven culture” type stuff across a ton of product teams.

I’m also in a rather unique position of supporting 8 semi-related products, so a lot of prioritization discussions have to happen.

Day-to-day I’ll be using Python and various dialects of SQL, occasionally some spreadsheet magic thrown in.

You’ve built up a large following through your blogging. How important do you think it is for data professionals, at all stages of their career, to share publicly what they are doing and learning?

I think the data community in general is very open and receptive, so if someone has something to share, they should just go ahead and do it. I’ve personally spent over a decade without really sharing anything outside of twitter comments, but that eventually led to writing articles and people finding those useful (to my surprise).

Where do you see your own data career going next? Building on your technical skills as an Individual Contributor or moving into a more management-based role?

For now my goal is to move somewhat towards management. There’s always “bigger problems” that need more time and resources to analyze and make happen. Since I can’t do it alone I want to do it with the help of others.

If you had a list of “best-kept-secrets” (websites, books, coaches), which would you recommend?

I think the big cluster of people interested in data on Twitter is the best thing ever. There’s always someone around sharing a cool finding, a new tool, or collectively bonding over the insanity that is time zones. Participating in that leads to many good things.

What is the number one piece of advice you give to aspiring data scientists?

Know your data! =D

A close second is respect and leverage domain knowledge experts, which I need to write about… eventually.

Where can readers find you online?

Probably easiest on Twitter: https://twitter.com/randy_au

Q&A with Helen Anderson, BI Data Analyst and Technical Consultant

2019-02-24T00:00:00Z

Interviewed on 24th February 2019

This interview in the Analysts Assemble series is with New Zealand-based BI Data Analyst and Technical Consultant, Helen Anderson. I first ran into Helen while she was writing an excellent series on SQL on Dev.to.

I’ve since found her to be one of the most helpful and approachable members of that site and a great proponent of helping encourage everyone in the community. I’m very pleased to say she’s agreed to share her story and data journey with us here at Analysts Assemble.

Over to Helen…

Tell us a bit about yourself, how did you get into the data space and what does your data journey look like so far?

Hello! I’m Helen Anderson and I’m a Data Analyst and Technical Consultant on Xero’s Data Services team. I support Xero’s Analyst Community with code review, building datasets they need, maintaining the database they use for their work and providing guidance for junior analysts.

My data journey so far hasn’t been the most traditional one.

I didn’t study computer science or even IT at university. After graduating with a Business degree I landed in the world of Supply Chain Analysis. I really enjoyed solving the puzzle of how to get the right stuff to the right people at the right time.

Putting together a plan based on the customer’s needs, the shipping timetables and in the case of my first job shipping apples from New Zealand around the world, what the growers estimated they would harvest. Even though we did everything using Excel it set me on the path to where I am now.

Since then I’ve worked for the Royal NZ Ballet working on revenue projections based on theatre seating configurations, Timex in London planning manufacturing of watches for Europe and Icebreaker balancing stock between locations around the world.

Making the move from working in Excel to coding in SQL happened when I joined Xero three years ago. I started in the Marketing team as an analyst pulling lists for email campaigns and doing post-campaign analysis. I was pretty late to the game when it came to using SQL, but was hooked. Now I am working in the BI team in a role that allows me to support those junior and not so technical analysts, who were ‘me’ three years ago while growing my technical skills working on projects to build data models a new database platform.

What’s a typical day look like for you in your current data role? (Which tools and languages do you use? Big team/small team/lone wolf? Office-based or remote?)

For the first time in my career, I’m on a big team of analysts, developers and engineers. There are almost 30 on the Data Services team. A big difference from being the lone analyst in most of my roles so far.

When I first joined the team the imposter syndrome hit me pretty hard.

When you are the lone analyst you do what you know because there’s no one but StackOverflow to ask. I’ve spun those feelings around and made a point of asking for help and learning from those around me with many years of experience. We have an incredible team culture, everyone is happy to help and we celebrate success with a monthly Awesome Award to celebrate everyone’s good work.

We have a team of analysts who work on ad-hoc data requests, a BI team that build models that are used in reporting, a Data Engineering team that maintain the data pipelines and platforms and MicroStrategy consultants who make sure the self-service visualisation platform is user-friendly.

Because my role is to support Analysts in teams across the business I get to do a little bit of everything. Some days I’m building a custom data set in Redshift to support analysts work, showing them how to use Microstrategy to present their reports or troubleshooting the Airflow jobs that move data to the dedicated Aurora database we maintain to support their work.

I began blogging on my self-hosted WordPress site the last time I was on the job hunt. To no surprise, it wasn’t exactly flooded with new visitors.

After seeing a colleague blogging on Medium, I weighed up my options and decided to take the plunge on Dev.to. I’m really glad I did. I’m not a traditional web developer or software engineer like a lot of the community, but I still have something to offer.

My first few posts were ‘listicles’, easy to read, but felt a little too Buzzfeedy. I reassessed the tone and found a more conversational voice. The same way I’d talk to a colleague about a technical subject, but without recreating technical documentation.

Blogging is beneficial in so many ways.

You’re reinforcing your learning or understanding of your chosen technology or tool. So even if you only get a handful of views, you have still done something worthwhile.

This is all about your unique point of view. Which doesn’t mean you have to know everything on a topic. That’s why you are writing a blog post, and not rewriting technical documentation.

Even if you are beginner explaining how you are learning how to use a new tool, your perspective is important. You never know who may stumble across your post and find your explanation of a topic helps it all click into place for them.

Where do you see your own data career going next? Building on your technical skills as an Individual Contributor or moving into a more management-based role?

I’m incredibly lucky to be on a team that encourages both. Right now I’m building up my knowledge of the quirks of PostgreSQL, as that’s the flavour of Aurora database the analysts I support have moved to. I’m working on gaining more of an understanding of the data pipelines that load the data in and the AWS services we use to build the infrastructure for my own interest and to get to know more about what our DevOps and Data Engineers do.

I’m looking forward to supporting the junior analysts more as the analyst community grows and will be giving public speaking a go with my first tech talk really soon. Even though I’m not making the career change to Management I’m still able to support and teach those around me.

If you had a list of “best-kept-secrets” (websites, books, coaches), which would you recommend?

I put together a list of resources I think are great for Junior Analysts recently that cover not only the technical side with SQL but the human side too – requirements gathering, visualisation and communication.

My favourites blogs at the moment are:

Data36 – for tutorials and hands-on learning.

Simple Analytical (a previous incarnation of AnalystsAssemble.com) – for commentary on being in the data world and the ups and downs of being an analyst.

Mode – content for analysts by analysts.

Soft Skills Engineering – my favourite podcast advice show about non-technical topics.

6) What is the number one piece of advice you give to aspiring data scientists?

Being a great data analyst or scientist is more than just churning out SQL and knowing your way around the database. It’s important to learn how to listen to stakeholders and determine what it is they need from a report or dashboard. Put equal amounts of effort into learning communication skills, interpreting the story behind the numbers and presenting data in a way your end user finds the most digestible.

By honing these skills, as well as building models and your technical skills, you’ll go far.

Where can readers find you online?

You can find me blogging regularly on Dev.to, on Twitter, on LinkedIn or my own personal site.

Generalists Are The Real Data Science Unicorns

2019-02-10T00:00:00Z

Rarer Than Rocking Horse Droppings.

I started chasing unicorns early in life. 1983. Four years old. Iveagh Cinema. Watching the first movie I ever saw on the big screen – The Last Unicorn. The beautiful old-style movie theatre I saw it in has since been knocked down to build houses. The price of progress I suppose.

Little did I know that, thirty five years later, I’d be one of those on the hunt for that elusive last unicorn. Of the data science variety.

It seems like everyone is searching for their own kind of unicorn these days though. VCs want to grab the new Uber or AirBNB before the competition gets their paws on them. And in data science it’s become the go-to phrase to describe that mythical skill combination of all aspects of the dark arts wrapped up in a neat one-person meaty human package.

Data engineering. Data analysis. Software engineering. Data wrangling. Machine Learning. Artificial Intelligence. Data storytelling. Model building. Product management. People management. Soft skills. Hard skills. Tech skills.

SELECT * from above.

The Data Science Unicorn has to be proficient in all areas of data science to truly make the grade. (Or so a lot of job listings would have us believe.) Plus they have to hold at least a PhD. Possibly multiple Master’s Degrees as well. And have multiple side projects on their Github. And the right kind of internships and work experience on their resume before they even submit an application form.

And…and…and…and…

Is it any wonder so many young potential data scientists are getting turned off from the so-called “sexiest job of the 21st Century” before they’ve even left the safe bosom of their academic studies?

Where the rubber meets the road.

So how realistic is it to search for our own DS unicorns? And, more importantly, are they really what businesses need to deliver results from their data in the real world?

The modern data scientist has two options as their career progresses:

Specialist – go narrow and deep and really specialise on one particular area such as machine learning or data engineering,

Generalist – go broad and shallow and cover off more of the general bases.

The further you go in your career, the likelihood will be that you will eventually gravitate more towards option 1. There will be an inevitable attraction (and aptitude) towards one aspect of the profession over the others. This is perfectly normal and, due to the vastly different skill-sets needed for each different area, understandable.

The case for specialising – and where it makes sense.

Jeremie Harris of Sharpest Minds advises against going down the generalist route altogether. He has built up a strong working knowledge of what companies want through his work mentoring young data scientists and advises them to think first of the “kind” of data scientist they want to be.

I have a problem with this. At an early stage in their career they won’t have had enough exposure to the different areas of data science to really know which avenue they should concentrate on.

Does size matter?

Specialisation does, however, lend itself to working in very large corporate companies. They usually have the luxury of well resourced data science departments with very specific tasks for each team. If you have a data engineering team, an analyst team, a BI team and a modelling team, it makes more sense to look for new hires that fit into one of those specific pigeon holes.

They are also much more likely to be hiring you to look after a specific problem area they are facing so, of course, they’ll be more interested in a particular specialization.

That theory gets flipped on it’s head if you are running a smaller organisation, like a small company or early stage startup. With a small (or non-existent) data team, you’re much more likely to look for new hires with skill-sets that cover more of the bases at once. The luxury of hiring someone who “only” does machine learning engineering just isn’t available to you.

Full-stack. Fully loaded.

Which brings us back to option 2 – the Generalist.

My own experience of running an analytics team is very much on the small team with extremely wide remit side of things. I may be a tad biased on what I look for when recruiting because of this so best to make that clear up front. YMMV.

When you are running a team that covers business intelligence, ad-hoc reporting, analysis, regulatory reporting, model building and general data project troubleshooting and internal consulting – you can’t be picky about the salami slice of skills you might need from a team member.

On a day-to-day basis, you will need a small combination of THE LOT. Maybe not to a level that a skilled specialist who only concentrates on their own particular element offers but Just Enough Experience to Perform. That acronym of JEEP has to underlie everything you are hiring for when you never know what lies around the next corner.

Randy Au has a great quote in his article Succeeding as a data scientist in small companies/startups which sums up the “not quite knowing” what you’ll need the data team to do in your small company:

It’s usually quite likely that they don’t really have a full understanding of what they need. There’s just a generalized sense of “we have data, it seems useful, but we don’t have anyone who has the skills to make it useful.”

As Randy goes on to say, the main thing is being able to roll up your sleeves and make a difference on the ground TODAY. Then make sure that you’re building the right infrastructure to make full use of data/analytics/data science there TOMORROW. A machine learning engineer, regardless of their talent and smarts, just won’t be the right person for that particular job. You NEED a generalist.

When all is said and done.

When the shouting is all over though and the stack of CVs is sitting in front of me, I know exactly what I am looking for. Give me the analytics equivalent of the 1970s Dutch Total Footballer every time over the American Football special team kicker. Someone who can slot into any of these positions and not look too out of place. That suits my needs much more than a one trick pony, even if they are the best in the world at that one very special thing.

Then maybe, just maybe, over time and through experience, one of my Total Footballers can find they want to specialise and move on to a more suitable home for that stage of their career. Horses for courses. I’d be willing to wager that the other skills they’ll have picked up in the meantime will never be wasted even if they do go on to specialise and that, for me, makes it all the more reason to keep looking for my own unicorns wherever I might find them.

Is SQL Really A Game Changer For Data Science Careers?

2018-12-09T00:00:00Z

I’ll nail my colours to the mast right away on this one.

In my opinion, SQL is the most fundamental skill you MUST have to get started in a career in data analysis. We can argue it out until we are blue in the face over whether R is better than Python. Or whether SAS and Java are dead. Or Spark is more useful than Azure. None of that matters unless you have the lowest hanging fruit plucked and safely placed in your skills basket.

And that is SQL.

Some relevant statistical research.

Data Scientist Jeff Hale carried out an excellent piece of analysis recently on the most in-demand skills for data scientists. He wrote it up for KD Nuggets and it’s well worth a read for both aspiring and current data analysts.

Jeff scraped job postings on LinkedIn, Indeed, Monster, SimplyHired and AngelList to see which terms were showing up most regularly for various data science related job vacancies. He searched only within the United States and did exact match searches on the terms: “data scientist” “[keyword]”.

The most interesting part for me was the rankings he put together for various technology skills. It will come as little surprise to anyone who has scanned data science job vacancies in the past 6-12 months that Python, R and SQL came in 1st, 2nd and 3rd respectively.

But didn’t you say SQL was most important?

You might think I’ve shot my own argument in the foot here by using a piece of research that shows SQL as only coming in third in the list. If over 70% of listings wanted Python skills and over 60% specified knowledge of R, shouldn’t we concentrate on those first? I don’t think you should. And here’s why.

SQL is easier to pick up the basics from a standing start. Even if you only learn some basic uses of the SELECT statement you will already be ahead of 99% of Excel jockeys who never bothered to move beyond copy/paste and VLOOKUPs.
SQL crosses all areas of the wider data science world. Want to be an analyst? Learn SQL to SELECT the data you need from the data warehouse. Want to be a data scientist and run machine learning? Use SQL to clean up your data before getting into algorithms. Fancy building data pipelines as a data engineer? SQL. SQL. SQL. SQL.
RDBMS databases are here to stay. There will always be a shiny new object in the database world to distract us. (I’m looking at you MongoDB.) But the vast majority of companies and online applications still run on the big three of MySQL, SQL Server and PostgreSQL. You don’t have to have worked with them to notice something jumping out of their names at you. Hint: it’s three letters long and rhymes with prequel.

There’s another element to SQL coming in third in Jeff’s analysis though and it was brought up by Kristin Kehrer in her analysis of Jeff’s analysis. (Yes I know, it’s like turtles all the way down with analysts analysing each other’s work.)

How important is SQL?

Kristin asked how important is SQL and was surprised to find it only came in 3rd. She believes that may be explained because hiring companies see SQL as being so fundamental as to be a prerequisite when hiring data scientists.

I would wholeheartedly agree. When hiring data analysts I immediately discard the CV in front of me if they haven’t demonstrated SQL knowledge. It’s not even a question of it being desirable. It’s a necessity right out of the gate.

I was lucky enough when I fell into the data world as I came from a web and software development background. I’d been building desktop and online applications in healthcare and e-commerce and a fundamental part of that was being my own development DBA.

Not only designing and developing table structures, I was writing and optimizing queries and stored procedures in both development and production stages. Small teams need full stack skills and knowing SQL was one of the top ones I relied on: day in, day out.

SQL is a superpower.

When I first started as a data analyst in my current workplace, I just picked up where I’d left off writing SQL as a developer. The other analysts on my team looked at me as if I was some kind of witch when I was writing sub-queries and hand-coding SQL. They were still using the drag n’ drop query designer in Microsoft Access. I didn’t know any different though and it helped me hit the ground running and make a big impact early on.

The case for the defence is almost ready to rest.

I see questions like this one asked regularly on internet forums and my answer always starts and ends the same. If you want a career in data analysis, you will be doing yourself a massive dis-service by not learning SQL. Keep building on that foundation throughout your career but please learn the basics.

SQL is the gift that keeps on giving.

You will not regret it. I promise.

The MOOC System Is Broken. Where Does Online Learning For Data Science Skills Go Next?

2018-12-08T00:00:00Z

The build-up to the holiday season is well and truly upon us. And with it the customary race to get all loose ends tidied up before the end of the year. It also becomes a time for contemplation over a few glasses of mulled wine as we look back at the year we’ve just slogged through. And, just as importantly, a look ahead to how we want to step things up next year in our data careers.

As it’s easy to get left behind by standing still, the most common method for anyone looking to up their data skills is to take a training course. (Don’t panic at this point. I’m not trying to sell you anything. This isn’t one of those soft intros before the hard sell kicks in and the sob story of my kids facing a present-less Christmas begins.)

Learning By Doing.

I have been thinking a lot recently though about the changes in learning in the data and analytics industry. I wrote an [article][1] about my own experiences and the different learning methods I’d tried over the past 10 years or so. From MOOCs to books, expensive in-person courses to Just F*cking Doing It, I’ve done the lot with varying levels of longer term success.

Check out that article and see how much of it resonates with you – I’m willing to bet quite a bit if you are anything like the other data analysts I’ve spoken to about it.

What Isn’t Working?

What kicked my thoughts on learning off was a podcast called Level Up Your Course (episode 073), hosted by Janelle Allen. Janelle’s guest is online entrepreneur and course creator Justin Jackson. They had a very interesting conversation about the reasons why online courses and MOOCs aren’t working for the vast majority of people.

I won’t ruin the podcast as it’s well worth a listen if you have a spare hour on your commute / walk / gym session. But it really boiled down to two things: 1) Lack of accountability and 2) Lack of consequences for failure.

I subscribe to a LOT of email newsletters in the data science world. So recently my inbox has been filled to bursting point with everyone and their dog trying to hawk their paid “How To Get Started In Data Science” courses. I get that we all have to put bread on the table. And I’m all for experienced people in the industry passing that knowledge on to a hungry new generation. I don’t write articles (only) for my own narcissistic pleasure. I do legitimately want to help people learn from my own experiences. And I know the majority of others offering these courses and bootcamps do too.

I’ll Say It. MOOCs Don’t Work For The Vast Majority Of People.

What I’m conflicted about is how to successfully empower that learning for people if we know that less than 2% of all MOOCs purchased (in all domains, not just DS admittedly) actually get completed.

What is the best way to give new entrants enough of a grounding to get interviews, jobs and make a new life for themselves?

I don’t think the current system of online self-monitored learning works. It clearly doesn’t. But then, increasingly, neither does the old model of paying thousands upon thousands of pounds/Euros/dollars to a traditional university. They force accountability on you and there is a consequence for not doing the work (i.e. flunking out). But they usually consider their job done as soon as the scroll is handed over. Then you’re just a name on a mailing list being asked for alumni donations each year.

Where do we go next?

I’m always eager to hear the thoughts and experiences of working analysts on this topic. What has worked for you? What hasn’t? And how do you see the world of learning new data skills developing over the next few years?

Maybe The Students Aren’t The Problem.

I’m not advocating throwing the baby out with the bathwater. Online learning has put the entirety of human knowledge in the hands of anyone with a smartphone and an internet connection. And any adult has to have the self-discipline within themselves if they want to keep improving their skills.

But as educators (and anyone passing on the smallest piece of knowledge to someone who doesn’t know it IS an educator), we have to take responsibility to not just sell dreams of magic beans to willing marks. Helping the student actually stick with the material and get the positive benefits at the business end of the course HAS TO BE the main goal. Not just the warm fuzzy feelings and dreams for them at the beginning when the credit card number has been typed in and the Transaction Successful message pops up.

We owe people more than that. Now we all just have to find a way to provide it.

WTF Is Web Scraping?

2018-11-03T00:00:00Z

Web scraping. Screen scraping. Web harvesting. Web data extraction. Web indexing. Web mining. Web crawling.

So many names but what does it mean and how can we get started doing it?

So what is web scraping?

Information is power. And getting our hands on information through the medium of data is the way we get to harness that power. With the growth of the World Wide Web over the past twenty years, there is now an unimaginable amount of data (almost) freely available to us.

There are several methods for extracting data from websites, the main one being the use of official APIs (Application Programming Interface). Larger companies usually prefer to offer an API to users (for a price of course) as they can control the structure, content and delivery of the data they want to make available to 3rd parties.

When they don’t, we have to get a little more creative in our extraction attempts.

This is where web scraping comes in.

What does web scraping do?

In basic terms, we know that web pages are built using text-based mark-up languages such as HTML. This collection of HTML tags tells our web browser how to display the content within the web page. What it means is that we can use our knowledge of HTML to look at the source code, work out the structure of the page and find where the data we want is located.

Of course we COULD manually look at the page in our browser and do a lot of manual copying and pasting. Brute force sometimes is the best option when it’s a limited dataset or it’s a one-off job. But where’s the fun in that?

What we want to do is write is some code that fetches the HTML from the web page, scans through it for the important parts containing the data we want, extracts that and puts it into a usable format we can work with.

This is essentially what Google does when it’s crawlers go out to index the web.

Why would we do it?

If an API is not available then there is often little other choice than to set up a web scraper. I’ve used web scraping in the past to extract football results for football leagues all around the world on a daily basis. I then stored them in a SQL Server database and used it in a prediction model I’d built.

Other possible uses are: price comparison sites, aggregated job listings, property websites. Really anything where data has to be accumulated from multiple sources without any other way to get it or where the data updates so quickly it is of greater benefit to scrape a site to get it.

Why would we not do it?

Truthfully, web scraping is a bit of a legal grey area. It’s generally against the terms of use of most websites although the appetite of the courts to enforce that is changeable to say the least.

In the United States, large companies such as eBay, Craigslist and American Airlines have taken court action against scrapers on three fronts: 1) copyright infringement, 2) violation of the Computer Fraud and Abuse Act, and 3) trespass.

Cases have also been taken in courts in Denmark, Ireland and Australia with varying decisions by the local judiciary.

Web companies use a wide range of methods to prevent web scraping, beyond legal action, and, from experience, they will be more of a deterrent in your early scraping ventures than any fears of landing in court.

Blocking IP addresses, monitoring user agent strings and high traffic monitoring bots are just some of the tools available to stop your trusty web crawler bot in his/her/it’s tracks.

What do we use to start web scraping?

There are a range of different approaches we could take to build a web scraper, ranging from VBA in Microsoft Excel to Javascript tools in jQuery or Node.js.

My preferred method is to use Python. Using the vast array of Python libraries and packages at our disposal, we can quickly code up a basic web scraper and test it out live on a web page.

Libraries such as BeautifulSoup make scraping super simple. BeautifulSoup “provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree”. Er…it lets you dig through the page data, find what you want and do something with it. Let’s not confuse the issue any more than we have to…

How To Web Scrape Wikipedia Using Python, Urllib, Beautiful Soup and Pandas

2018-11-03T00:00:00Z

In this tutorial we will use a technique called web scraping to extract data from a website.

We’ll be using Python 3.7 through a Jupyter Notebook on Anaconda and the Python libraries urllib, BeautifulSoup and Pandas.

What is Web Scraping?

Web scraping (also known as screen scraping, data scraping, web harvesting, web data extraction and a multitude of other aliases) is a method for extracting data from web pages.

I’ve done a quick primer on WTF Is…Web Scraping to get you up to speed on what it is and why we might use it. Have a quick read and re-join the tour group as soon as possible.

Step By Step Tutorial

OK. Now we know what web scraping is and why we might have to use it to get data out of a website.

How exactly do we get started scraping and harvesting all of that delicious data for our future perusal and use?

Standing on the shoulders of giants.

When I first tried screen scraping with Python I used an earlier version of it and worked through Sunil Ray’s Beginner’s Guide on the Analytics Vidhya blog.

Working with Python 3.7 now I had to change some libraries plus do a few further corrective steps for the data I’m looking to get hence not just pointing you straight to that article.

Which Python libraries will we be using for web scraping?

Urllib.request

As we are using Python 3.7, we will use urllib.request to fetch the HTML from the URL we specify that we want to scrape.

Learn more about urllib.request.

BeautifulSoup

Once urllib.request has pulled in the content from the URL, we use the power of BeautifulSoup to extract and work with the data within it. BeautifulSoup4 has a multitude of functions at it’s disposal to make this incredibly easy for us.

Learn more about Beautiful Soup.

Anything else we need to know before we kick this off?

Are you familiar with HTML?

HTML (Hypertext Markup Language) is the standard markup langauge for creating web pages. It consists of a collection of tags which represent HTML elements. These elements combined tell your web browser what the structure of the web page looks like. In this tutorial we will mostly be concerned with the HTML table tags as our data is contained in a table. For more reading on HTML, check out W3Schools Introduction to HTML.

Right, let’s get into it.

1. Open a new Jupyter notebook.

You do have it installed, don’t you? You didn’t just skip the advice at the top, did you? If so, go back and get that done then come back to this point.

2. Choosing our target Wikipedia page.

Like our friend Sunil, we are going to scrape some data from a Wikipedia page. While he was interested in state capitals in India, I’ve decided to pick a season at random from the English Premier League, namely the 1999/2000 season.

When I went and looked at the page I instantly regretted picking this season (fellow Tottenham Hotspur fans will understand why when they see the manager and captain at the end but I’ll stick with it as some kind of sado-masochism if nothing else).

3. Import urllib library.

Firstly, we need to import the library we will be using to connect to the Wikipedia page and fetch the contents of that page:

# import the library we use to open URLs
import urllib.request

Next we specify the URL of the Wikipedia page we are looking to scrape:

# specify which URL/web page we are going to be scraping
url = "https://en.wikipedia.org/wiki/1999%E2%80%932000_FA_Premier_League"

Using the urllib.request library, we want to query the page and put the HTML data into a variable (which we have called ‘url’):

# open the url using urllib.request and put the HTML into the page variable
page = urllib.request.urlopen(url)

4. Import BeautifulSoup library.

Next we want to import the functions from Beautiful Soup which will let us parse and work with the HTML we fetched from our Wiki page:

# import the BeautifulSoup library so we can parse HTML and XML documents
from bs4 import BeautifulSoup

Then we use Beautiful Soup to parse the HTML data we stored in our ‘url’ variable and store it in a new variable called ‘soup’ in the Beautiful Soup format. Jupyter Notebook prefers we specify a parser format so we use the “lxml” library option:

# parse the HTML from our URL into the BeautifulSoup parse tree format
soup = BeautifulSoup(page, "lxml")

5. Take a look at our underlying HTML code.

To get an idea of the structure of the underlying HTML in our web page, we can view the code in two ways: a) right click on the web page itself and click View Source or b) use Beautiful Soup’s prettify function and check it out right there in our Jupyter Notebook.

Let’s see what prettify() gives us:

6. Find the table we want.

By looking at our Wikipedia page for the 1999/2000 Premier League season, we can see there is a LOT of information in there. From a written synopsis of the season to specific managerial changes, we have a veritable treasure trove of data to mine.

What we are going to go for though is the table which shows the personnel and kits for each Premier League club. It’s already set up in nice rows and columns which should make our job a little easier as beginner web scrapers.

Let’s have a look for it in our prettifyed HTML:

And there it is. (NB. since I first wrote this tutorial, Wiki has added another table with satadium name, capacity etc. that also has this class identifier. We’ll allow for that further down in the code.)

Starting with an HTML <table> tag with a class identifier of “wikitable sortable”. We’ll make a note of that for further use later.

Scroll down a little to see how the table is made up and you’ll see the rows start and end with <tr> and </tr> tags.

The top row of headers has <th> tags while the data rows beneath for each club has <td> tags. It’s in these <td> tags that we will tell Python to extract our data from.

7. Some fun with BeautifulSoup functions.

Before we get to that, let’s try out a few Beautiful Soup functions to illustrate how it captures and is able to return data to us from the HTML web page.

If we use the title function, Beautiful Soup will return the HTML tags for the title and the content between them. Specify the string element of ‘title’ and it gives us just the content string between the tags:

8. Bring back ALL of the tables.

We can use this knowledge to start planning our attack on the HTML and homing in only on the table of personnel and kit information that we want to work with on the page.

We know the data resides within an HTML table so firstly we send Beautiful Soup off to retrieve all instances of the <table> tag within the page and add them to an array called all_tables:

# use the 'find_all' function to bring back all instances of the 'table' 
# tag in the HTML and store in 'all_tables' variable
all_tables=soup.find_all("table")
all_tables

Looking through the output of “all_tables” we can again see that the class id of our chosen table is “wikitable sortable”. We can use this to get BS to only bring back the table data for this particular table and keep that in a variable called “right_table”. As I said above, there is now another table with this classname in the HTML, we’re going to use find_all to bring back an array and then look for the second element in the array which we know is the table we want:

right_table=soup.find_all('table', class_='wikitable sortable')
right_table[1]

9. Ignore the headers, find the rows.

Now it starts to get a little more technical. We know that the table is set up in rows (starting with <tr> tags) with the data sitting within <td> tags in each row. We aren’t too worried about the header row with the <th> elements as we know what each of the columns represent by looking at the table.

To step things up a notch we could have set BeautifulSoup to find the <th> tags and assigned the contents of each to a variable for future use.

We’ve got enough to get getting on with getting the actual data though so let’s crack on.

10. Loop through the rows.

We know we have to start looping through the rows to get the data for every club in the table. The table is well structured with each club having it’s own defined row. This makes things somewhat easier.

There are five columns in our table that we want to scrape the data from so we will set up five empty lists (A, B, C, D and E) to store our data in.

To start with, we want to use the Beautiful Soup ‘find_all’ function again and set it to look for the string ‘tr’. We will then set up a FOR loop for each row within that array and set Python to loop through the rows, one by one.

Within the loop we are going to use find_all again to search each row for <td> tags with the ‘td’ string. We will add all of these to a variable called ‘cells’ and then check to make sure that there are 5 items in our ‘cells’ array (i.e. one for each column).

If there are then we use the find(text=True)) option to extract the content string from within each <td> element in that row and add them to the A-E lists we created at the start of this step. Let’s have a look at the code:

A=[]
B=[]
C=[]
D=[]
E=[]

for row in right_table[1].findAll('tr'):
    cells=row.findAll('td')
    if len(cells)==5:
        A.append(cells[0].find(text=True))
        B.append(cells[1].find(text=True))
        C.append(cells[2].find(text=True))
        D.append(cells[3].find(text=True))
        E.append(cells[4].find(text=True))

Still with me? Good. This all should work perfectly, shouldn’t it?

We’re looping through each row, picking out the <td> tags and plucking the contents from each into a list.

Bingo. This is an absolute gift. Makes you wonder why people make such a fuss about it, doesn’t it?

11. Introducing pandas and dataframes.

To see what our loop through the Personnel and Kits table has brought us back, we need to bring in another big hitter of the Python library family – Pandas. Pandas lets us convert lists into dataframes which are 2 dimensional data structures with rows and columns, very much like spreadsheets or SQL tables.

We’ll import pandas and create a dataframe with it, assigning each of the lists A-E into a column with the name of our source table columns i.e. Team, Manager, Captain, Kit_Manufacturer and Shirt_Sponsor.

import pandas as pd
df=pd.DataFrame(A,columns=['Team'])
df['Manager']=B
df['Captain']=C
df['Kit_manufacturer']=D
df['Shirt_sponsor']=E
df

Let’s run the Pandas code and see what our table looks like:

Hmmm. That’s not what we wanted. Where’s the Manager and Captain data?

Clearly something went wrong in those cells so we need to go back to our HTML to see what the problem is.

12. Searching for the problem.

Looking at our HTML, there does indeed seem to be something a little different about the Manager and Captain data within the <td> tags. Wikipedia has (very helpfully/unhelpfully) added a little flag within <span> tags to help display the nationality of the Managers and Captains in question.

It sure looks nice on the Wiki page but it’s messing up my screen-scraping tutorial so I’m somewhat less than happy to have it in there.

Using the knowledge we’ve gained above, is there a simple way to workaround this problem and just lift out the Manager and Captain names as we planned?

This is how I decided to do it.

Looking at the HTML code, I can see that there are two sets of <a> tags i.e. hyperlinks within each cell for both the Manager and Captain data. The first is a link over the flag’s <img> tag and the second is a link on the Manager/Captain’s name.

If we can get the content string between the <a> and </a> tags on the SECOND of those, we have got the data we need.

I did a ‘find_all’ within the individual cells to look for the <a> tags and assign that to a variable (mlnk for Managers, clnk for Captains). I knew it was the second <a> tag’s content string that I needed to get the name of the Manager and the Captain so I appended the content of the second element in the mlnk/clnk array I had created to the specific list (list B for Managers, list C for Captains).

As so:

A=[]
B=[]
C=[]
D=[]
E=[]

for row in right_table[1].findAll('tr'):
    cells=row.findAll('td')
    if len(cells)==5:
        A.append(cells[0].find(text=True))
        mlnk=cells[1].findAll('a')
        B.append(mlnk[1].contents[0])
        clnk=cells[2].findAll('a')
        C.append(clnk[1].contents[0])
        D.append(cells[3].find(text=True))
        E.append(cells[4].find(text=True))

Now run that and re-run our pandas code from before and “hopefully” we’ll fill in those blanks from the previous output:

Hurrah!

We now have 20 rows for the 20 clubs with columns for Team Name, Manager, Captain, Kit Manufacturer and Shirt Sponsor. Just like we always wanted.

(I’ll ignore the names in the Manager and Captain columns for Tottenham, must research my examples better before getting started…)

Want an even easier way to do this using just pandas?

Eternal thanks to reader Lynn Leifker who has sent me an even quicker and easier way to scrape HTML tables using only pandas. You’ll still have to do some HTML investigation to find which table in the overall page code you are looking for but can get to the outcome quicker using this code:

import pandas as pd
url = "https://en.wikipedia.org/wiki/1999%E2%80%932000_FA_Premier_League"
tables = pd.read_html(url) 
tables[3]

Hurrah once again and a big thanks to Lynn for the top tip!

Wrapping Up

That successful note brings us to the end of our Getting Started Web Scraping with Python tutorial. Hopefully it gives you enough to get working on to try some scraping out for yourself. We’ve introduced urllib.request to fetch the URL and HTML data, Beautiful Soup to parse the HTML and Pandas to transform the data into a dataframe for presentation.

We also saw that things don’t always work out just as easily as we hope for when working with web pages but it’s best to roll with the punches and come up with a plan to workaround it as simply as possible,

If you have any questions, please send me a mail (alan AT alanhylands DOT com). Happy scraping but if you get caught…we never met!

WTF Is A Data Lake?

2018-09-14T00:00:00Z

Let’s jump in and get the boring, dry, formal part out of the way first.

WHAT IS A DATA LAKE – THE FORMAL DEFINITION

According to Wikipedia, a Data Lake:

Is a system or repository of data stored in its natural format, usually object blobs or files.
Is usually a single store of all enterprise data. This can include raw copies of source system data and transformed data used for tasks such as reporting, visualization, analytics and machine learning.
Can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and binary data (images, audio, video).

WTF DOES THAT MEAN?##

In simpler terms:

Storage is cheap. Businesses are generating vast quantities of data every day. Everyone knows there is money to be made mining all of this data so they decide to just keep everything. You never know what might turn out to be useful in future. So they keep ALL of it.

Structured data – like in your traditional Relational Database System like SQL Server. Log files from web traffic. CSV files of transcripts of customer complaint calls – hell even the recorded audio of the calls themselves. ALL OF THE DATA.

Stick it all together in a great big box (let’s call it The Hadoop) and work out how we are going to do something with it later.

This, my friends, is a data lake.

Sounds like a hoarder’s paradise?

That’s because it is.

We all know the kind of people who like to keep every old item of clothing or broken piece of electronics they’ve ever owned. Just in case it comes in useful in future.

Their attic is full. Their wardrobes are overflowing. There is a maze of old newspapers stacked high all around their house.

And yet they can’t/won’t throw any of it out. JUST IN CASE.

It gets to the point where they have to move some of their junk/stored treasures to a self-storage facility as the house is overflowing. Now they’ve filled a shipping container as well and still more stuff is arriving through the front door every day.

There’s an analogy in there somewhere.

Businesses can find themselves in the same position. They can become so focused on not missing any data that might become useful and just bung it all into their virtual cupboard until the door won’t close. Cloud storage services are usually the next level (the shipping container if you will) once they’ve filled the on-premises storage box.

Sounding more like a data ocean than a data lake.

When it gets so stuffed full of every conceivable kind of raw, unstructured data, it descends into more of a data swamp than a data lake.

It can get so that analysts and data scientists are spending more time digging around fruitlessly in the dirt hoping to find a small nugget of gold than actually adding any real value to the business.

Sounds horrible. Why would we use a data lake at all?

Horses for courses.

What works at the gigabyte storage level won’t necessarily work at the petabyte level and this can be where data lakes come into their own.

The gaming industry is an example where so many data points are generated and stored per game that relational databases sometimes can’t keep up. When that data needs analysed further down the line, data scientists can appreciate the flexibility the data lake’s lack of defined structure gives them.

Deep analysis with constantly changing structure and requirements can be accommodated more quickly and easily in a data lake than in a traditional data warehouse. It really depends on what the business actually plans to use the data for.

How can we get the best value out of our data lake?

With some forethought and consideration there are definite ways to get value-for-money and business benefits out of your data lake.

Some prerequisites to think about before jumping on a multi-million dollar data lake bandwagon:

What are your business use cases? What are you really looking to get out of this lake that you can’t get from your existing data warehouse?
Does it have to be data lake OR data warehouse? No, they can co-exist. If it’s one or the other, again, consider what you will be using it for. It’s not one size fits all especially with the investment in time and money that goes into either choice.
Is a vendor with some self-serving interest at play pushing you down this road? Some slick salesman from one of the big players been taking your CTO out to dinner and suddenly a new data lake is on the table? Buyer beware!
Does your business have the necessary analytics and data science skill-sets already available to make the most out of this? If not, you’re adding in another major cost item to the whole project. It’ll also take a great leap in business thinking to accommodate the data driven agenda they will push. It’s this aspect of it all that makes me think of the Meat Puppets song “Lake of Fire”. Mostly because it’s not just the analysts using the new data lake that can start to feel the heat radiating out from it.

Examples of Data Lake providers

Microsoft Azure
Amazon S3
Apache Hadoop

Case Study: Using Your KPI Scorecard To Solve Small Business Problems

2018-09-09T00:00:00Z

Digging for KPIs at the coalface.

I’ve been measuring KPIs recently as we move forward with my wife’s boutique fitness studio business. As well as being a damn fine corporate data analyst, I wear quite a few hats for this business and one of them right now is measuring exactly where we stand.

We moved into new premises recently and have ramped up the number of classes and services we offer. What we measured at the start of the year when Elizabeth started running fitness classes is not what we need to be measuring now we have staff, rent, rates, music licensing fees and a multitude of other outgoings.

What is the rest of the industry measuring?

I researched what other studios measure, why they do so and which metrics I’d put down in my big wish-list might actually be vanity metrics for this industry. Being able to get some peer feedback from other fitness studio owners has been vital. While the only metric that really matters is whether there is enough cashflow to meet the outgoings, there are very real business problems that I can represent by a simple “Big 5” KPI dashboard.

In this case, I’ve concentrated on:

Member retention.
Revenue per square foot in our studio.
Average class size.
Average cancellations.
Revenue per member.

I’ll revise these after a few more months-worth of data comes through but they already give me a finger on the pulse of what’s happening with the business and where we need to shake things up.

What did we find when we started measuring these KPIs?

When we started to track these KPIs we were able to spot some trends that were starting to grow within the business. One worrying one was a steady increase in the number of class booking cancellations.

We had allowed members to book into classes up to 14 days in advance to allow them to plan out their schedules better in advance. This meant many were booking their next fortnight of classes but were forgetting which they had booked by the time the second week came around.

We decided to bring this back in to 7 days which was more restrictive on our members. At the same time, I implemented an email reminder notice that goes out the day before a class and also beefed up the information on the Manage Your Bookings section of our booking system.

How did it work out (pun intended)?

Despite having their time to book classes reduced by 50% we’ve received many positive responses from members about how the reminders and improved UX on the booking system had really made it more user-friendly and helpful for them. Cancellations dropped and continue to drop, average class size has increased and customer satisfaction is (anecdotally) up.

None of that would have been possible if we hadn’t made measuring cancellations one of our big 5 KPIs.

It will be a matter of trial and error in any size of business to find which KPI measures you should focus on. The low hanging fruit such as number of customers over the door might be #1 in the early days but less important 12 months down the line.

Measuring what turns out to be the wrong thing (or the less good thing) is a learning in itself. Analytics is all about the iteration and (not so surprisingly) so is business.

The only bad way to do KPIs is to not do KPIs at all.

Picking Your Big 5 KPIs To Measure Corporate Business Success

2018-09-02T00:00:00Z

I usually despise acronyms. Completing TPS reports. Filling in MAD-C forms. Horrible abbreviations invented by boring people to make themselves feel smarter than the rest of us who don’t know what the hell they are talking about.

They are the devil’s spawn.

Except in this case.

There is no way I’m typing out Key Performance Indicators every time I reference them in this article so KPIs it shall be from here on in.

So what are KPIs and why do we need them?

KPIs are specific, measurable values that show how effectively an organisation is meeting it’s key business objectives.

Basically, are we doing what we want to be doing in the areas we think we should be doing it in?

They are INDICATORS so they should tell us something.

And not just anything but something about our PERFORMANCE.

And not just any aspect of our performance – the magic word for me out of the three is KEY.

I’m kind of a big wheel on Facebook…

I’m not interested in vanity metrics. It astounds me how often I see them trotted out as the reported big “wins” for so many companies. Large corporates should know better but, alas, the public appendage measuring goes on there too.

Facebook likes. Instagram followers. Monthly “web hits” (is it 1999 again?) They all may be interesting metrics if you’re bored (maybe). Do any of them help you really move the real needle though?

We all have limited time to take care of business and a world of competing measures vying for our attention. So you have to ask yourself – are these really the key indicators showing us how we are doing against our most important business goals?

What are our business goals?

Every business will be different so it will be down to your own business knowledge as a data analyst (and your business partners and colleagues) to work out just what exactly you should be focusing on in the first place.

Early stage start-ups may be looking to prove out their business model and get their user numbers up to a critical mass as quickly as possible.

Enterprise software companies might focus on their market share percentage as new competitors enter their market.

Healthcare trusts may want to know how long their patients have been on waiting lists before getting their operations.

It will be different for every business but the main thing is to work down from the top of the organisation. In a corporate environment it’s imperative that you get the heavy hitting support of the executive or senior management level first. They know the strategic direction the company is taking and what targets they have to hit (be it in terms of users, active customers or Cost of Sales to Revenue ratio).

Behind every good KPI is a real business problem that needs properly sourced data to support it’s measurement. If we can’t measure it then we can’t improve it. If we can’t improve it and it really is one of our KEY objectives then, to be quite honest, we’ve dropped the ball and there is nowhere to hide when the shit eventually hits the fan.

Can we just measure everything then?

Well, yes, of course you could. But then you could stick your fingers in an electric socket to jolt you awake in the morning rather than take a second cup of coffee as well. Just because you could doesn’t mean you should. And that definitely applies here.

I am a big proponent of SIMPLICITY.

Regular readers will recognise my never-ending drive to simplify matters that often get completely bogged down in a mire of unnecessary complexity.

I propose you pick no more than 5 main KPI metrics to measure the performance of your overall business.

Pick your BIG 5 and name them out loud.

“But that’s not enough!” I hear you say. Betsy in Accounts wants to know this and Robbie in Marketing wants to know this, this and this and Pepe in Customer Service wants to know etc. etc. etc.

Here’s some news hot off the press. It’s perfectly fine for each department or area in your business to have their own Big 5 KPI scorecard. In fact, I insist that they do.

Obviously, the metrics that matter most to Marketing are not going to be the same ones that will matter most to your Accounts team. What they will all have to understand though is that there is a bigger picture and, first and foremost, that’s what I’m addressing here.

The concepts can remain the same. The audience will just change.

You will still need subject matter experts from the business area to guide the focus and dig down into the specific business problems of that distinct area. Resist the urge though to throw the kitchen sink at a dashboard if at all possible. I beg you. Stay simple. Which doesn’t mean easy. It just means it cuts out the superfluous shit that gets tacked on when more and more middle managers stick their oar in on anything.

All organisations should be measuring their Big 5 KPIs on a regular basis if they have plans to keep their business running towards any kind of success.

At the C-level you might even find that the Big 5 is too much information for them but I’d stick to 5 for now regardless. Even if it’s used to give a quick notification to your executive level that things are on the right track, it’s worth it.

Just one thing.

I know of one very senior executive who, when being presented with a detailed twenty page slide deck by a very talented analyst team, asked them to stop on page 2 and asked where her one number was.

“Your one number?” they replied quizzically.

And she told them that as far as she was concerned, there was one number, one metric, that mattered to her and her business division and that was it. The rest was fluff for her senior managers and middle management layers to dig into but all she needed was the one number.

I scoffed at the lack of sophistication when I first heard the story but over time I’ve grown to think she was spot on in her analysis – for her level. Maybe there is just one main metric to keep an eye on. The real problem is finding out what that one metric actually is.

It does remind us though that any KPI dashboard or scorecard is merely a quick state of the nation starting point to keep our finger on the pulse of the business. By all means dig deeper off the back of what you learn from watching the trends it reports but don’t over-complicate it.

For your own sake, I say keep it simple. Always.

What Should I Do If My BI Project Gets Ignored?

2018-08-26T00:00:00Z

Let’s see if you recognise this one.

You’ve spent weeks deep in the bowels of a BI request.

You’ve chased down the data sources, you’ve wrangled those mothers and made sure the ETL pipeline is flowing freely.

The midnight oil has been burned.

You’ve made sure everything is where it’s supposed to be and looking just like you wanted it to.

You’ve been:

dashboard building
perfecting the visualisations
and making sure the reports don’t just display some numbers – they sing them loud and proud for all to hear.

The magic day of delivery arrives and, although you’re scared of letting your baby out into the big bad world, you know it’s time.

You want to see it soar like the proud eagle you know it is.

So you send it live and…

Crickets. Tumbleweed.

Not so much as whimper.

Certainly not a thank you.

Hell, after a few days you’d take a stakeholder walking up to you and slapping you in the face. Even if it’s for handing them such a steaming pile of dog poo, you’d take it.

But you don’t even get that.

No news is good news, isn’t it?

Well no, not in this case but (not so) deep down you already know that.

Time to check the usage stats to see what exactly has been going on. That’s when the big fat zero in the active and lifetime users columns hits you like a hammer.

Right in the fleshy sore parts.

They haven’t even looked at it.

The animals. The scoundrels. The useless dirty *%!*!*%!*@ers!

All of that time wasted and it’s not like you just magically decided to spend weeks building something they might want:

THEY ASKED FOR IT IN THE FIRST PLACE!!

You have two choices at this point:

break down in floods of salty, bitter tears, rage quit your job and go off to tend a garden in a monastery somewhere in Nepal

learn one of the most important lessons of a BI analyst’s career – how to make sure what you build is what the business really needs.

As I’m not writing this using my own blood for ink on papyrus scrolls in the Himalayas you can assume I prefer option #2.

Let’s dig deeper.

Why did this happen to you?

Let’s consider why your dashboard might have been ignored in the first place.

1) Timeliness

We all know that BI reports are only as important as the amount of time left until they are due to be used in an executive pack or a regulatory return. If said time period is less than a day then the importance factor reaches maximum levels.

Did you take the original request away and spend four weeks building a full end-to-end BI solution to cover all eventualities?

It might just be that you took too long over it.

It happens.

Agree drop dead deadlines upfront and get some actual confirmation that the deadline being claimed is actually a proper deadline.

If you sit on for weeks after it should have been in then you’ve missed the boat. It doesn’t matter how good your actual dashboard delivery is.

This ship has sailed without you, my friend.

2) Complexity

It’s a well-known fact that as you keep adding more bells and whistles, the likelihood of anyone ever using it starts to fall rapidly to zero.

Stakeholders will always keep adding on more elements as you go through requirements gathering with them. But you know they’ll give no thought to whether they will actually be able to decipher what the final report is telling them.

That’s where you need to make your expertise felt.

I am all about the simplicity. This site isn’t called SIMPLE Analytical for nothing.

Take a long look at the dashboard you’ve produced and put aside your parental feelings of love for it. From a cold analytical viewpoint, have you went overboard and made it too complicated for the average user to understand?

Not the average BI analyst. The average BUSINESS user.

Remember who you are doing this for.

Just because it makes perfect sense to you doesn’t mean it won’t look like the control panel on the Space Shuttle to them.

Better still, ask your users when you deliver it for some specific feedback. Sit with them for half an hour and talk it through. Make sure they feel comfortable asking questions if they don’t understand something.

When the choice between simple and complex arises again in future, think simple. Every time.

Your users will thank you (maybe quietly and internally but they will appreciate it).

3) You didn’t ask the customer what they REALLY wanted

Yes, you may have scanned their job request form and gleaned the bare minimum of what they seemed to be asking for.

What you then went and did was take it upon yourself to decide what they really wanted. Even better, what they really should have wanted to make their working life that extra bit more special than it already is.

Problem there being that you aren’t a mind reader.

There is no substitute for actually sitting down (or getting a phone call) to go through the requirements with the customer.

I repeat: NO SUBSTITUTE.

But this is how we’ve always done it.

There is no excuse for locking yourself away in a tower and blindly driving on with a long development process on your own.

The general gist of proper Agile project management seems sound to me. Short development sprints, work on well defined chunks of requirements and regular show-and-tell sessions should be no-brainers in most situations.

When it comes to re-doing a badly specced job – Avoidance is always better than cure.

4) No champion

Do you remember back at the start when we looked at the problem that got us into this mess? Back about the time you hit Submit and just sent the dashboard live into production?

Did you notice how there was no mention of sending any comms out to stakeholders who might be interested in what you’ve built?

Did you not think that was strange? Of course it was. But, be honest, how often have you done it?

I know I have.

Sometimes you get so sick of a job that you just want it cleared off your plate. But if you feel like that about it, why should the customer feel any differently?

Let’s say you have:

completed your requirements gathering phase correctly
set a realistic and desirable timeline for the job
AND kept the complexity to a bare minimum

Shouldn’t you already have your stakeholder pre-sold on the dashboard being a major success?

They should be shouting from the rooftops about what they are going to get their hands on.

They should be out evangelising you and your dashboard long before it hits their screen.

What if they aren’t?

If you don’t have at least one champion on the business side of your request before it lands then you better double down on your post-delivery meet and greet plans.

Set up at least one feedback session about a week after you have handed over the keys to the new reports. Invite the main stakeholders to attend and encourage them to give proper feedback. Don’t let them off the hook easily if they haven’t even looked at it.

Sometimes you have to be your own champion to ensure take-up actually happens.

Better to carve out the time to do that than sit weeping because you delivered your best work and no-one cared enough to appreciate it.

How should this all work in the real world?

So how far do you have to take it in terms of:

getting the reports out quickly enough
talking to the stakeholders before you build anything
talking to the stakeholders after you build something
then making sure all the way through that it’s simple and easy enough for them to understand and use?

Each job will be different and it’s experience in the role (and your own company) that will dictate the amount of hand holding that will be required.

I understand it’s usually the coding and development phase that gets analysts interested in a project. The last thing you want to do though is ruin all of the great work you put in by telling yourself it’s the customer’s problem if they don’t wind up using what you built for them.

It’s not, it’s your problem too.

Maybe not just today when the feelings of frustration are overwhelming you either.

It might be further down the road when, despite the great work you’ve produced, some pen-pusher comes along when job cuts are being made. If they can’t find anyone at a senior enough level to vouch for the value you’ve produced over the years, what do you think the result will be?

Visibility is key.

If you’re locked away tinkering in your ivory tower all the time, to the untrained corporate eye, you might as well not be there at all.

You have been warned.

Take ownership of your project and your end result.

If you don’t, who will?

How Do I Prioritise When Everything Is Top Priority?

2018-08-18T00:00:00Z

OK, hands up in the room if you work in a peaceful, serene, non-stressful, slow-paced environment?

The kind of place where every data request is politely sent in with all details, criteria and contingency well documented.

Even better, you also get an honest appraisal from the requestor of the real urgency of the request when compared to everything else that is being asked of you right at this very minute?

Don’t be shy now, we’re all friends here. Get those hands up nice and high.

Oh. Right. No-one then.

Probably because such a utopia doesn’t really exist. If you think this is how your place operates, please drop me a note.

I need to hear more. Really.

Meanwhile back in the real world

In reality, it’s more likely that your work reception process and the priority setting discussion that goes with it resembles something a bit like the Battle of the Bastards from Game of Thrones.

Too many people piling in. Dead bodies piling up. And you in the middle of it all feeling like you are suffocating under the never-ending onslaught of urgent priorities you are having to face off against.

(Does that make me Littlefinger?)

I may sound like a broken record continually going back over the same topic of dealing with the urgency (or perceived urgency) of demands that will come at you in the most stressful of ways as a data analyst.

We’ve covered some tactics for dealing with a difficult customer who makes their own poor planning your emergency.

As part of the Frustrations Of The Data Analyst Life series, I took another go at that in the event that they keep on doing it over and over again and how to deal with that situation.

What I’m talking about now goes beyond the single pain in the arse customer who keeps pulling the same Urgent! Urgent! Urgent! shit on you. This is when you find EVERYONE in the company does it – all of the time.

When you are constantly being bombarded by TOP PRIORITY requests from all corners, with no cover from your manager, you’ve reached peak toxicity and it’s the company culture that has been poisoned.

What are the tell-tale signs?

You’ll recognise the symptoms in your co-workers as much as in yourself.

The unending simmering tension that erupts into passive-aggressive swearing and tirades/insults/outright threats of physical violence and death (usually behind people’s backs).
The hammering the life out of defenceless, inanimate objects – usually their keyboard, mouse and/or desk.
The head in hands routine.
The getting up, storming out of the office and slamming the door routine.

So many routines. Such internal strife and suffering.

Let’s be clear, these are the coping mechanisms most people in this situation will use to deal with the stress and sense of impotent rage they are unable to fully release on those causing it.

It’s a symptom of a broken corporate culture, not necessarily the cause of it – but it doesn’t help fix the situation for anyone.

If the work keeps piling up and the hours we have to work to keep all the plates spinning keeps creeping up and up, how do we ever start to remedy things and straighten it all out long term?

You’ve got a decision to make

Firstly, you have to take a look at the big picture.

Are you really getting enough out of the situation to make it worth your while to keep working there?

Maybe the money is too good to give up. Mortgages/rent. Family. Kids. Massive student loan debts. Crippling gambling debts. We all have our reasons on the money front.

Maybe there are limited options in your area and you feel this is as good as it gets in the analytics field.

Either way, do an honest summary of the pro’s and con’s of the whole picture before making any rash decisions.

If you decide to stick it out then get something very clear in your mind right from that point:

You can’t change a whole company culture by yourself – especially if you have no authority to enforce your will on others.

You just can’t.

But that doesn’t mean that you can’t start to put processes into place to help protect yourself.

This whole situation arises from poor, weak management so you need to enforce some order on proceedings.

Some tried and trusted techniques to make the situation work a little better

Always put the onus onto the requestor to fight for their place in the priority list. If their business area has another conflicting high priority request on your workstack at the minute, ask them to work it out with the other manager which should come first. You are not an infinite resource, none of us are. People need to be reminded of that constantly.
Ask what the downside is going to be if the deadline they are claiming isn’t met. Sometimes it’ll be legit like if a regulatory body has made the request and it’s a proper “send it in or get fined” deadline. Sometimes it’s because the person asking is heading away on holiday for a fortnight and just wants their boss to see they’ve done something before they go. If they can’t give a proper reason and consequence for the deadline, it can de-prioritised.
If the two high priority competing players are from different parts of the business and of a similar position in terms of power, you should invoke the tried and trusted mob method of the “sit down”.

What is a sit down?

In his book “I’ll Make You An Offer You Can’t Refuse”, former Mafia capo Michael Franzese says the sit down is La Cosa Nostra’s version of a high-stakes negotiation at the corporate bargaining table.

He used it as a tool to think on his feet, build his case, get to know the parties involved – and how to pull their levers.

While you mightn’t have the option of whacking either party like Franzese, you will find that there is nothing to bring the chest-beating machismo of an email warrior back down to earth as quickly as a face to face meeting.

Get two of these threat-making, name-dropping big hitters in a room and let them have at it amongst themselves. 99% of the time I find you get a totally different response and attitude from all parties as you would have done by email or even over the phone.

You’ll also get a chance to weigh people up for future reference but mostly it helps let natural order decide which is REALLY the top priority. If one of them isn’t prepared to fight their corner you’ll know they were the bluffer all along.

Beyond that, it’s obviously very difficult to constantly get caught in the middle of everyone else’s turf wars.

Looking after yourself is the real top priority

If you stay in that environment make sure to prioritise your own self-care as much as you can.

Exercise regularly. The endorphins help enormously in dealing with the stress and in clearing the toxic brain sludge that builds up over any stressful period.

Run. Walk. Bike. Lift weights. Hit the heavy bag. Climb up walls. Whatever works for you. Just Do It (this is not an advertisement for a popular running shoe company, I prefer Adidas).

Disconnect from the office and the job as quickly as you can when you leave.

And always consider the long-term effects of ongoing stress on both your mental and physical health.

The money may seem good right now but if you have a stroke or heart attack before you reach 40, was it ever really worth it?

I won’t launch into a full-blown Maslow’s Hierarchy of Needs lecture but the older you get, the more it starts to hit home about what is really important in life.

Which leads me back to the other path you have available to you when you consider if you really want to keep on fighting the good fight from within the belly of the corporate beast.

Taking the other path

And that’s to quit.

Trite platitude alert but life is short, too short to do something that continually makes you a worse version of yourself over a prolonged period of time.

There are other jobs out there and this is not the only way for companies, big or small, to operate. It’s dysfunctional and eventually it will collapse because of it but not without ruining a lot of lives in the process.

Just remember that you are the one making the decision to stay in that seat and stay in that company.

Don’t take it out on your poor keyboard, own the decision and once you’ve made it then make the changes, in or out, that work for you.

Good luck.

Veruca Salt And The Just Give Me Everything Routine

2018-05-16T00:00:00Z

This is part 1 of a six part series titled “Frustrations of the Data Analyst Life…And How To Fix Them“

For those business stakeholders who can’t make a single decision in their entire lives we have the #1 method for pissing off your entire data team: Just Ask For Everything.

I call them the “Veruca Salt”.

They want the world.

They want the WHOLE world.

The conversation usually goes something like this:

Veruca Salt

Hey guys! I’m working on a big project with a lot of complex moving parts and despite having zero background in working with data, I’d really appreciate it if you sent me all of the data so I can work out what it is I need.

Data Analyst [unfortunate enough to be covering the request process that day]

Hmmm, ok. Can you narrow it down a little bit, anything in particular you want to see? Maybe trends in usage or sales or customer profiling?

Veruca Salt

Nope. I’ve given this zero thought altogether.

Now I’m freaking out because I have an action from a project meeting and really can’t go back with the great big page of nothing that’s on my desk.

So can you just give me everything right now?

When I look at it the real answer will magically appear on my screen like one of those magic eye 3D pictures where a dolphin swims out at you and playfully dances like Flipper in that old 1960s TV show.

Data Analyst

{Oh please someone help me} OK, let’s say if I was to give you the whole dump of millions of rows of data in the data warehouse.

How would you like to receive that?

Veruca Salt

Now I wouldn’t be the most technical but sure just stick it into Excel and email it over.

One of the other guys here knows how to do pivot tables and we can have a poke about through it and see what we find.

(Screen fades as our heroic Data Analyst bangs their head off the desk repeatedly until the warm embrace of unconsciousness rescues them and snuggles them to it’s homely bosom.)

Sound at all familiar?

Their thought process is obvious.

Step 1: Get all of the data.

Step 2: ….?

Step 3: Profit.

Bona fide 100% guaranteed to successfully nail the metrics you need to push that project into Green status. The big win that cements their future career progression as the brightest young thing of the project management world.

Isn’t it?

No. Of course it isn’t.

So why do they do it then?

It’s the lazy option of not having to think about what they are trying to solve and it’s frustrating as hell.

It also really highlights the importance of building a robust process for requirements gathering.

The data team has to be involved as early as possible in projects. It’s no good just being a mailbox waiting on a cold request arriving from the business and blindly being delivered without further question or input.

I get embarrassed for the person asking this question. The simple fact is that they are out of their depth but don’t necessarily want to admit it, either to the data analysts or to their managers.

Or worst of all, to themselves.

What if they are just hedging their bets?

I’ve had customers who say they would rather ask for something and not need it than need it and not have it. This is another issue altogether.

Running a small analyst team means we only want to strip requests down to the bare bones of what is really needed.

What this actually is might not always be easily apparent. But it’s another important skill to level up on for the requirements gathering and general business education stage.

Leading From The Front

As analytics becomes more vital to the business, it becomes even more necessary to take the lead and set the agenda.

If you have any plans for progressing your career as an analyst, you will have to embrace the pain of sometimes telling customers what it is they REALLY want. Not what they THINK they want.

Rob Collie and Avichal Singh put the conundrum perfectly in their book “Power Pivot and Power BI: The Excel User’s Guide to DAX”:

“Human beings do not know what they need until they have seen what they asked for”

This x 1,000,000.

Part of your role as an analyst is helping them visualise where they want to go and what they want to see when they get there.

If you only want to do the technical work then that’s fine. There are more than enough tech-only paths to take in analytics. Just remember that you lose the opportunity to craft the direction of the requests if you remove yourself from the customer facing part of the process altogether.

So how do we handle the Veruca Salts without dumping them down a garbage chute in the data factory?

I’m thinking back to similar requests over the years and shuddering at the memories.

Part of that comes from remembering my own unwillingness back then to assert my own knowledge on older, more experienced stakeholders. There was a certain reluctance to not just give them whatever they asked for.

Sometimes it takes time to learn and grow the confidence to do that. Having good air cover from your manager helps.

The frustration of getting in a request like this never goes away. These days I treat it as a puzzle that needs solved. An investigation that needs a little creativity to tease out the real underlying question. Sometimes that makes it a little easier to deal with.

Sometimes…

(And sometimes I just want to leave it to a crowd of oversized squirrels to drag the offender off to a big garbage chute and let the data version of the world of Wonka do it’s magic but I’m not allowed to say that out loud of course…)

-> Read Frustration #2: The Fallacy That Everyone Wants (And Needs) Self Service BI

How To Deal With The Urgent! Urgent! Urgent! Boy Who Cried Wolf

2018-05-16T00:00:00Z

This is part 3 of a six part series titled “Frustrations of the Data Analyst Life…And How To Fix Them”

I’ve covered this one off at length in another article including how to try and fix the noxious behaviour but it really never ceases to grind my gears.

You’ll know these people because they pull the same stunts over and over and over again.

See how many of these warning signs you recognise:

Unrealistic “deadlines” that are nothing of the sort (note to all: ASAP is NOT a date).
The name dropping of company big hitters in a game of Top Trumps with ever increasing stakes to force the request to the top of the queue.
The complete inability to see that just because they want something done right now there might be other wider priorities for the business as a whole which have to take priority.
Running to your own manager to tattle tale when you don’t do what they want.
Ignoring your very detailed explanations of how a work reception process actually works.
Promising this time will be the last time, apologising for putting you in such a position then doing the exact same thing a week or two later.

If it’s all starting to sound like an abusive relationship that’s because it is.

It’s a form of mental abuse, often going hand in hand with gaslighting (“I never said that”, ”that’s not what we agreed on the phone”,”I thought you said…” etc. etc.) and is a terrible situation to be put in regardless of being paid to do it or not.

I did cover some strategies and tactics for working the situation out in my Urgent!Urgent!Urgent! article but it doesn’t stop it being a mental drain every time it happens.

Some top tips to survive:

Recognise how harmful dealing with this behaviour is to you and realise you don’t have to keep putting up with it.
Don’t be afraid to push back. Assertiveness is a learned skill for introverts and extroverts so build that muscle through practice.
Formalise and enforce a process to alleviate the worst excesses of the behaviour.If you can force the offender down a pre-defined path with barriers on either side to restrict their rule breaking then you can start to manage the situation better.
Tell them how their behaviour is impacting everyone else.Yes, this could invoke further confrontation especially if the person has some sociopathic tendencies (and we all know people like that) but if done in a professional, non-confrontational manner it might help alleviate the built-up rage you are feeling.
If all else fails, grass them up. I know snitches wind up in ditches and all the rest of the street warnings but if this behaviour is causing you enough mental anguish over a prolonged period of time, please tell your manager. It’s their job to protect their analyst crew. If they don’t then there are more serious problems with your position and you need to find that out sharpish.

Your health comes first. Always.

However this problem is manifesting itself, always remember to put your own mental and physical health at the top of your priority list.

I don’t care where you work or how integral you feel to the overall system. No-one is irreplaceable in any business but your health is. Don’t waste it on people who don’t deserve it.

They sure don’t tell you about this side of things in the Data Analyst finishing school but they damn well should.

I can feel my blood pressure rising just writing about this subject, not good.

Grrrr!

-> Read Frustration #4: Shiny New Objects and Boardroom Buzzword Bingo

The Unending Creep Of The Unspecified Scope

2018-05-16T00:00:00Z

This is part 5 of a six part series titled “Frustrations of the Data Analyst Life…And How To Fix Them“

So who are we dealing with here?

I call them the Lieutenant Columbo’s. The folks who, when they get exactly what they asked for, always find “just one more thing” to stick onto the request.

The cheeky little addition with a wink emoji on the email.

Room for a little one? ROFL. LOL. Cheers mate.

Just a “quick easy one that won’t take you too long” as they completely revise the whole thing you and the team have spent a week working on and they had never mentioned at any stage of the requirements gathering process.

AAAAGGGGHHHHH!!!

As a species we always want more.

We’re hardwired to it and just as entrenched in the general corporate drone’s DNA is the complete inability to cover off every eventuality for what they actually need at the point when they are supposed to ask for it.

Can I learn how to put a stop to scope creep?

I’ve read countless articles from freelancing and consulting experts who set out specific ways to gather requirements, set milestones and project plan using Agile or Lean methodologies.

Anything to ensure scope creep doesn’t derail the profitability of an assignment or a project.

In my own experience, building a robust work request and requirements gathering process is one of the main tasks for an analytics manager if they want to reduce scope creep.

Discovery is everything.

Making sure it works equally well for both the business stakeholders and the analytics team is the difficult part.

Wearing many hats

Leading that process will see you become, interchangeably, at a moment’s notice:

The font of all business knowledge (even for the requestor’s own business area)
Their therapist (like Dr. Frasier Crane, you’ll be listening and spill their guts they shall)
Their priest (see the therapist but now you offer absolution and penance, lots and lots of penance)
Their prophet (seeing into the short, medium and long term future for any possible changes that can derail your whole plan).

It’s not a job for the faint of heart but by doing it properly you’ll deliver one of the main gifts you can bestow on your analyst team:

The gift of effective air support against unnecessary internal attack PLUS a well specced out, signed off set of requirements to work to.

If you half ass it at this point it will keep biting your own ass and that of your team potentially for weeks or months ahead.

Pay it forward to yourself. Future You thanks you kindly.

What Do You Do If The Inevitable Creep Happens Though?

I quoted Rob Collie and Avichal Singh in Frustration #1 and am considering getting their line tattooed across my front, Tupac-style, just so it’s always close at hand:

“Human beings do not know what they need until they have seen what they asked for”.

No exaggerating but that is the quote I’ve been waiting for my whole working life when it comes to dealing with business users in a service capacity.

The hardest part is overcoming your natural hatred of their guts for derailing your whole perfectly planned process and having to go back a few steps before you can go forward.

How do we deal with it though if it’s pretty much inevitable?

Best I can say is to prepare yourself for a Phase 2 to every request and view it as a necessary iterative step to getting your customer to the place they really need to go.

We all want a successful outcome, sometimes it just takes a little longer to work our meandering way there than we might want.

This applies doubly if you’re in a team which is under the cosh constantly in terms of fending off an inordinate amount of MI requests.

Sometimes we can be victims of our own success but it’s no reason to half-ass a job just to get it cleared off the stack early.

One step at a time

If you view the first initial customer-led request as the first draft and the analyst-led discovery stage as the second, there is no reason not to take a third pass once the analysis has been completed and see what we have learned and can further dig into.

Write it up as a new request by all means to bump your numbers if it makes you feel better but don’t shy away from it – provided you did your job correctly in the requirements gathering phase.

If you didn’t, you really have no-one to blame but yourself.

-> Read Frustrations #6: But This Is The Way We’ve Always Done It

The Fallacy That Everyone Wants (And Needs) Self Service BI

2018-05-16T00:00:00Z

This is part 2 of a six part series titled “Frustrations of the Data Analyst Life…And How To Fix Them“

The rise of Just Do It Yourself

We’ve grown used to doing things for ourselves that used to be done for us.

We pump our own petrol/gas.

We poke at buttons on an ATM machine which gives us cash from our bank account. Then we continuously flatten out and re-feed notes and cheques back into it to deposit money back into that very same account.

Take the kids to McDonald’s for a “treat” and you can swear freely at the multitude of options on the self-service touchscreen. Is this really progress from getting eye rolls for your indecision over the Big Mac meal or salad from the teenager behind the counter?

Don’t even get me started on the self-service checkouts at the supermarket.

“Unexpected item in the bagging area”. Sirens going off. Staff jumping over barriers to get to you and wrestle the punnet of strawberries from your hand.

(It’s a bag FFS, how could it be unexpected in the BAGGING AREA?)

Self-Service In The Workplace

Self-service isn’t a new notion then. So when did it grow into the movement that everyone should be doing their own data analysis and reporting?

When I started my first data miner job back around 2005, I became the de facto Master of the (Business Objects) Universe for a healthcare trust.

I built new universes in Business Objects and took over admin for the existing ones. I also built out reports that secretarial and nursing staff could run themselves.

– How many patients had been waiting over 3 months for their appointment?

– What was the age breakdown of our patient list over the past year?

– Which treatments did our rheumatology patients get?

Etc etc.

All relatively simple and straightforward. Either easily pulled by the user, using the drag and drop measures, or set up centrally (by me). They were then given to them to run any time they needed it.

Secure, consistent, measures set up and managed centrally.

Perfect.

It was a relatively small trust though and the set of systems we supported was also limited. The potential for catastrophic mis-communication of figures was reduced.

This isn’t always the case in the private sector though. Especially when an organisation gets over a certain threshold in the number of business users given free rein to run their own reports.

The potential for nefarious motivations to creep in is inevitable especially if bonuses and performance reviews are based on what the numbers say (or don’t say).

With that in mind, over the years I’ve moved well away from the notion that everyone should have self-service BI.

Who is it good for?

As an analyst it seemed like a magic wand to ward off the mundane, small-time MI requests that sucked up time and resources to complete Especially those that were little more than pointless busy work to keep the requester’s manager off their back and make them look busier than they actually were.

As an analytics manager, it was tempting to offer an alternative to working through these painful requests to dig out the useful requirements. The downside started to become apparent pretty quickly though.

Customers can’t always be trusted to know what they want. More importantly, the role of an analytics team isn’t to sit there like a dumb terminal waiting for the first passer-by to batter some basic instructions into us.

At senior management or executive levels with responsibility for reporting and analytics, it’s even more dangerous. I can’t see how anyone would trust any level of self-service BI reporting that wasn’t so stringently policed it might as well have came from the dedicated BI team themselves.

The downsides are stacking up here

The potential for incorrect figures to impact the business dramatically becomes exponentially greater, the more people you allow to plough their own furrow.

Some actual data on the proportion of business users who actually make use of a full self-service BI platform when it’s made available to them would make for very interesting reading.

If you build it…

Once the initial enthusiasm for the new tool subsides, I doubt you’d find even 1% becoming power users who know what they want and how to get it without any support.

A few more might need a little hand holding but be fairly competent in running their own general reports. Another smaller subset will jump straight into brain dead “just dump everything you have into Excel” mode.

The vast majority, however, will quickly just fall back into the mindset of “just send me the fucking report”.

And you’re back to square one.

The memory of some reports that were sent back to me in my early analyst days still haunt me. Those joyous ones when I was ordered to add another column to calculate a percentage because the user was just too lazy to do it themselves.

By this point Excel had been a staple of the modern office for over 25 years. If you couldn’t be arsed working out how to do a percentage column with the whole world of info available on Google, how could I ever trust you to accurately run and communicate your own reports off the whole MI system?

I’ll make a bold statement on it then:

There should always be a dedicated BI function setting up and overseeing the reporting.

They should be the internal police force to ensure consistency across the organisation. If self-service is permitted, it should be for properly skilled, experienced power users who have demonstrated they have the wit to use it correctly. And even then I’m still sceptical.

With great power comes great responsibility.

Uncle Ben knew. Disregard his advice at your peril.

-> Read Frustration #3: How To Deal With The Urgent!Urgent!Urgent! Boy Who Cried Wolf

Shiny New Objects and Boardroom Buzzword Bingo

2018-05-16T00:00:00Z

This is part 4 of a six part series titled “Frustrations of the Data Analyst Life…And How To Fix Them“

Picture the scene. Monday morning team catch-up with a senior executive who has just returned from a business trip to a conference he got a free junket to and had no business being at.

“So, I’ve been reading a magazine article on the plane all about the blockchain, big data and machine learning.
It’s revolutionising everything so we need to get involved.

It’s LITERALLY taking over everything.

We need to move all of our data to the Hadoop right now or we’ll get left behind, can you get onto it straight away?”

FML.

Is there anything more destructive to the productive time of an analytics team than a senior executive with a magazine or website link and the bare minimum of technical knowledge to help them apply it?

If you’ve been around long enough you’ll have seen this scenario come up over and over again, only with a different Buzzword Bingo sheet each time.

Right now, it’s Machine Learning and AI and every analyst became a Data Scientist overnight.

A couple of years ago it was “Big Data” (how big was actually “big” seemed to be left deliberately vague).

This led to a mad rush to decommission relational database systems and data warehouses in favour of data lakes, unstructured databases, ELT instead of ETL and other such innovations.

I don’t want to appear like an analytics Luddite.

I’m still (relatively) young enough to want to get on with developing and building the next generation of analytical tools and processes and my hankering for the glory days of MS Access 2003 are thankfully far behind me.

What I do object to is change for change’s sake.

There is, of course, a place in the forward-thinking organisation’s analytics arsenal for these technologies.

For the right use case, they are pushing analytical capability far beyond what we have seen up to this point.

And that’s obviously a good thing.

But it’s not for everyone, all of the time.

The analyst’s frustration then should be with having to continually chop and change their stack to suit the latest fashion or trend being pushed by the technology giants that want to sell them onto the publicity seeking C-level exec with a reputation to build.

Analysts will always want to work on cutting edge tech. It’s good for the CV, good for workplace morale and good for the soul.

What they don’t need is to have to change everything they’ve built every other year until the end of time.

How do we find the middle ground between progress and consistency?

Damn good question. Prepare for splinters from fence sitting.

Set the buzzword bingo cards aside, let our teams work on a consistent stack and concentrate on driving business value rather than chasing rainbows and the latest fashion.

If a new technology can potentially make a substantial improvement to the business then do a Proof of Concept on a small specific problem and see how that flies.

Don’t throw the baby out with the bathwater if the PoC is a success but don’t miss the chance to innovate where you can in small, incremental stages when the opportunity arrives.

And please ban those pseudo-tech business magazines on flights. Maybe then we’d all be a lot more content with what we have, not always chasing what we don’t.

-> Read Frustration #5: The Unending Creep Of The Unspecified Scope

Frustrations Of The Data Analyst Life…And How To Fix Them

2018-05-16T00:00:00Z

Let’s be honest with each other.

Working as a data analyst seems like a pretty sweet deal.

Some days though, it’s just difficult to drag your ass out of bed.

Between office politics, your boss busting your balls, early starts and late nights staring at the screen and having to deal with customers who seem to have the mental and emotional range of a tired, hungry toddler – it wears you down.

We’ve all been there.

Hell, most of us still are there most of the time.

But there are strategies and fixes we can put in place to help alleviate some of the frustrations and bring joy back to our analytical working lives.

Help is at hand.

What if you could side-step the petty office politics, get a clear, concise set of requirements and just settle into a long, peaceful, uninterrupted session of analysing data like you always wanted to?

Let’s release those inner demons of the data analyst life, put some names on our pain and find out how to fix them once and for all with our pain-busting six part series.

Frustrations Of The Data Analyst Life…And How To Fix Them

But This Is The Way We've Always Done It

2018-05-16T00:00:00Z

This is part 6 of a six part series titled Frustrations of the Data Analyst Life…And How To Fix Them

Something Just Ain’t Right Here

You’ve spent days investigating something. There was this constant niggle every time you ran a report and something about it just didn’t sit right with you.

So, in you jumped, two footed, brave of heart, steadfast of mind and foolhardy enough to ignore the nagging doubts that you were wasting your time on a fool’s errand.

You wanted to make a difference, right a wrong, reset the true course for future generations of report-running BI analysts.

With your sword of Truth and your shield of Virtue, onward you plough, digging through the legacy spaghetti code and mountains of flat file inputs towards your goal.

Bingo

And then you find it. The smoking gun. The data WMD that you knew was hidden in there and now your hands are on it.

Some muppet years ago had inner joined when they should have left joined and it’s blown the whole report into the realms of random number generator for years.

A Harlem gospel choir has struck up in your head, singing Hallelujahs to the heavens.

James Brown is dressed like a priest, dancing and jumping on a trampoline.

A shaft of light comes through a window and Illuminates you in a clear message from the data gods.

You are the data Neo, the chosen one, bringing balance to the Force and whatever other pop culture iconography you want to throw in here.

Strutting to the boss’s office

You can’t wait to take it to your boss and show what a smart, intelligent, motivated badass of the analytics world you are. You are walking on air.

Floating like a butterfly and getting ready to sting like a bee but what’s this? He doesn’t look too impressed.

Quite the opposite. His top lip is curling up. Now both lips are puckering and his eyes have narrowed.

Forehead creased, he’s steepled his fingers and is looking at your comparison report on the screen and nary a breath has passed his lips in nearly a minute.

Oh fuck.

It’s not looking good. Your palms are sweaty. Knees weak. Your arms are heavy. The music in your head has switched from hosts of angels to O Fortuna and it’s about to get loud.

The sense of foreboding sits in your stomach like a lead bowling ball and, as your boss turns to look directly at you, it starts a slow but steady movement up into your throat.

“Oh fiddly fuck, what have I done?” plays on repeat. You think it’s inside your head but it might be on the office tannoy system.

Oh shitting hell.

Time for the judgement. He pauses and says something that shocks you.

“Yeah, you’re right”.

Say what now?

Your heart starts to beat again but there is a kicker.

“We still can’t change it though. Better to be consistently wrong than tell anyone we made a mistake for the past five years and it’s going to be right from now on”.

Excuse me? Beg your pardon? Did I hear that right?

So it’s better to leave a report consistently incorrect rather than correct it as that will knock out all of the historical trends and actually make the department look worse by improving something?

Or even worse, knowingly report incorrect figures as long as the general trend is right as that’s all that “really matters”?

Truth is often stranger than fiction

The stories I have heard from far and wide would make the average analyst’s hair curl. And their butthole pucker. And their butt fall off.

If you haven’t experienced this particular variation on the old “but that’s the way we’ve always done it around here”, you just haven’t tried hard enough to fix something you should have left well alone. It’s often accompanied by the wide eyed piercing stare of death from the longtimer who always runs the report.

In the manager’s shoes I’d never accept that attitude of course but then I’m a proper diamond geezer that way. Always kicking at the future’s door. Continually pushing for improvement and squeezing that last ounce of progress out of seemingly bone-dry sources.

Or so I’d like to think anyway.

Corporate culture in this regard speaks volumes for the things you may be asked to cover up in future. If ethics are your thing (and they damn well should be), take the warning signs seriously.

The Analytics Manager’s Pledge

“I shall wear no crowns and win no glory. I shall live and die at my post. I am the sword in the darkness. I am the fire that burns against cold, the light that brings the dawn, the horn that wakes the sleepers, the shield that guards the realms of men.”

GRRM

And the best I can do is to not allow any of these god-awful frustrations to ever enter the lives of my analyst team.

This is the analytics manager’s pledge.

We don’t choose this life (anyone that does is a proper masochist) but we take on the burden anyway.

It’s what we’ve always done around here (oh godammit…)

How To Eliminate Stupid Mistakes In Your Data Reports

2018-04-19T00:00:00Z

Mistakes. I’ve made a few but then again, too few to mention.

If only the second part was true…

Big mistakes, small mistakes

I don’t mean the huge life altering mistakes of taking the wrong job or not jumping at a great opportunity when it comes along. More the day-to-day mistakes in a data analyst’s life: the incorrect reporting, the poorly cleansed data, the completely inaccurate analysis.

It’s unlikely that one incident of rushing a job out the door will completely derail your career. Over time though, the death by a thousand cuts of making small stupid mistakes can seriously dent your own confidence in your ability. And more damaging to your career prospects, your manager’s belief in your skills.

The blossoming cold sweat of realisation that you have f*cked up

We’ve all felt the heart stopping terror of realising our report to senior management is as useful as a soiled paper napkin full of randomly generated nonsense. And even worse: IT’S ALL YOUR FAULT.

If that doesn’t call for an immediate change of trousers and a blast on the defibrillator then I don’t know what does.

You mightn’t have done anything as reckless as deleting an entire production database. (Been there, viewed that). But there will be plenty of opportunities in the course of your analytics career to feel the sheer panic of the Fuck Up Moment. Even worse is the sphincter tightening realisation that you’ll have to ‘fess up before you get found out the hard way.

Over the years I’ve worked through some techniques and strategies for alleviating this potential situation as much as possible.

Check Your Own Homework

We all know what a wink and a nod process it was at school if we got to mark our own homework. Last minute “editing” of answers. Judicial use of discretion on what you meant to say rather than what was actually written on the page. Always finishing with a lot of top marks for teenage kids who didn’t really understand the whole concept of cheating themselves by fiddling answers.

Hopefully you’ve moved beyond that stage by now. The only way to cut 80% of errors out of your reporting is to be your own homework checker from hell.

Plan out your process

Set out a work flow process before you begin. Document it on paper or in a text file and plan your high level approach. Write down your data sources. Set out what you understand the problem statement to be. Figure out where you want to get to in the analysis and what you will expect to see at the end.

Keep a data journal

I’d recommend keeping notes as you work through each part of the process. If you found a new data field to use, document what it is, what the possible values are and where you found it. Memory is not reliable and you are not Rain Man. I’m certainly not, especially as I get a little older and try to cram more and more into my head. Write it down. You’ll thank yourself 1 month/6 months/3 years from now when you need to revisit the same thing again and have only the vaguest memory of what you did.

Write tests into your code

For example, check if the output after a SQL join statement has more or less rows than you expected from your base tables. Are there duplicates that need handled early on rather than a full scale battering at the end?

Write comments in your code

Again, you’ll thank yourself. I’ve lost count of the times I’ve re-visited old code and thought “what the hell have you done here?” Your colleagues will thank you even more when you jump ship and they get handed your code to run. Pay it forward, even from a selfish perspective.

Create a validation tab on your output spreadsheet

Run some counts and summaries. See how this matches up with other numbers your team produces that you know are gospel to the senior management. Top KPIs like number of customers by product, average holding, product penetration. If you don’t have a regular cheat sheet to check these off against then make it a priority to put one together.

Eyeball the raw data

Know your business. Know what “good” looks like in terms of how the data should have been entered. If something looks wrong at the data sourcing stage then get it sorted there and then. Shit In, Shit Out will remain the data analyst’s mantra until the end of time.

Ultimately though, it’s all down to you, yes you

I understand that time pressure and people banging on doors often leads to corners being cut. This can lead to catastrophic consequences. What you need is a set process that YOU follow to ensure the analyst making the mistake isn’t you.

If your manager doesn’t formalise the personal QA process then just do it yourself. Sometimes structure is good. You don’t have to be the freewheeling, creative, seat of your pants genius of the analytics world on every job. Sometimes being slower and steadier is ok too. Just not too slow…

We all need some guidance from time to time

Your manager should have more formal processes to follow to ensure that they are comfortable with the output you are shepherding out into the world. This will include formal Peer Review, Subject Matter Expert Quality Assurance and Senior Manager Oversight. I’ll cover these off in a separate post for the managers amongst us.

As an analyst you should want to be responsible for the quality of the code you write and the output you deliver. The steps above should become second nature. Even if you don’t have to use all of them every time, having them up your sleeve helps close the door and that’s what matters.

What If I Did Make A Mistake?

We all have. Lots of them. Years in the job do not act as a repellent to stupid mistakes. Processes to help avoid them do not always catch 100% of them either.

Accept this. You are not alone.

What I will suggest is to learn quickly the most important takeaways from the whole situation:

Own your mistakes.
Be upfront with your manager.
Work out quickly how to fix them.
Learn from the mistake for next time.

Confess your sins

Integrity is everything. Do not go for the short-term option of hiding the mistake and hoping it will go away.

It won’t.

What will go away is your team and manager’s confidence in your ability and character if you cover it up.

As a manager, I want to be told about mistakes quickly and completely.

Learn from your mistakes

I want to see learning so even if a mistake happens again in future, it’s not the same old mistake over and over and over again. There is no excuse for that.

If you are paddled every time the shit hits the fan though, maybe it’s just a bad environment to work in. The main learning you need to take there is that you should start looking elsewhere.

Otherwise, pick yourself up, fix the problem and learn from it.

As a manager, that’s all I could ask of you.

Technology Is Eating My Excel Reporting Job, What Should I Do To Survive?

2018-04-13T00:00:00Z

The robots are coming for us.

I’ve written before about the gnawing anxiety many of us feel about the threat of technological change engulfing us all. This is more applicable to those of us working in technology-centric jobs But there are few areas these days that aren’t under threat from the terror of being automated out of existence.

The technological change doesn’t have to be as high tech and innovative as self-driving cars or robotic assembly lines. In analytics and business intelligence it could be as straightforward as a new BI platform replacing the manual production of reports.

It makes complete sense to spend the time upfront ensuring your data warehouse and ETL processes are fit for purpose. Build out the reporting infrastructure and the power of the platform removes all of that old-fashioned human drudge work.

Good news for users. Great for the business. But not so good if you were one of the staff cobbling together the reports. Where do you go next then when you find yourself being automated out of your BI job?

When the comfort blanket isn’t so comforting any more.

Data scientists are supposed to be the new rock stars of the analytics world. But how good are your future prospects if you don’t have those few years with a Data Scientist job title to pad out your CV? What if your role wasn’t even titled as a Data Analyst?

What if you got rather cosy in your small remit and actually just copied and pasted data from reports generated from another data delivery team? Maybe your saleable skill-set really only covers Excel? Is there a place for you in this bright, shiny new analytics world?

Of course there is. But it’s high time you gave yourself a shake. You need to start looking at what you really have to offer AND where you can start to level up your data skills.

We’re going to look at two specific areas which will give any MI analyst the best opportunity of carving out a new seat at the table for themselves.

TOP Tips For Those Who Want To Survive

**#1: Learn SQL.**

Forget Python, forget R, forget Machine Learning or Hadoop or Hive or Spark or any of the other sexy shiny new technologies you’ll read about on r/DataScience. At this point, they are as useful to you as the proverbial ashtray on a motorbike. Ignore them.

What you can not afford to ignore are the basic fundamentals of SQL. Learn your SELECT, UPDATE, INSERT and DELETE statements. It’s not glamorous. It’s not particularly rock n’ roll.

Reeling off a few SQL statements is unlikely to get you any positive attention from your preferred sexual partners at any time of the day or night. Maybe that’s just me though.

If you are coming from a position as a 100% Excel Jockey, it will move you up a level in terms of expertise and career desirability.

Don’t skip this one – You can NOT afford to miss this step.

I’ve spoken to business managers in many industries. The consensus is that if a consulting company sent them over a data analyst who didn’t know SQL, they would never use that company again. It’s that important.

My skin crawls at the memory of some truly horrible SQL statements I’ve been asked to QA. But I’d rather see someone giving it a go and learning from their mistakes than ignoring this gaping hole in their analytics arsenal.

SQL – Say it (and I don’t care if you pronounce it S Q L or Sequel). Know it. Love it. It’s your new best friend.

**#2: Business Knowledge and the Communication Thereof.**

Are you one of that pot of people who haved worked in a reporting job for years and still have absolutely no notion of what the numbers actually mean in context? I hope not. After a couple of years experience, you should have built up a wealth of business domain knowledge. If not then maybe this whole data world isn’t really for you. You might be safer giving Uber a shout for a couple of years before the self-driving cars come in.

If you have then you need to start letting other people know just how much you know. Even more so, let them know how expensive the loss of that knowledge would be if you were shown the door. A varied, long running career in many different areas of one business doesn’t necessarily send up too many red flags for me. We know organisations change over time. Moving from area to area can help give you a more rounded view of how the whole business inter-operates.

Getting pigeon-holed is good to go deep on a subject but I think it’s detrimental over too long a period. If you’ve been bounced from role to role because of HR issues however that’s another matter entirely. All of the domain knowledge in the world shouldn’t save you. Another set of stories for another day perhaps.

Bringing it home

Always remember that you don’t have to be the fastest mover in the office. Just a little faster than the slowest member of the herd. If you keep levelling up your skills incrementally at each crunch point, it won’t be you getting picked off by the cost saving lions.

Go with the fundamentals. Get your basic SQL sorted. Let people know just how much you know about the business. Knowledge is power after all and we all need a little power behind us when the wolves of progress start closing in.

URGENT! URGENT! URGENT! How To Deal When Someone Else’s Poor Planning Becomes Your Emergency

2018-04-06T00:00:00Z

The Moment Your Blood Pressure Starts To Rocket Arrives

We’ve all experienced the immediate stress of an email subject title like “URGENT! URGENT! URGENT!” hitting our inbox at the very worst time. Usually when we are up to our eyes in work and the tension headache kicked in several hours before.

Tightness in the chest. Blood pressure going through the roof. Blood pounding through that vein in your temple. Don’t tell me you don’t recognise it – we’ve all had it. And that’s before you’ve even clicked on it and read the actual message.

You probably know the culprit before you properly look or, at the very least, could put together a rogue’s gallery of possible suspects for a line-up. Some people seem to live their working lives sailing as close as they possibly can to the wind of a fast approaching deadline. Mostly with no regard for the stress levels of the other people they will inadvertently pull along in their wake.

I Like To Help

I like to think I’m generally a rather helpful sort of person. In my work-life, I take pride in being able to help recover seemingly unsalvageable situations. When colleagues have been let down by other parties, I can help pull them out of a data hole.

This does have the rather pleasant side effect of building genuine professional respect over time. Work at any organisation over a prolonged stretch of years and that is obviously a very good currency to store up.

It also brings the potential for certain parties to try and take advantage of that helpful nature from time to time. The result after several episodes gives us the symptoms I mentioned above. The tight chest. The BP shooting through the roof. All of the physical symptoms that a man approaching his fortieth birthday really doesn’t need to be experiencing on a regular basis.

I’ve got two young daughters. You think I need any additional stress in my life? Oh hell no.

Stop being taken advantage of

What do we do then when our good nature is repeatedly taken advantage of and someone else’s poor planning keeps on becoming our emergency time after time after time?

First up, you need to confront the reaction within yourself. Recognise that this is not a one off. Bottling up the rage or taking it out on your poor unsuspecting mouse or keyboard is not a sustainable way for you to approach these situations.

We live in incredibly trying times on many levels. Our mental health is under increasing attack from all sides and we have to know when to draw the line on potentially harmful incidents that are continually triggering us.

The first step is admitting to ourselves the very negative effect certain situations are having on us.

Confronting the situation is difficult – but necessary

If certain people are putting you under undue stress or pressure then you have to face that confrontation. This is not always easy for the stereotypically introverted tech worker.

A good start would be sending an appropriately worded email to the offender. Tell them that the pressure you are being put under is not conducive to good work product. (Please don’t fire off the first angry response that comes into your head. We’ve all been there and done that. Less said, soonest mended etc. etc.!)

Sometimes the people responsible aren’t even aware they are doing it. Sometimes they are which makes them a bit of a shit at one end of the spectrum and a bloody chancer at the other.

Finding out which it is would be helpful. Maybe it will nip it in the bud without further escalation (or heart attack inducing email requests).

Often the person asking is under pressure from their own management. As the shit keeps flowing downhill, you end up being the last resort recipient of said effluent. Welcome to working as a data analyst.

You hold the keys to the kingdom which is a great gift. It’s also a pain in the ass when someone else wants to get in and you are locking up for the evening.

Sometimes we have to be the one who says no – and that’s not easy

Imagine yourself as the bouncer on the club door. You are checking names on the list for entry and, when you say no, sometimes it gets a bit physical. Sometimes it gets really out of hand and you need to call for backup and this is where your manager comes in.

Providing air cover for you as an analyst and making sure you have enough protection to allow you to do your actual job is a large portion of THEIR job. Don’t be afraid to pass the buck to them.

They need to take over the discussion and start to look for the reasons behind the constant last minute requests.

Sometimes they’ll have to kick it up a level themselves to get to the real root cause of the breakdown. (You do HAVE a formal work request process don’t you?) Sometimes it becomes more political than it needs to be.

Whichever way it goes, the only acceptable end game is for the formal process to be recognised and followed. Your role is to concentrate on wrangling that data into an acceptable form to help drive the success of the business.

Building skills to keep on moving up in your career

Learning stakeholder management is a very important part of moving up through your career. You’ll find it everywhere from junior analyst to senior and then on to management.

It’s going to mean confrontation and conflict at times but that’s unavoidable at any level. Set your personal boundaries and when they get crossed then don’t be afraid to stand your ground.

Knowing your manager has your back is vital for the personal growth and overall success of the analysts on a data team.

I’ve been very lucky in my career to have had managers who would go into battle for me. And I hope I’ve carried that on as an analytics manager myself.

Bottom line is that you don’t have to accept being put under unacceptable stress to cover for someone else’s shoddy planning.

Speak up, try to get to the root of the problem and don’t be afraid to call for backup if the stakes get too high.

That’s part of the game and it has to be learned.

What if none of this works?

If after this you are still getting the same kind of requests, you really need to take a look at whether you want to be part of a company that continually treats it’s people in that way.

Your mental and physical health are intrinsically intertwined. No job is worth putting them both in danger over a long period of time. No exceptions, no matter what.

If you have experienced any of these situations in your day to day work or want advice on dealing with work-related mental health issues and stress, please drop me a note. I’m listening.

Which Degree Course Should I Choose For A Career As A Data Analyst?

2018-03-29T00:00:00Z

When The “Best Days Of Your Life” Are Drawing To A Close.

I’ve covered off my own painful experience of choosing a degree course when I was at school in another article. The memory of that pain and indecision is still raw, even after twenty plus years.

It’s easy to fall into the trap of thinking that the degree choice decision you make at 17/18 years of age will ultimately dictate the exact path of the next fifty years of your career. But that’s rarely the case.

Unless you want to follow a strict career path like medicine or law, I don’t think it has any real detrimental impact over the long term. Much more emphasis should be placed on the opportunities you take post-graduation.

This is personal experience talking.

In hindsight, my own subject choice didn’t necessarily hurt my CV when I applied to work in Financial Services. (Note: I studied Accounting.) If it had been Maths, Economics or Physics it would have been equally as useful to me in practice i.e. not particularly.

The fact that I’d ran a mile from the world of accounting after graduating didn’t work against me too much. I meandered my way to banking via the Civil Service, software development, Health Service Trust and a couple of bouts of self-employment.

In fact, the variety of industries I’d encountered between graduating and starting at the bank actually worked in my favour. I’d been lucky to see the inside of a few different working worlds. And the degree itself was nothing but a piece of paper to me by then.

What Do I Look For When Recruiting Analysts?

When I’m recruiting data analysts, I don’t rank a candidate any higher because they studied Data Analytics or Maths instead of Music or English. I’m much more interested in what you have actually DONE and how you communicate that than which degree course you studied.

For junior roles, it’s harder to prove you have the necessary chops when you don’t have the history to base your interview answers on. That applies just as much to those who have a more “relevant” degree or Masters in Data Science or Analytics.

Even the general requirement of having to possess a Bachelor’s degree really frustrates me about traditional recruiting. I know many candidates with good business experience who would fall at the first hurdle in the majority of recruitment openings. They’re self-starters with years of client management skills but no university degree.

Which one should we put more value on? It’s a no-brainer.

Taking their time to study for a degree would take their focus away from becoming better analysts on the job. And all that just to fulfil some box-ticking exercise to eliminate a misguided barrier to entry. What a monumental waste of potential talent.

We set the degree requirement to filter out more noise in terms of unsuitable CVs received. But it also stops good candidates from non-traditional education and career backgrounds getting through. It’s a terribly unfair side effect.

We need to encourage more people into analytics. Not pull up the drawbridge and fill the moat to protect ourselves and keep everyone else out.

The bottom line is to not get caught up in the analysis paralysis of degree selection. Personal experience tells me that choosing a course because you think you might want to work in that area often leads you down the wrong path.

It may be years before the twists and turns lead you to a place you actually want to get to.

Find a course that interests you at a college or university that suits your situation. If you can boost your analytics skills through side projects or part-time work then make the most of it.

Even if you haven’t and decide after graduation that you want to get into analytics, make the most of what you did do.

Find you can’t demonstrate critical analysis skills you should have developed in projects working towards your History or English degree? Red flag time. I’m going to wonder what the hell you did for those three or four years.

And that’s where the really interesting interview questions will start…

How Much Experience Do I Really Need For An Entry Level Data Analyst Role?

2018-03-23T00:00:00Z

A Question From The Floor.

I’ve been answering a few questions over on Reddit recently for young graduates. Most are finding it tough to get their foot onto that first rung of the data analyst career ladder and need some guidance.

One that I thought might do with a little more detail was this:

For applications which say “you should be proficient in these” should I not apply to them if I don’t think I’m proficient? Like on applications do they really mean it when you say you need X level of experience with a certain language, or X years experience in the industry?

ANSWER:

It depends on the level you are applying for. For an entry level post I think it’s bad form to ask for X years experience. The whole point of recruiting at that level is to bring people in and train/mould them from the ground up. It’s not about spending a year breaking the bad habits they’ve picked up elsewhere.

Worst is when when lazy recruiters advertise posts looking for X+ years experience in a technology that has only been picked up in the wild for a couple of years. Especially when they ask for this at a junior or entry level. Come on, give people a chance!

If it asks for proficiency then let the interview panel be the judge of how proficient you are. Don’t BS people and say you are an expert when you’ll quickly be shown up on the first day on the job. But don’t self-defeat yourself and let a bit of imposter syndrome work against you either.

Provided you have actually gotten some experience, I think a willingness to learn and be open to new challenges is more important. This could range from just having opened the app to have a poke around to doing a college or side project on your own time.

Will this always apply?

Senior roles are different as you are expected to hit the ground running. I would be looking for those X years experience in a similar role or one with overlapping skills. The proficiency level would need to go up to actual regular use as well. If not in the exact tool or language then in something where the understanding is similar. For example, if not Tableau then Qlikview. If not R then SAS or Python.

When I interviewed at $BANKJOB they told me that SAS knowledge was extremely desirable. They knew however that there was a very limited talent pool in Ireland that actually had that kind of SAS experience. If I’d been put off by that requirement I would never have even applied. But they took a shot that my SQL, database dev and programming background would get me through and they were right.

It was over two years before I wrote a single line of SAS code. Funnily enough, no-one we hired in between was a SAS expert either. We got several damn fine analysts that added a HELL of a lot of value though which shows that hiring shouldn’t be a strict, rigid affair.

Show, Don’t Tell. But Tell As Well. Just To Be Sure.

Careers take many roads, twists and turns. It’s the job of the recruiting manager to realise that and pick out what you can bring to the table. They’ll only get that from what you tell them on your CV and in interviews.

For those just starting out, it’s about taking a punt on someone who gives you an inkling of what they could do rather than what they have done. That’s always much harder to spot. That’s why side projects are great options for showing what you can do. I’d use them to help open up a conversation as much as for the actual end result.

Ultimately, if you don’t make an effort to sell yourself then no-one else is going to do it for you. And that applies however many years you have on your personal career clock.

WTF Is Data Wrangling?

2018-03-20T00:00:00Z

“Data wrangling, sometimes referred to as data munging, is the process of transforming and mapping data from one “raw” data form into another format with the intent of making it more appropriate and valuable for a variety of downstream purposes such as analytics. A data wrangler is a person who performs these transformation operations.” Source: Wikipedia

Thanks Wikipedia. Nothing like a dry, officious sounding definition to kick off the post. Especially describing something that will take up around 80% of any data analyst’s working career.

So what is data wrangling?

Data wrangling, in simple terms then, is the part of the analytics process where we get our hands on the raw data. We learn a bit about it. Clean it up. Join it together then find a more useful way of looking at it than we could have done before we started.

The average data analyst will spend around 80% of their time on this stage of the analytics process. If they don’t then they are likely to be wasting the other time they spend actually analysing the data. Your laziness will have rendered it virtually meaningless due to poor data quality and/or inaccurate or incomplete elements.

Be warned.

This doesn’t sound like the sexy, high octane world of data science that the media led us to believe we were embarking upon. Have we been sold a lie?

Well, yes and no.

If you don’t enjoy the prospect of taking data from:

the neat and tidy data warehouse
text file data dumps from legacy systems that don’t speak to the DWH
Excel spreadsheets from the finance team
public datasets from government bodies
raw data files gleaned from social media accounts
and potentially thousands of other data sources

…then sifting manually through them to see where they need cleaned and joined – maybe this isn’t for you after all.

I’ve written before about the importance of an inquisitive nature to really making a good stab at this profession. No sugar coating – it’s in the data wrangling arena that you’ll likely spend most time working on this. Picture yourself manually scanning through thousands of rows looking at data structure and content creating training datasets. That’s long before you get the thrill of writing scripts to automate the same job over the other 10 million rows. But it’s where your business knowledge comes in to play and that old detective-esque need to ask questions of the data. And ultimately find the answers within it.

It’s not the 1970s, can we not skip this part or automate it?

I don’t believe there is any shortcut to learning the dark arts of data wrangling. No route other than getting your hands dirty and delving ever deeper into each of your data sources. Digging and digging until you know their intricacies and foibles like the back of your hand. Master this skill and you will learn to savour the joy of a new untamed data source and the endless opportunities it can bring your analysis.

Likely to soil your undergarments at the first sign of a unstructured, unclean dataset that you are unprepared and unwilling to wrangle? You’ll always founder on similar rocks in future and your overall desirability as a data scientist or analyst will suffer.

My favourite question when interviewing analysts is when we ask the candidate for an example of a time they faced a challenging situation and how they worked through it.

When a real data analyst is given that question it inevitably comes down to working with difficult datasets. I like to see how they worked their way through the potential minefield to build a workable solution out of the parts they’d been handed.

Show me how you approached the data wrangling, what your thought process and investigation methods were and how you made good out of it. I guarantee you’ll have went a long way to winning me over when it comes to grading your overall performance.

So what’s in it for me?

I know how much of an impact getting your data wrangling right can have on the overall success of a project. If I can judge the analyst involved feels the same way then I’ll be a hell of a lot happier trusting them to get it right for me.

Wrangle like your life depends on it.

Skimp on this stage and I’ll conclude that you don’t give a shit if your results are as close to accurate as they possibly can be. If you only want to get some numbers back on a spreadsheet then you might as well write a random number generator and save wasting all of our time. Give the data wrangle the respect it deserves and we can all reap the benefits.

What Is The Best Way To Learn A New Data Analytics Skill?

2018-03-16T00:00:00Z

The world of analytics moves on quickly. Everyone is talking about shiny new toolsets and languages they are working on and in and you’re afraid of being left behind.

What’s the best way to get those skills up to speed then. Should you do a side project? Should you do a training course?

TLDR: Simple answer = Both. And neither. And a few other things. Thanks Al, helpful as always, is it even worth my while reading on? Why yes dear reader, follow me down the rabbit hole of my own data analytics learning experiences where I have done both and sometimes wound up using neither and I’ll explain why.

So let us begin

In my experience there are five main ways of trying to pick up a new technical skill, language or technology.

Formal Training Course
Online Training Course / MOOC
Side Project
Read a book
Just Fucking Do It

When I started as a contractor at $BANKJOB I had no idea what SAS was. The parent company currently has one of the largest SAS deployments in the world and even back in the mid-2000s their analysts were big on SAS, mostly on the mainframe.

The team I joined was only getting put together so everything was green field for us and the main skill-set needed was SQL and VBA through MS Access and Excel. Low hanging fruit abounded but after that had been picked off we started looking longingly across the water at what we were missing out on.

Formal training

Access was quick and handy for small analysis jobs and reports but it was painfully slow for large data pulls and queries so we lobbied and five of the wider local team got a 5 day formal Intro to SAS course ran in a local hotel with a Brazilian JuJitsu master trainer whose girlfriend was an exotic pole dancer. Damn fine teacher, knew his stuff inside out and with a week’s grilling, stress and intense learning all five of us came out the other side as certified SAS Base Programmers. Hurrah!

I remember vividly when all of this happened as on the first morning of the SAS course I got a phone call from my then girlfriend (now wife) telling me that the pregnancy test she’d taken was positive. Try concentrating on learning about SAS when you’ve found out you are going to be a daddy for the first time. I should have got extra credit never mind another five days graft plus an exam to battle through!

The formal training was good, it gave us all a basis to work from but it took me having to force myself to use what I’d learnt (plus a ton of Googling) to put it into practice when we got back into the office and had tight deadlines to meet on requests that were coming in.

Several of the others reached back for the comfort blanket of MS Access rather than take the rough of the transition to SAS and it took much longer for it to sink in with them when they did eventually get somewhat forced into using it.

Taking longer to do simple things you know how to do in the old tech or language, sideways glances from your boss when deadlines are tight and frantically Googling for any help in what you are trying to do are great ways to bring humility back into your working life. Hard to feel like a Rockstar when you are trawling newbie questions on Stack Overflow and still not quite getting it.

The real learning from that experience was that formal training is great for hothousing and getting you a flavour for something new but without Just Fucking Doing It you may as well not bother. I’ve done plenty of training courses over the years that had no practical application in my day to day work and without that hands-on usage it was all a waste of time and money.

Even worse is the embarrassment at getting questions about those skills or technologies at an interview when you really know damn all about them. Use it or lose it is the bottom line with getting any value out of formal training.

Online Courses / MOOCs

MOOCs are another matter altogether. We can now study virtually anything for either free or a really low price from the comfort of our own homes, stick that on our CV and call ourselves an expert. Except it hasn’t really worked out that way in practice.

Anecdotally, most people agree that MOOCs have been a failure with only approx. 2% of courses actually being completed. That’s a terrible statistic regardless of how you look at it.

The freedom to study anywhere, anytime and on your own terms is clearly the reason why they haven’t worked out. No accountability, no real investment and no real skin in the game all add up to a less than valuable experience and courses being ditched left, right and centre before completion.

I’m as guilty of that as anyone else. I’ve got courses in Python, R, Data Science and Qlikview lined up in Udemy that I haven’t finished (half of them I haven’t even started) but for $10 or $20 a pop, it’s no real loss to me.

The one cheap/free MOOC I actually finished wasn’t in a technical discipline and my natural interest in the subject matter helped pull me through it (Yale’s “The Global Financial Crisis” on Coursera).

Working in banking at the time of the 2008 crash and watching all of the documentaries and movies that covered it helped keep me interested as well and while it mightn’t help me get any further in my career I certainly don’t see it as wasted time.

The $10 Python course I started on Udemy however, despite being well taught and covering all of the basics you would expect when starting on a new programming language, wound up boring me in no time. I persevered but working through examples and modules wasn’t floating my boat and without having a use for it at $BANKJOB, I dropped it when other things started looking more appealing. Like Game of Thrones marathons. Or clearing out the outside drains.

I will say that there is a massive difference in self-motivation for completing a course between the low cost (or free) options and more premium online training courses. When you have invested $2k in a course because you believe in the people running it and how they can help you improve yourself and your business/skills there is much more likelihood that you will get your ass in the chair and see it through to the end.

Skin in the game is a big factor and really should be the big learning any online education platforms or providers look at to really make a difference in this space.

Books

Books. Paper. Virtual. Whatever. I don’t think I’ve ever properly learnt any new programming language only by reading a book and following the tutorials. Back around 2005/2006 I was self-employed and a proper fanboy of 37 Signals, now Basecamp. (Note to past self - be careful who you look up to in the tech world - or any world for that matter…)

I was a freemium customer of Basecamp, I’d bought the original PDF version of Getting Real, printed it out, put it in a lever arch file and carried that on my commute for a good six months, just reading and digesting their philosophy for small tech businesses.

With DHH being the founder of Rails I naturally thought I should level up my web dev skills from classic ASP and some PHP I’d been toying with and jump into Ruby On Rails. I bought two books, started one, don’t think I cracked the spine on the other and within six months they were both largely out of date as the language and framework moved on.

I’d done pretty much the same with VB.Net when it came out. I’d been self-employed then as well and figured I might as well move with the times rather than get left behind as the Dot Net tidal wave threatened to engulf my fledgling VB6 and classic ASP skillset (for more reading on dealing with the pace of progress in data and tech, read this article). Same outcome. I followed some tutorials, I read a lot, I never deployed a live Dot Net application. Not one live line of code.

Fast forward a decade and this time I have a tablet and Amazon Kindle app and I bought a book on learning Python to read on my morning bus journey. It was well written and, much like my Udemy Python MOOC, covered the basics well.

Reading it with the gentle undulations of the bus and the constant white noise of my fellow passengers it also had the very speedy effect of putting me over to sleep. Not much learning going on but the nap was very well received and I was fully refreshed by the time I got to work. Not so good for learning Python however. Attempt #2 = failure.

Side Project

My next option then morphed into a side project when I found I had an itch I wanted to scratch. There are no shortage of publicly available datasets to play around with these days but what if you had a favourite coffee table book which had lots of stats about a certain group in it that you felt might be interesting to mine for some insights?

So I have a long time interest in collecting books, movies and memorabilia on Italian-American organized crime during the twentieth century. It’s not quite Father Seamus Fitzpatrick’s collection of “Second World War” memorabilia in Father Ted but a late night encounter with The Godfather Part 2 when I was 14 or 15 has left a lasting impression.

One of the books I have is a hardcover coffee table book called Mafia: The Government’s Secret File On Organized Crime which is a collection of stats and info gathered by the Federal Bureau of Narcotics in the United States against these gangsters in the first half of the twentieth century.

It’s a fascinating book to flick through but difficult to really analyse due to being pages and pages full of paper (as actual books tend to be). I decided to get the e-book version and use Python to data mine it, build a database of the info it held and do some analysis and visualisations to see what it really tells us overall.

I had an interest in the subject material. I had an end point in sight. I knew the language I wanted to learn. It was all systems go. It was a proper side project. To the code cave!

The actual Python parts went well. Without the constraints of sticking to a book or MOOC’s tutorials I had a steep learning curve as you don’t start off gently when you are setting the pace yourself. I converted the e-book to HTML, did a lot of data cleansing and initial data wrangling and set to work looking for the patterns and key data fields I could pull out.

It turned into a really enjoyable learning experience and I wound up with a SQLITE database with a few tables of connected, cleansed, workable data that I could use for analysis. I even did a plot of gangster locations onto a Google Map to kick off the visualisations.

Someday I may even write it all up as a case study and finish the job properly. (EDIT: I finally bit the bullet and started writing out how I approached the full project. For a walkthrough with code and data check out Analysing The Mafia: Part One.)

At the time I got side-tracked in building out more features on my wife’s e-commerce store in Woocommerce and the Mafia data side project got mothballed.

I learnt a lot from that project though but the only problem afterwards was that I wasn’t using Python on anything else, either at $BANKJOB or at home on other projects and as the weeks and months went on I lost my newly found powers again.

JFDI

I still have the code of course so it’s never completely lost and while it may not have been the most elegant code base, it was the most successful attempt I’ve made at learning Python. The old adage of Use It Or Lose It still stands true and that has been my biggest learning over the years about any kind of self-taught technical upskilling.

Unless you JFDI and somehow manage to integrate the new skill into your day to day job, projects or contract work, you will never retain the knowledge. Anyone with a modicum of programming experience can pick up any of the main analytics-friendly languages out there these days. Whether it’s R, Python, SAS or anything else, you can learn quickly how to operate at the Keep It Simple level and get yourself a foothold in the language.

Google is your friend, so is Stack Overflow but you have to be using the skills in the real world to get them to stick. It doesn’t matter how much you spend on a formal training course or how many books or online courses you batter through and stick on your CV, there is no substitute for Just Fucking Doing It and not stopping when the newbie pain gets too much to bear.

What now?

Have a look at your CV now and see how many of the buzzwords you’ve listed out there could you actually turn your hand to if called upon TODAY. How do you learn and make sure the knowledge sticks and isn’t wasted? Is it better to specialise your CV for the tech you definitely have a proficiency in rather than going wide and listing everything in the hopes of catching a recruiting manager’s eye and winding up looking like a chump when they question you on it?

Send me your experiences and observations, I’d love to see how it is for everyone else out there.

Should I Feel Nervous That The Data World Is Changing At A Faster And Faster Pace And I Can’t Keep Up?

2018-03-12T00:00:00Z

I see variations on this question a lot and it inevitably takes me back over two decades to when I was getting into web and software development (the precursor to my data analyst career).

Adventures In Accounting

I studied Accounting at university but lost any interest for that midway through second year. By the time I graduated I had barely managed to keep my scores high enough to scrape into a 2:1 classification.

At age 17 I’d thought Accounting was a sound choice. I was good at Maths and Economics, and there was a professional qualification path afterwards that could lead to other areas in business and finance.

I wasn’t wrong there but with little life experience of what the job or degree path offered beyond the UCAS handbook it turned out to be a poor choice at that time.

Hindsight is 20/20

It wasn’t what I was looking for and with a little hindsight, and a lot of years in between, my advice to myself then would have been to do the Finance degree I’d really had my eye on and make a shit ton of money in the City before the 2008 Financial Crash hit. (Disclaimer: I’m only sort of joking about this).

I enjoyed the Statistics and Economics modules we did, a throwback to my A Level days in Mr. Scott’s Maths and Mr. Christie’s Economics classes.

But beyond a couple of other random classes in Financial Market Theory and an Economic and Social History comparison of Britan and Japan in the 20th Century I was bored and, worse still, struggling massively with anxiety.

Panic attacks and avoiding tutorial classes for fear of being put on the spot and not knowing an answer were increasingly common occurrences to the point where I barely went to class at all in third year. I was getting increasingly close to the big bad world and real life and my introverted nature and anxiety was not making it easy for me to take full advantage of what university should have been about.

Moving On

The only thing I left QUB with, other my BSc Accounting degree and a burgeoning state of depression, was a deep held knowledge that I never EVER wanted to be an accountant (sorry accountants, you do a fine job and have served me very well over the years but it’s just not for me).

(READER: I’m sure that’s been very therapeutic for you Alan but what does this have to do with the pace of technological change and my nervousness about an unknowable future?)

Stick with me, I’m getting there, I promise.

I graduated in June 2000. The Dot Com Bubble had already started to burst a few months earlier and pretty soon it would lead to complete carnage in that sector. At that point, all I’d used the internet for was getting guitar tabs off the Online Guitar Archive and chatting to friends on MSN Messenger.

My band at the time had a webpage but our drummer was the only one who knew the dark arts of HTML and could maintain it. I’d dabbled in BASIC when I was a kid, typing programs out of the manual of my Commodore Plus4, but had never considered IT as a career in any way.

The world was a different place then kids, round here it was all green fields as far as the eye could see. The web was simpler, more homespun, more personal.

Before any of the modern tech giants took off and hit that hockey stick growth period. Before Facebook. Before Google took off. Before Twitter. Before Amazon sold more than books.

What Came Next

Fast forward a year and a bit. I’d killed time working in the Civil Service, filing paperwork and, foreshadowing the future, even did a few months on secondment in a Financial Analyis Unit, loading stats and figures into Excel spreadsheets. They even offered to put me through my CIMA Management Accounting exams but that freaked me out and I started looking for an escape rope to climb out on.

It came in the form of a Government funded scheme called the Rapid Advancement Programme (RAP) which was kind of a bootcamp that fast tracked graduates from non-IT backgrounds into IT.

My first day at the course was particularly memorable as the internet went down at one point and someone eventually came in to tell us that a plane had flown into the World Trade Center in New York. Much like how everyone of a certain age knows where they were on the day JFK was shot, my first day at RAP will be equally easy to remember, if only for the very worst of reasons.

9/11 aside, we did some intro classes to see where our interests lay. Some took the hardware path, some the software development and I joined the Emerging Technologies path. It included short courses in ASP, VBScript, Javascript and databases. Just enough to wet the lips really but it was a start.

I went out on placement for 3 months at a very prestigious financial software development house and was shown up for the know-nothing wannabe I really was. I bombed big time, totally out of my depth and very much aware that they thought I was a clown.

I got my head down when we went back to RAP after placement and got two Microsoft exams in Visual Basic 6 (pre Dot Net days of course) and used those, along with a welcome family introduction, to get an interview and eventual junior developer job at a local web and software development company.

Don’t Bore Us, Get To The Chorus

I could fill this whole site with tales of my (often sordid) working life but (finally) getting this back on track, my point is this:

The brief excerpt of my life above covers about a 5 year period between 1997 and 2002. The pace of change even then was frightening in how quickly things moved on.

Between then and early 2007 when I started working as a data analyst at The Bank, things moved on even further. They didn’t slow down, they accelerated and continue to do so. There is no getting away from that.

If I had panicked about an unknowable future at any of those stages and jumped onto every new platform, technology or programming language that put itself in front of me I would be no further on than I am now.

Is there ever a “safe” option?

Maybe the most future proof option would have been the Accountancy professional qualifications but then accountants are under threat from automation these days as much as factory workers were in the recent past.

If I’d worried when I was a software developer working in VB6 and ASP and then Dot Net was launched, I would have freaked out and dropped everything to jump on that band wagon. (Note to all, I did start learning it a few times but my own world changed again and I’ve never had to use it professionally).

When I started in data and analytics we were using MS Access to create standalone small database apps to link into data warehouses and run reports and analysis into Excel. If you wonder how I’m qualified to answer the question “Can I use Microsoft Access For Data Analysis?” then remember I’ve built more data solutions in Access than Pele scored career goals.

What Should I Do To Stop This Rat Gnawing At My Stomach?

The jobs I’ve worked at didn’t really exist back when I was flipping desperately through that UCAS Degree Handbook back in 1996 in the Careers library at school. How could I possibly have future proofed myself for twenty years down the line, never mind forty or fifty?

Simple answer: I couldn’t.

You can’t. Not fully, not completely. There is no insurance you can buy that will stop the world moving on beyond you. IF. YOU. LET. IT.

What you can do is stop focusing on the tools and technologies and thinking that the bleeding edge you read about is what is suddenly being used everywhere and without it you will sink like a stone to the bottom of the nearest (virtual) mill pond.

Take a deep breath and focus.

Your value as a data analyst is in learning how to apply your tech skills to extract value for the business you are working in, based on the knowledge and experience you have built up over your life.

Keeping your tech skills sharp is important but don’t get distracted by every bright new language or platform that comes along. Industry, especially at large corporate level, is INCREDIBLY slow to change and move away from their existing stacks or established analytical environments.

Build up your skills in your current role especially the soft skills. If you want to get used to a new language then do a side project. Something with an end goal and result you can write about or put online to show people.

There will always be people who are better than you at a certain skill but that doesn’t mean that your own skills Venn diagram intersection won’t still be incredibly valuable for a long time to come.

It may not feel like it when reading the tech news or mainstream media tolling the death knell for anyone but our new AI-automated robot machine overlords.

But remember scaremongering sells for them. It doesn’t mean they know what they are talking about.

It’s All Going To Be Alright

We are still in on the ground floor of this thing.

Change is coming quickly but if we are adaptable and concentrate on applying our knowledge to business problems using data and analysis, there will be a place for us for a long, long time at every level of business in every industry.

That thought lets me sleep peacefully at night. When it comes to your place in the future workplace, it should do so for you too.

Homework

Take some time now to write a list of the skills you have put together over your working life or through projects you’ve completed in your studies. That doesn’t only mean technical proficiency either, all of the soft skills as well e.g. working with other people, extracting business needs from stakeholders, planning etc.

Be honest with yourself and if you see holes in that, think about how you can fill those holes. Drop me a mail if you want to chat it out, I’ll be happy to help anyone who really wants to help themselves.

And remember to breathe deeply. Always. It’s all going to be fine.

Is My Job Considered Business Intelligence?

2018-03-09T00:00:00Z

Having worked in the same company for over a decade now I’d forgotten a little of what it was like to experience the general confusion many people can get over their job description and what it entails compared to other similar roles.

Is this what I think it is?

I read a thread on Reddit /r/businessintelligence where the OP was working their first job out of college and was being pushed by their manager to get a Master’s degree in Business Intelligence. Their main problem was that they didn’t know if the job they were currently doing was even BI itself.

A quick list of their basic job tasks went like this:

Run reports.
Collect stats from business users, research what caused the issues and present the findings to management.
Build pivot table reports in Excel with data from Cognos and Access.
Act as the education department’s “data analyst” and provide recommendations to help improve call centre performance.

Based on that it’s a fairly straightforward answer for me – yes, you work in Business Intelligence. Going back to my own post on the four stages of an analytics process, this person clearly covers off the reporting and analysis portions which I would roll up under the Business Intelligence umbrella.

If Predictive Analytics looks forward to see what will happen, Business Intelligence is the part that looks backwards to see what did happen and why.

Great so now what?

The other side of that answer is this though: now you know, what are you going to do with this information?

I’ve worked as a data miner, data analyst, senior data analyst and senior database manager and, while the title changed, the main data-driven job tasks and skills needed didn’t.

I extracted, transformed and loaded data. I built and ran reports. I analysed the results and presented to management and executives. I modelled the customer base and added more value to the existing data to help generate leads or improve customer service.

The skills overlapped and enabled me to grow further in each competency and that’s what really matters.

Buzzword Bingo On The CV

Unless you head down a very specific siloed path in your data analytics career I don’t think it really matters what your job title is. Having your own definition of what each aspect of the job entails (e.g. do I do business intelligence work?) is helpful when putting your CV together to show what you have worked on but, with so many skills overlapping within the general area, I’m more interested in seeing what an applicant has actually done, not what they were called while doing it.

Am I an analyst? A BI developer? A data scientist? A data engineer? An ETL developer?

Knowing what you are is difficult in any industry or profession where the definitions on what is what keep changing or aren’t ever agreed upon fully.

Even more so if you wind up wearing a lot of hats like the OP in the Reddit thread above (and me in my own career). That suited me because I like a breadth of experiences rather than being pigeon holed into one thin salami sliced remit. You might differ and prefer that narrower focus that lets you go deeper on a particular skill or area.

My advice is just don’t tie yourself in knots wondering if your job title properly matches your job description and role. Even worse if your worrying is down to where this will force your career to go in future.

No matter where you work, I am sure they have much more interesting questions for you to be using your valuable brain power on right now. If they don’t, that’s when you really need to worry.

Is My Business Analyst Job Title Holding Back My Business Intelligence Career?

2018-02-24T00:00:00Z

Quite a specific question I’m addressing here based off a post I saw on Reddit but one that has implications beyond the Business Intelligence/Analytics world. Does your current job title really have a major impact on the types of roles you will be considered for in the future?

In the Reddit thread the poster had been offered what, going by the job description, was essentially a Business Intelligence analyst role. Main remit included working on the data warehouse, designing reports, coding SQL queries and using BI tools.

Pretty straightforward you would think. What threw them though was that the job was titled as a Business Analyst, not a Business Intelligence Analyst.

What is a Business Analyst?

My own understanding of what a Business Analyst role should entail is usually a conduit between business units and technology. Someone who can look at business processes, translate that into written documentation and then work with a development team to turn that into technical specs for technology solutions.

I’ve worked with a lot of Business Analysts over the years, mainly as components of project teams, and have found my own definition is only very loosely applied in the wild.

Some have been virtual email forwarding posts while those who really made a substantial impact on their projects went deep with their business understanding as well as working closely with data and technology subject matter experts to cross the divide.

While many had basic Excel and Access skills to pick out some data points, none of those I’ve worked with would meet the job requirements our friend above has been offered in their “Business Analyst” role.

What Should I Do Then?

That being said, it highlights why everyone should give proper thought to including a thorough summary of work under their current and previous roles on their CV / resume.

As a hiring manager with a vacant BI analyst role, I would be much more likely to filter yours into the Maybe pile if your summary included tasks like data warehousing, SQL coding and report design, even if your title was Business Analyst.

If you don’t put it on the page, I’m unlikely to go contacting you for more information just in case. Show and tell. Always.

Bottom line then would be to not get hung up on your job title, it’s only a few words. Your real value and future selling points will be in the work and the experience you have gained doing it, regardless of whether it’s building dashboards or writing process documentation.

Next Steps

Take a look at your CV today and see if you are really selling yourself. If not, change it. Drop me a mail if you want me to run my eye over your CV and point out anything that would be a red flag for me, I’m always happy to help.

Can I Use Microsoft Access For Data Analysis?

2018-02-20T00:00:00Z

Microsoft Access.

Two words guaranteed to bring any data analyst or IT professional out in a cold sweat but is the somewhat tarnished reputation justified and is there any real place for MS Access in your analytical arsenal?

Why does it get so much hate?

“Serious” analysts claim to not touch Access with a twenty foot barge pole. It’s more uncool than your dad dancing at a family wedding. It’s been uncool for so long I keep waiting for hipster analysts to adopt it and start to wear using it as a badge of honour against the entrenched snobbish establishment. (Update from 2024: I’m still waiting.)

In certain circles mentioning that you use MS Access is akin to saying you never want to work professionally in the data world again.

They say it’s too slow. They say it’s not able to handle high volumes of data. They say it’s not fit for more than one user. It’s a toy. It’s too “easy” to use. It’s not secure. It’s too easy to corrupt. It’s not suitable for enterprise applications.

And in many cases those complaints are justified. That’s not to say that it doesn’t have it’s place underpinning both daily workplace tasks and as a quick, easy way to pull some data together and give it a rudimentary once over.

Why should I use it?

All of the things people use as arguments against using MS Access turn out to be reasons for others to use it.

It’s super easy to use – hell yes it is. Setting up tables, queries, forms and reports is a veritable walk in the park. You can import data from spreadsheets, text files or other more suitable data sources like SQL Server and not be afraid of Janey from Accounts overwriting chunks of the production tables with an ill-advised UPDATE statement in Management Studio.
It’s not big enough and my database just kept growing – over time you will eventually max out it’s size limits (somewhere around 2GB from personal observation) but for small scale datasets and quick analysis jobs this will give you more than enough space for very little server space overhead and all you need is MS Access installed on your machine. This should come as standard with most MS Office installs so no fighting with the IT Department to install unwelcome 3rd party software.
It’s not secure enough – well don’t keep business critical or top secret secure data on it in a publicly accessible place then. You don’t need a full user access infrastructure to get started, load it up, fire in your data, run your queries or VBA and it’s all there in front of you.

Maybe I should use it then…

Slow down cowboy/cowgirl. It’s horses for courses and both sides of the debate are technically correct (yes, my rear end does hurt from splinters from sitting on the fence).

Access is never a suitable database system for business critical processes or for something that should be housed in an enterprise level DBMS. Using it to suck in large volumes of data from your main data warehouse will make you want to cry and stamp your feet in frustration. It’s not worth it, trust me, I have the feet stamping scars to prove it over many, many years.

If you have a need to turn some numbers around quickly or investigate a sub-section of a data table then I do recommend having MS Access as part of your toolkit. It’s ease of use and shallow learning curve belie it’s ability to get some vital, if unsung, jobs done in any size of business. It’s not for no reason that MS Access remains the most used desktop database solution in the world.

By all means, know it’s limitations but that doesn’t mean that you should throw the baby out with the bath water. When options are limited, as they often are in highly secure, locked down corporate environments, you’d be surprised just how far you can push this much maligned black sheep of the data analysis family and just what you can get out of it, if you approach the job in a realistic fashion.

3 Relational Database Management Systems For Data Newbies To Get Started On

2018-02-19T00:00:00Z

Choosing which database system to use for your early forays into the world of data analysis can be a foreboding task at the best of times.

While there really isn’t a wrong answer, you can set yourself up for an incredibly steep learning curve that can put the prospective analyst off before they’ve even had a chance to get the training wheels off.

Where To Begin?

My contention in my What Should An Entry Level Data Analyst Know? post that relational databases are still the best way to get up to speed quickly and have the most potential value in your career is borne out by the highly regarded DB-Engines.com rankings for 2018.

Six of the top seven Database Management Systems (DBMS) are Relational Databases with only MongoDB sneaking in at #6 to spoil the Relational DB party.

With that in mind, let’s take a quick look at the top three most readily available Relational DBMS that you should be installing and getting to know right now.

1) MySQL

Second only to parent company Oracle’s main commercial RDBMS on the DB-Engines Rankings, MySQL is the world’s most popular open source database. Used extensively by tech luminaries such as Facebook, Twitter and Youtube, MySQL provides the database backend for many of the web applications you will come across online.

Reliable, quick and easy to get into, you can download the MySQL Community Edition and install it on your local dev machine or find it pre-installed as standard on many popular shared and VPS hosting providers.

I’ve used MySQL for years as the standard backend for WordPress and administered it with a minimum of fuss through the web-based front end phpMyAdmin and their own MySQL Workbench environment.

Download MySQL here.

2) PostgreSQL

Another open source RDBMS that has came on in leaps and bounds over the past few years is PostgreSQL. While not quite as widespread and popular as MySQL, it’s development team do pride themselves on their standards compliance and strong conformity to the ANSI-SQL:2008 standard.

Mightn’t mean a lot to the newbie but that, along with it’s high performance and reliability in high volume environments, have seen it make real inroads into the web application space.

Best of all for those looking to get a start, you can get PostgreSQL for most operating systems, from BSD, Linux, MacOS, Solaris and even Windows.

Again, it’s becoming more widespread on all levels of hosting package as well which saves even having to install it yourself.

Download PostgreSQL here.

3) Microsoft SQL Server

Microsoft have been pushing their SQL Azure cloud database service but it’s still extremely hard to look past their dogged old RDBMS workhorse, SQL Server. I’ve been using it since I started out nearly 20 years ago on SQL Server 7.0 and the whole ecosystem has continued to come on strongly over recent years.

Available on Linux, Docker and MacOS as well as it’s native Windows, you can get a free trial version, a full free Developer edition or the smaller Express edition to get yourself up and running.

Commercial licenses are still a costly alternative to open source options like MySQL and PostgreSQL but there is little argument about the power and quality of the connected tools like Management Studio, SSIS, SSAS and SSRS.

Download SQL Server here.

Which Should I Choose To Best Help My Career?

If you are using any of these three to build up some skills in setting up tables and schemas, building some ETL routines to load data then doing some basic reporting you will find those skills transfer easily across all three.

For websites and working online I’ve predominantly used MySQL over the years.

In industry, earlier in my career especially, I used SQL Server a lot and again a few years ago when I built a sports betting analytics system.

The exact tool or DBMS isn’t really the point at this stage though, it’s the skills and basic understanding of how relational databases work that you need to get more experience with and I’ll cover more of that in future posts.

My Five Stages Of The Analytics Process

2018-02-17T00:00:00Z

(EDIT 12/03/2018: I’ve given this some thought and am sticking a fifth and very necessary stage onto the front of my process here, not that I discounted it at first but I do think it’s a vitally important step for overall success and needs to be in there.)

I’ve covered a bit of the widespread debate on Analysis vs. Analytics in a previous post but to really demonstrate the relationship between the two I’m going to look at the four main stages in my analytics process.

DISCLAIMER #1: This is how I group some of the main terms up and how I understand and apply them in my day to day work and in discussions in the wider community. YMMV and if it didn’t I wouldn’t have seen so much confusion in forums and blog comments.

DISCLAIMER #2: I’m not trying to enforce my method on anyone else, there are a number of sub-levels and complexity to this, don’t write to me and tell me I’m doing it wrong. Paraphrasing Swiss Toni, Data Analytics is very much like making love to a beautiful (wo)man. If you both get to a mutually acceptable end result, does it really matter what the exact order of the steps in between where?

ALAN’S FIVE STAGES ON THE ANALYTICS JOURNEY

1) Requirements Gathering

Yes, I know I missed this one out in the first version of this post but it’s so important to overall success that I’m adding it in now. Learning how to speak to business users (especially those without a technical background) is one of the most important skills an analyst will learn. It will stand by you much longer than any coding or database technical knowledge and will ultimately dictate a large part of your career trajectory.

Listen, pick out what the real pain they are feeling is and use your knowledge and their experience to find the best solution. Often it’s not what they originally ask for. Learn (the hard way if necessary) that the customer is not always right and your job is to help guide them and you’ll have a lot more success in every area of your analytical work .

2) Preparation

Invariably the bulk of your time will be spent at this stage. Gathering, cleaning, prepping, cleaning again, joining datasets, extracting, transforming, loading, unloading, cleaning again. You get the picture.

No point going through with the rest of the steps if your data is in a terrible condition, is full of holes or doesn’t actually contain the really important parts you need for the rest of the journey. Cut corners here and you might as well forget about the rest of it, run a random number generator and go to the pub.

3) Reporting

Covers the bulk of Management Information (MI) and Business Intelligence, this is looking at WHAT has happened. Got yourself an Excel spreadsheet with the weekly sales figures for Widget #568, split by date, region and sales location? That’s Reporting.

Fancy a nice pie chart, line graph or interactive heat map by geographic region for your presentation? Yep, still Reporting (even it does look damn pretty).

4) Analysis

WHY did that happen. Looking at your weekly sales figures and see a massive spike in week 24? Now you’re analysing your reporting. Dig into what happened in the business in the weeks leading up to that point and find a certain location was running a local promotion and they cleaned up when the price dropped for that week – that would be your Analysis.

This is where the Inquisitive Nature of the data analyst comes in. We are all wannabe detectives or else we wouldn’t be in this line of work in the first place. See something strange? Step into your inner Poirot alter ego and start digging. Without understanding the business or being willing to learn, you will never level up at this nor the next stages.

5) Predictive Analytics

Working out WHAT WILL HAPPEN NEXT? The next level up from analysing what has happened to see why that happened is predicting where things will go from here into the future. Algorithms and trend analysis are two tools for building the crystal ball that shows you were you can expect to go in terms of sales or customer sign-ups (or losses).

Get through all five of those stages and you’ll have a lovely clean dataset, a fine spreadsheet or data viz of reportable numbers, a (succinct) story to tell senior management what happened and why and a set of numbers predicting where it will go for the next few weeks/months/years.

Revise. Rinse. Repeat. Retire.

If after all of that you still find yourself worrying whether it was Analysis or Analytics you were doing I suggest asking your manager for more work to do.

Worry about cleansing your data. Worry about not spotting the reason behind a shift in your sign-ups or drop in turnover. Worry about missing a coming trend that will blow your predictions out of the water.

Anything else is just noise and not worth your time or energy.

Data Analytics vs. Analysis – What’s The Difference?

2018-02-17T00:00:00Z

Forget the Python vs. R or Tableau vs. Power BI wars, there is nothing guaranteed to divide the data community quite as much as the Analytics vs. Analysis question.

What is the difference between Analysis and Analytics? Does one look at the past and one to the future? Is one of them reporting and the other data science?

So many questions, so many opinions and, unfortunately, no real concrete definitions that everyone subscribes to and agrees on.

Being controversial (for once…), I’ll say that if you have enough time to get so deep into the debate of what covers which area that you find yourself obsessing and arguing about it online, you probably aren’t giving the actual work the real focus it deserves.

It’s great to know exactly what someone means when they use a certain term but it’s not going to move the dial significantly on your results either way to know that it’s analysis or analytics you were supposed to be doing while you shot witty put-downs back and forward on Reddit or Hacker News.

Let’s Ask An Expert

David Kasik, the Senior Technical Fellow in Visualisation and Interactive Techniques at Boeing, defines the Analysis vs. Analytics divide as this:

“In my terminology, data analysis refers to hands-on data exploration and evaluation.

Data analytics is a broader term and includes data analysis as necessary subcomponent.

Analytics defines the science behind the analysis.”

If, by Kasik’s definitions, Analysis is a part of Analytics, what are we actually looking to distinguish here? I think it’s fair to say that the simple trope of Analysis looking at the past and Analytics looking at the Future isn’t quite correct.

Neither is being able to use the tools involved to split the two. If I use Excel for analysis, does that mean anything done in Excel can’t be Analytics? Of course not yet I’ve seen the argument used all over the web.

For what it’s worth, I agree with David Kasik.

Analysis, for me, is one of four main stages of an Analytics Process and I dig deeper into that in the next post – My Five Stages of the Analytics Process.

What Should An Entry Level Data Analyst Know?

2018-02-09T00:00:00Z

Having recruited analysts for a number of years, I’ve noticed a worrying trend in people who are eager to join the profession but don’t have an analyst or data background.

Recent graduates or folks with a few years of experience in other fields are being put off before they even send out an application form for an entry level BI or data analyst job because they’ve self-certified themselves as being under-qualified.

Imposter Syndrome is a real problem no matter what stage of your career you are at. And in the fast paced, always changing world of tech, that applies doubly so.

How do you know what you don’t know?

Reading articles from the tech press lionising data science as being the new rock n’ roll doesn’t exactly help. When you hear about PhDs scrapping it out for top data jobs it’s hard to see how your Bachelor’s degree (or, shock horror, no degree at all) can ever get you to an interview, never mind get selected for a role.

As a profession it’s important that recruiting managers speak out more about the reality of what they really need. We need to do all we can to stop losing a stream of potentially very capable analysts before they’ve even ran their first SELECT query.

Obviously, the bar rises as we go up the experience levels. For entry level analyst or BI developer jobs I have three tick-boxes that I want to see and NONE of them require a PhD.

Three requirements for entry level roles

1. SQL.

It’s not sexy. It’s not particularly cool. In fact, if the likes of Spark are the Flying V of the data world then SQL is the Fender Telecaster. It’s more of a utilitarian workhorse but (within reason) you can do 90%+ of the reporting and analysis you will ever need at this level with only a rudimentary knowledge of it. IMHO there is no better grounding available for minimal time spent educating yourself and that applies across all industries and flavours of analysis.

2. Relational Databases.

Parts of the tech world have moved on to pastures new with NoSQL databases like MongoDB and Cassandra but the vast majority of tech and industry still runs on relational databases and will do for a very long time.

Working hand in glove with your SQL education from #1, take a little time to get relational database structure and joins into your head. From MySQL to Microsoft SQL Server you will encounter them in virtually every BI or data analyst job you apply for.

3. Inquisitive Nature.

This one isn’t a technical requirement and will grow over time with experience but an inquisitive nature is my #3 pre-requisite for getting your foot in the door in the world of data analysis. Run a report and see that one month has twice the volume of sales as another month but decide the query seems right and send it out anyway? I don’t think so.

You are in a prime position to ask questions in whatever business or industry you are in and learn everything about it from the inside. If you can’t apply the natural nosey nature of a Lieutenant Columbo when you have all of the raw data at your fingertips then maybe you aren’t cut out for this job after all. Have a look at the underlying transactions and see one outlier that is skewing the whole month’s total? Bingo, now you’re on my radar.

I can’t speak for all of the recruiters at the companies I’ve worked for but, to me, if you can demonstrate these three things in the course of your CV and back that up in person at interview you’ll be on my short list for the role.

You are not alone

My underlying point is that it’s normal to be put off by thinking everyone knows more than you or that the educational choices younger you made have precluded you from getting into a certain industry or profession.

Maybe you won’t get an interview for a Senior Data Scientist role at Google if you don’t have a doctorate from an Ivy League school. That doesn’t mean that you can’t work your way into data analysis or business intelligence roles from another angle.

Not everyone who picks up a guitar can become a rock star. That doesn’t mean that wedding singers can’t make a very good career for themselves doing something they love that brings real value to a lot of people.

The world doesn’t need another Bono. Sometimes we all just need a Robbie Hart.

So where do I begin?

To kickstart your career in data analysis, check out any of the Intro to SQL courses online at Lynda.com, Coursera or Udemy.