How to become big data – data analyst

Anyone who works in the tech industry is aware of the rising demand of Analytics/ Machine learning professionals. More and more organisations have been jumping on to the data driven decision making bandwagon, thereby accumulating loads of data pertaining to their business. In order to make sense of all the data gathered, organisations will require Big Data Analysts to decipher the data.

  Data Analysts have traditionally worked with pre formatted data, that was served by the IT departments, to perform analysis. But with the need for real time or near-real time Analytics to serve end customers better and faster, analysis needs to be performed faster, thereby making the dependency on IT departments a bottleneck. Analysts are required to understand data streams that ingest millions of records into databases or file systems, Lambda architecture and batch processing of data to understand the influx of data.

Also analysing larger amounts of data requires skills that range from understanding the business complexities, the market and the competitors to a wide range of technical skills in data extraction, data cleaning and transformation, data modelling and statistical methods.

Analytics being a relatively new field, is struggling to resource the market demands with highly skilled Big Data Analysts. Being a Big Data Analyst requires a thorough understanding of data architecture and the data flow from source systems into the big data platform. One can always stick to a specific industry domain and specialize within that, for example Healthcare Analytics, Marketing Analytics, Financial Analytics, Operations Analytics, People Analytics, Gaming Analytics etc. But mastering the end-to-end data chain management can lead to plenty of opportunities, irrespective of industry domain.

The entire Data and Analytics suite includes the following gamut of stages:

• Data integrations – connecting disparate data sources
• Data security and governance – ensuring data integrity and access rights
• Master data management – ensuring consistency and uniformity of data
• Data Extraction, Transformation and Loading – making raw data business user friendly
• Hadoop and HDFS – big data storage mechanisms
• SQL/ Hive / Pig – data query languages
• R/ Python –  for data analysis and mining programming languages
• Data science algorithms like Naive Bayes, K-means, AdaBoost etc. – Machine learning algorithms for clustering, classification
• Data Architecture – solutionizing all the above in an optimized way to deliver business insights

The new age data analysts or a versatile Big Data Analyst is one who understands the complexity of data integrations using APIs or connectors or ETL (Extraction, Transformation and Loading), designs data flow from disparate systems keeping in mind data security and quality issues, can code in SQL or Hive and R or Python and is well acquainted with the machine learning algorithms and has a knack at understanding business complexities.

Since Big Data and Analytics is constantly evolving, it is imperative for anyone aiming at a career within the same, to be well versed with the latest tech stack and architectural breakthroughs. Some ways of doing so:

• Following knowledgeable industry leaders or big data thought leaders on Twitter
• Joining Big Data related groups on LinkedIn
• Following Big Data influencers on LinkedIn
• Attending events, conferences and seminars on Big Data
• Connecting with peers within the Big Data industry
• Last but not the least (probably the most important) enrolling in MOOC (Massive Open Online Course) and/ or Big Data books

Since Analytics is a vast field, encompassing several operations, one could choose to specialise in parts of the Analytics chain like data engineers – specializing in highly scalable data management systems or data scientists specializing in machine learning algorithms or data architects – specializing in the overall data integrations, data flow and storage mechanisms. But in order to excel and future proof a career in the world of Big Data, one needs to master more than one area. A data analyst who is acquainted with all the steps involved in data analysis from data extraction to insights is an asset to any organization and will be much sought after!

Advertisement

Managing corporate innovation

The time is ripe for corporates to embark on a journey of innovation. Having said that it is a rocky road for enterprises that have been in existence for half a century or more, having made hay when the sun shined and thus gathering a good amount of legacy systems and processes on the way. Some organisations that have have chosen to invest in innovation prefer to run the innovation labs separately, far from the core business, the reason being that innovation should not get bogged down by corporate beauracracy. A big part of innovation is experimentation — experimentation with thoughts and ideas and prototyping. Business leaders have a herculean task of ensuring innovation for the businesses of the future while continuously improving the incumbent corporate machinery that generates the revenue necessary for future investments.

Innovation is about inventing new products or services that solve customer needs and can be monetised. Innovation involves trial and error and learning from the same, a structured method of experimentation leads to better tracking of ROI. The pentathlon framework of innovation articulates the methodology from ideation to market launch. The influx of ideas, which form the innovation backlog, can be either disruptive ideas or new ways of resolving existing business challenges.

Each part of the innovation funnel has to imbibe a fail fastand and an iterative approach for further ideation with feedback loops. Every new invention undergoes inception, improvements and adoption followed by stability and subsequent depreciation. If there are many trains of thought in the innovation projects pipeline, there has to be an order in the way of prioritizing the projects. Prioritization of innovation projects should depend on

• Marketability — is there a potential market for the product/service?
• Feasibility — is it possible to deliver the project with the resources the company can afford?
• ROI — How soon is the breakeven point?
• Time to market — how soon can the MVP be launched?

The implementation of innovation projects are not very different from other corporate projects, involving a portfolio selection based on the prioritization and business urgency. The next stage involves scoping and development, preceeded by prototyping before scaling. Post validation and launch, the impact has to be measured and analyzed to understand the product adoption and customer experience. The insights from measuring innovation efforts lead to newer ideas or incremental improvements to exisiting business processes and/or the innovative product under consideration. There has to be a relentless flow of insights into the innovation funnel to finetune the ideas being considered for prioritization, implementation and launch.

The very existence of an innovation foundry within a coporate house has to be justified to the investors and shareholders. As a means to create a process of accountability there should be KPIs defined, some examples being:

• Number of ideas considered for prioritization
• Number of ideas that were productionalized
• Number ideas that have lead to business process improvement
• Value added (Value = Accrued benefits — costs)
• Time to breakeven

The major factors that influence an innovative culture at a corporate level depend on the company culture – the ability to thrive during change and adaptability to new market conditions. It is of utmost importance to recognise and reward the people who contribute to innovation and showcase and communicate the succesful results. Innovation labs within corporates cannot be run separately forever, the outputs from the innovation exercises need to flow back into the day to day business and the challenges from the core business need to be worked upon in the innovation labs.

With innovation being a crucial area of focus for most organisations, there are a lot of ideas floating regarding innovation management. It can be rather chaotic with buzzwords like Big data, AI, Robotics, Augmented Reality etc. being thrown around callously. Technology is a great enabler for business strategy, but great ideas arise from understanding customer requirements and the lack of products or services that fulfill the same.The most significant point of focus in any corporate innovation should be customer centricity and business gain.

Chasing dopamine — The Neo-generalist

Dopamine a chemical released by nerve cells, which plays a major role in reward-motivated behavior. For some of us the adrenaline rush comes seeking and mastering new challenges and then moving onto the next.

Chasing dopamine

This habit of seeking new problems to solve goes beyond job titles, roles and responsibilities, educational background and age. These polymaths, knowledge seekers or autodidacts bloom where planted. They thrive when things need to be fixed, they use the knowledge gained from one industry on others, experience with one method leading to another. Monotonous tasks seem arduous.

“It still holds true that man is most uniquely human when he turns obstacles into opportunities.”–Eric Hoffer

I find it hard to grasp that organisations hire specialists to break down silos to facilitate continuous flow of information between different business units but individuals on the other hand are encouraged to be specialists within a certain discipline. Unless there are people who can handle multidisciplinary roles transcending departmental borders, I do not see a solution to organisational silos. The divide between tech and business is one such area of concern. People with tech skills are assumed to have little grasp of business acumen while people with strong business understanding are assumed to have limited IT proficiency and people with both business and tech expertise are perceived as average in both. That’s due to our obsession with specialism. Generalists are looked down upon, likened to jack of several trades and master of none. On the contrary, the monkey minds are an asset, being able to connect several dots and improvise solutions based on their creative thinking, envisioning paths beyond their job titles.

My constant dilemma

Hailing from India, Jugaad was part of the everyday vocabulary. Jugaad in Hindi means makeshift solutions which requires resourcefulness. Resourcefulness is not part of any syllabus, it comes naturally when there’s a scarcity of means. Being able to do more with less. Doing more has to do with understanding several disciplines to put together a solution beyond frameworks and recognized theories.

I came across a book The Neo-Generalist by Kenneth Mikkelsen and Richard Martin which was sort of narrating my mental state — the nomad state, in search of the next problem irrespective of domain, technological challenges or borders. We, the neo-generalists are happy as long as our brains are being harnessed and we are involved in something meaningful. If you identify yourself as neo-generalist then it is a must read. You’re not alone, there’s a whole tribe of us, restless souls, trying to juggle several disciplines at the same time and loving every moment of it!

Settle down — this word does not appeal to neo-generalists. Constant learning and treading on paths not previously traveled are our only focus.

“Listen baby, ain’t no mountain high,
Ain’t no valley low, ain’t no river wide enough”

AARRR Metrics for a Fintech

 Lets assume this is a case study for a Fintech company’s KPI definition.

Company X is a Fin Tech company providing payment solutions to SME and small businesses via mobile app, card reader and NFC. Company X solutions provide bookkeeping and analytics features to its customers by means of tracking its product usage and events.

Tracking mobile app usage and web sites are done by using web and mobile analytics tools such as Localytics, Flurry, Google Analytics, Tealium, Xiti etc. But in some cases the data from the analytics tools are not enough to deduce conclusions and hence require additional data from various systems such as CRM, Financial transaction systems, CMS and inventory control systems. Due to the need for blending data from disparate systems, a data strategy needs to be defined and a robust and scalable data architecture needs to be in place.

I would like to provide two relevant blog posts from my own blog that point to the concepts of growth hacking and data blending.

Data Value Chain

Growth Hacking

KPIs

Data monetization for the growth of businesses, entails tracking user behavior both online and offline to optimize products and processes. A list of KPIs or metrics to measure product usage and means of revenue generation are used as a guideline for data monetization efforts. Whether it is to assess global performance of a site, measure the impact of a specific campaign or product feature change, a set of indicators will be needed to focus on the changing parameters.

There are 5 metrics defined by Dave McClure : Acquisition, Activation, Retention, Referrals, Revenues or AARRR also known as the pirate metrics that serve as a good indicator of business growth.

For each of the metric area there are several KPIs defined. For each of the KPIs there are again 4 essential components or ways of analyzing:

• Data points – Data points are the points in the app or site that generate interesting insights about the business in question. It could be individual features in the product or events.
• Funnels – Setting up funnels ensures tracking all the steps that lead to completion of a particular process on the site or app like tracking steps that lead to an online payment page or the steps that lead to a signing up for a newsletter.
• Segmentation – Segmenting the potential and existing customer base to be able to understand their wants and needs in order to be able to serve them better, which is a means of revenue generation. Segmentation can be
• Behavioral – Users who spend lot of time on the site or app, frequently login or rarely login, browsers, visitors that leave without making purchases or visitors that make purchases
• Technical – The browsers used, the OS versions, devices used and if the users have saved the site as a bookmark or enter the site through search engines or social networks
• Demographics – Clustering users based on their age, gender, location etc.
• Cohorts – Cohorts are also a type of segmentation but more from a time series perspective to be able to compare data sets at different points of time. For example checking trends or shopping behaviors at different points in time.

The pirate metrics for product usage can be broadly classified as below:

Acquisition

The process of acquiring customers, which would mean tracking new customers that visit the site or download the app or search the product. The KPIs for acquisition would include all the metrics that indicate a growth or changing trends:

• Number of unique visitors
• %mobile traffic
• %web traffic
• % traffic from social networks
• % traffic from search engines
• Number of app downloads
• Visit trends
• Page view trends
• App Download trends
• New User Account Creation Rate
• Bounce Rate
• Funnel analysis for conversion
• Number of new customers in the last Month/Quarter
• Number of new customers YoY growth
• Campaign effectiveness – measuring the number of customers signing up or deregistering

 

Activation

When the users have logged in and have started using the product, the usage needs to be tracked to be able to further develop the product for better customer experience.

• Page views
• Time spent on the site
• Hourly traffic
• Seasonal traffic
• Monthly Active Users
• Number of paying customers in the last Month/Quarter
• Number of paying customers YoY growth
• Type of payments
• Types of Merchants (small/SME/seasonal)
• Types of businesses/industry
• Type of most sold items
• Customer Segmentation (Technical, Demographics, Behavioral) to understand customer’s need to use the product to improve product development

 

Retention

Retention is the process of retaining existing customers by continued service leading to customer satisfaction. Measuring the factors that lead to retaining customers is a good indicator.

• Number of returning customers
• Average time for transaction
• of transactions
• Transaction failure rate
• Number of transaction per payment type
• Peak hour
• Peak Season
• Types of Merchants
• Average revenue per Merchant
• Average Revenue per Merchant per branch/Industry type
• Average time taken for deposit to merchants
• Competitor Analysis through web/Facebook crawling
• FaceBook engagement (Likes, Shares, Comments) per Month/week
• Number of Complaints per category of complaint type
• App Store Ratings/Review trends
• Text Analysis for tweets/ Facebook comments
• Number of cash payments Vs Card payments

 

Referrals

When the customer satisfaction index is high, the customers refer the products to others thereby acting as brand ambassadors. Referrals are a means to measure customer satisfaction because customers refer the product only when they are themselves happy with the product usage.

• Number of visits coming from social media
• Number of site entry from Facebook ads
• Number of shares on Facebook
• Text analysis of tweets and Facebook messages

Revenue

One of the most important part of a business is revenue generation as revenue is not only the sustenance factor but an indicator of growth.

• Total Payment Volume
• Total Net Revenues
• Transaction losses
• Net revenue YoY growth
• Net revenue YoY growth per type of business
• Net Revenue per type of card (Master/Visa)
• Sales turnover of customers
• Number of transactions per Month/Quarter
• Number of transactions per type of business
• Number of transactions per Location
• Net revenue per platform (mobile app {ios/Android/ipad}/ card reader/NFC)
• Net revenue per type of merchant
• Average revenue per client
• Average value per transaction
• Peak volume of transactions per hour
• Peak volume of transactions per hour per location per type of business (to be able to suggest to similar merchants about the optimum time and hour of transaction)
• %churn
• %churn per type of merchant/type of business/Month/Quarter
• Average Selling price per type of Merchant per type of business
• Average Selling price per type of Merchant per type of business trends – Monthly/Quarterly/Seasonal
• Number of customers that have applied for loan
• Type of customers (business/demographics) that have applied for loan via Company X

 

Conclusion

Product usage tracking to improve the overall product features and outreach is an iterative process involving several processes like continuous A/B testing, UX strategy, Analytics, ideation and product development. In order to create state of the art products, Company X needs to know who their audience is and how the product will make it easy for businesses to sell. By tracking product usage, the aim should be to learn deeply about the customers’ needs and behaviors to be able to generate great solutions, proactively. Iterating towards the solution that creates the most value by collecting and analyzing data is the key.

How to make your MVP the talk of the town

So you are launching your product or planning a launch but are you sure your product is going to make the waves? You don’t want to be spending your time, energy, $$ and efforts on a product, in vain. Maybe you are still working in a 9-5 job and are testing the waters in the startup community? Then it becomes, even more relevant to get your ideas validated by a targeted audience before you take the plunge, lock, stock and barrel!

Let’s get discussing how you can be sure that your MVP is sustainable and creates the necessary ripples in the market.

• Get feedback early in the process. Feedback from friends and family, who can be brutally honest with you. Doing so will also make you think deeply and clearly about your value proposition. Use your family and circle of friends to spread the word about your product.

          “If you can’t explain it simply, you don’t understand it well enough” – Albert Einstein

• Meetup.com – Gather a bunch of like minded people and brain storm your ideas. You will find out if someone has already treaded the path you are planning, learnings and consequences. You just might stumble upon your business partner or a seed investor.
• Social Media Marketing – Get going on the social media bandwagon, post information and images relevant to your product and get some feedback. Target your tribe through social media, utilise influencer marketing to check how your MVP is percieved. Follow other similar product brands and do some competitive analysis.
• Explainer video – Shoot an excellent graphical and easy to understand video while at the same time highlighting the features that define your MVP. Get this video out there on social media and analyze the popularity.
• Crowdsource – your branding and advertising efforts. Make your targeted customers be a part of the product development journey. They will not only feel connected to your would be product but also contribute to the making by with useful feedback. If they like it they’ll love to be brand ambassadors.
• Demand Vs Supply – Last but not the least check the demand of the product you are launching. If there already exist a dozen such products then it probably will be an upphill task fighting the competition. Thinking out of the box and finding an area where there is a demand supply gap is the wisest thing to do unlesss your product is so radical that it changes the way people have been experiencing other similar products.

You know you have a start-up itch when….

Just like one size does not fit all, not everyone is cut out for the start-up toil. It takes grit, determination and a tenacious can-do spirit apart from having a sustainable idea that could actually sell. So you want to find out if you have it in you what it takes?

Check if:

• you’re too wild to be tamed and bound by 9-5 jobs, run of the mill work and bosses? A lot of us hate the above but that doesn’t make us the start-up kinda folk
• Status quo is alien to you? To get your adrenaline pumping you need challenges? You’re always thinking out of the box and impatient and fearless about trying new stuff? You could be a person of that ilk!
• You’re full of well thought through ideas? You’ve indoctrinated in you that “if life hands you lemons you’re going to make Caipirinha out of it”? You see solutions only, problems are a learning experience for you? Oh dear! You could very well be one of those!
• You have a well thought through business plan with an adversity fund/plan, to keep you afloat while you’re testing the waters? If yes, you’re meticulous, man!
• You know how to design great products, network, market and sell your products? You may not be a master of all but you certainly can be a jack of few. If you’re going to branch out on your own, you’ll not have a huge capital to get the jazziest of analytics products or hire expensive consultants from the inception of your company
• In continuation with the above, you don’t mind getting your hands dirty and burning midnight oil to realize your dream?
• Failure doesn’t deter you? What are you waiting for??

If you’ve gone through the above list and realized you embody most of the above traits, then jump on the band wagon… the world is your stage!