How to become big data – data analyst

Anyone who works in the tech industry is aware of the rising demand of Analytics/ Machine learning professionals. More and more organisations have been jumping on to the data driven decision making bandwagon, thereby accumulating loads of data pertaining to their business. In order to make sense of all the data gathered, organisations will require Big Data Analysts to decipher the data.

  Data Analysts have traditionally worked with pre formatted data, that was served by the IT departments, to perform analysis. But with the need for real time or near-real time Analytics to serve end customers better and faster, analysis needs to be performed faster, thereby making the dependency on IT departments a bottleneck. Analysts are required to understand data streams that ingest millions of records into databases or file systems, Lambda architecture and batch processing of data to understand the influx of data.

Also analysing larger amounts of data requires skills that range from understanding the business complexities, the market and the competitors to a wide range of technical skills in data extraction, data cleaning and transformation, data modelling and statistical methods.

Analytics being a relatively new field, is struggling to resource the market demands with highly skilled Big Data Analysts. Being a Big Data Analyst requires a thorough understanding of data architecture and the data flow from source systems into the big data platform. One can always stick to a specific industry domain and specialize within that, for example Healthcare Analytics, Marketing Analytics, Financial Analytics, Operations Analytics, People Analytics, Gaming Analytics etc. But mastering the end-to-end data chain management can lead to plenty of opportunities, irrespective of industry domain.

The entire Data and Analytics suite includes the following gamut of stages:

  • Data integrations – connecting disparate data sources
  • Data security and governance – ensuring data integrity and access rights
  • Master data management – ensuring consistency and uniformity of data
  • Data Extraction, Transformation and Loading – making raw data business user friendly
  • Hadoop and HDFS – big data storage mechanisms
  • SQL/ Hive / Pig – data query languages
  • R/ Python –  for data analysis and mining programming languages
  • Data science algorithms like Naive Bayes, K-means, AdaBoost etc. – Machine learning algorithms for clustering, classification
  • Data Architecture – solutionizing all the above in an optimized way to deliver business insights

The new age data analysts or a versatile Big Data Analyst is one who understands the complexity of data integrations using APIs or connectors or ETL (Extraction, Transformation and Loading), designs data flow from disparate systems keeping in mind data security and quality issues, can code in SQL or Hive and R or Python and is well acquainted with the machine learning algorithms and has a knack at understanding business complexities.

Since Big Data and Analytics is constantly evolving, it is imperative for anyone aiming at a career within the same, to be well versed with the latest tech stack and architectural breakthroughs. Some ways of doing so:

  • Following knowledgeable industry leaders or big data thought leaders on Twitter
  • Joining Big Data related groups on LinkedIn
  • Following Big Data influencers on LinkedIn
  • Attending events, conferences and seminars on Big Data
  • Connecting with peers within the Big Data industry
  • Last but not the least (probably the most important) enrolling in MOOC (Massive Open Online Course) and/ or Big Data books

Since Analytics is a vast field, encompassing several operations, one could choose to specialise in parts of the Analytics chain like data engineers – specializing in highly scalable data management systems or data scientists specializing in machine learning algorithms or data architects – specializing in the overall data integrations, data flow and storage mechanisms. But in order to excel and future proof a career in the world of Big Data, one needs to master more than one area. A data analyst who is acquainted with all the steps involved in data analysis from data extraction to insights is an asset to any organization and will be much sought after!

Advertisement

Managing corporate innovation

The time is ripe for corporates to embark on a journey of innovation. Having said that it is a rocky road for enterprises that have been in existence for half a century or more, having made hay when the sun shined and thus gathering a good amount of legacy systems and processes on the way. Some organisations that have have chosen to invest in innovation prefer to run the innovation labs separately, far from the core business, the reason being that innovation should not get bogged down by corporate beauracracy. A big part of innovation is experimentation — experimentation with thoughts and ideas and prototyping. Business leaders have a herculean task of ensuring innovation for the businesses of the future while continuously improving the incumbent corporate machinery that generates the revenue necessary for future investments.

Innovation is about inventing new products or services that solve customer needs and can be monetised. Innovation involves trial and error and learning from the same, a structured method of experimentation leads to better tracking of ROI. The pentathlon framework of innovation articulates the methodology from ideation to market launch. The influx of ideas, which form the innovation backlog, can be either disruptive ideas or new ways of resolving existing business challenges.

Each part of the innovation funnel has to imbibe a fail fastand and an iterative approach for further ideation with feedback loops. Every new invention undergoes inception, improvements and adoption followed by stability and subsequent depreciation. If there are many trains of thought in the innovation projects pipeline, there has to be an order in the way of prioritizing the projects. Prioritization of innovation projects should depend on

  • Marketability — is there a potential market for the product/service?
  • Feasibility — is it possible to deliver the project with the resources the company can afford?
  • ROI — How soon is the breakeven point?
  • Time to market — how soon can the MVP be launched?

The implementation of innovation projects are not very different from other corporate projects, involving a portfolio selection based on the prioritization and business urgency. The next stage involves scoping and development, preceeded by prototyping before scaling. Post validation and launch, the impact has to be measured and analyzed to understand the product adoption and customer experience. The insights from measuring innovation efforts lead to newer ideas or incremental improvements to exisiting business processes and/or the innovative product under consideration. There has to be a relentless flow of insights into the innovation funnel to finetune the ideas being considered for prioritization, implementation and launch.

The very existence of an innovation foundry within a coporate house has to be justified to the investors and shareholders. As a means to create a process of accountability there should be KPIs defined, some examples being:

  • Number of ideas considered for prioritization
  • Number of ideas that were productionalized
  • Number ideas that have lead to business process improvement
  • Value added (Value = Accrued benefits — costs)
  • Time to breakeven

The major factors that influence an innovative culture at a corporate level depend on the company culture – the ability to thrive during change and adaptability to new market conditions. It is of utmost importance to recognise and reward the people who contribute to innovation and showcase and communicate the succesful results. Innovation labs within corporates cannot be run separately forever, the outputs from the innovation exercises need to flow back into the day to day business and the challenges from the core business need to be worked upon in the innovation labs.

With innovation being a crucial area of focus for most organisations, there are a lot of ideas floating regarding innovation management. It can be rather chaotic with buzzwords like Big data, AI, Robotics, Augmented Reality etc. being thrown around callously. Technology is a great enabler for business strategy, but great ideas arise from understanding customer requirements and the lack of products or services that fulfill the same.The most significant point of focus in any corporate innovation should be customer centricity and business gain.

Analytics – Implications on Digitization

capture

Digital is all about data, contrary to the prevalent method of creating Analytics as a silo all by itself. Analytics should be seen as one of the fundamental underlying processes that support the core business processes like product development, marketing, sales, customer relationship, finance and innovation. Data and Analytics provide value to core processes, for continuous improvement.

Most organisations are keen on innovation. Innovation could entail new market opportunities and could be an entirely new value proposition, discovered on a strategy canvas. But innovation could also be a by-product of a business process improvement. Such opportunities can only arise when business processes are tracked, measured and analyzed. Organisations that indulge in hypothesis driven product development or mass marketing could benefit by introducing  a data driven approach to the above processes, thereby uncovering the customer needs and product usage. Businesses may launch products with a certain outcome in mind, but sales, social media feedback and web analytics data may have another story to tell. It is in this story, that new opportunities can be unearthed. Understanding customer behavior is a way of discovering new marketing and/or product/service development opportunities.

Many organisations investing heavily in digitization, charting customer journeys, aimed at improving customer experience across all touch points, seemingly forget to make Analytics an integral part of this process.  The key to understanding  major business drivers like customer retention, ROMI, growth, customer engagement, monetization, finding new customer segments depend on deciphering the business data generated.

Analytics, therefore should be embedded in all business processes to capture the way the end customers perceive products or services or marketing and branding efforts made by any organisation. Analyzing the business data from existing processes could possibly give rise to future business prospects. To tread on a path of continuous improvement and innovation, companies will have to make Analytics a fundamental part of every business strategy.