How to become big data – data analyst

Anyone who works in the tech industry is aware of the rising demand of Analytics/ Machine learning professionals. More and more organisations have been jumping on to the data driven decision making bandwagon, thereby accumulating loads of data pertaining to their business. In order to make sense of all the data gathered, organisations will require Big Data Analysts to decipher the data.

  Data Analysts have traditionally worked with pre formatted data, that was served by the IT departments, to perform analysis. But with the need for real time or near-real time Analytics to serve end customers better and faster, analysis needs to be performed faster, thereby making the dependency on IT departments a bottleneck. Analysts are required to understand data streams that ingest millions of records into databases or file systems, Lambda architecture and batch processing of data to understand the influx of data.

Also analysing larger amounts of data requires skills that range from understanding the business complexities, the market and the competitors to a wide range of technical skills in data extraction, data cleaning and transformation, data modelling and statistical methods.

Analytics being a relatively new field, is struggling to resource the market demands with highly skilled Big Data Analysts. Being a Big Data Analyst requires a thorough understanding of data architecture and the data flow from source systems into the big data platform. One can always stick to a specific industry domain and specialize within that, for example Healthcare Analytics, Marketing Analytics, Financial Analytics, Operations Analytics, People Analytics, Gaming Analytics etc. But mastering the end-to-end data chain management can lead to plenty of opportunities, irrespective of industry domain.

The entire Data and Analytics suite includes the following gamut of stages:

• Data integrations – connecting disparate data sources
• Data security and governance – ensuring data integrity and access rights
• Master data management – ensuring consistency and uniformity of data
• Data Extraction, Transformation and Loading – making raw data business user friendly
• Hadoop and HDFS – big data storage mechanisms
• SQL/ Hive / Pig – data query languages
• R/ Python –  for data analysis and mining programming languages
• Data science algorithms like Naive Bayes, K-means, AdaBoost etc. – Machine learning algorithms for clustering, classification
• Data Architecture – solutionizing all the above in an optimized way to deliver business insights

The new age data analysts or a versatile Big Data Analyst is one who understands the complexity of data integrations using APIs or connectors or ETL (Extraction, Transformation and Loading), designs data flow from disparate systems keeping in mind data security and quality issues, can code in SQL or Hive and R or Python and is well acquainted with the machine learning algorithms and has a knack at understanding business complexities.

Since Big Data and Analytics is constantly evolving, it is imperative for anyone aiming at a career within the same, to be well versed with the latest tech stack and architectural breakthroughs. Some ways of doing so:

• Following knowledgeable industry leaders or big data thought leaders on Twitter
• Joining Big Data related groups on LinkedIn
• Following Big Data influencers on LinkedIn
• Attending events, conferences and seminars on Big Data
• Connecting with peers within the Big Data industry
• Last but not the least (probably the most important) enrolling in MOOC (Massive Open Online Course) and/ or Big Data books

Since Analytics is a vast field, encompassing several operations, one could choose to specialise in parts of the Analytics chain like data engineers – specializing in highly scalable data management systems or data scientists specializing in machine learning algorithms or data architects – specializing in the overall data integrations, data flow and storage mechanisms. But in order to excel and future proof a career in the world of Big Data, one needs to master more than one area. A data analyst who is acquainted with all the steps involved in data analysis from data extraction to insights is an asset to any organization and will be much sought after!

Advertisement

Analytics – Implications on Digitization

Digital is all about data, contrary to the prevalent method of creating Analytics as a silo all by itself. Analytics should be seen as one of the fundamental underlying processes that support the core business processes like product development, marketing, sales, customer relationship, finance and innovation. Data and Analytics provide value to core processes, for continuous improvement.

Most organisations are keen on innovation. Innovation could entail new market opportunities and could be an entirely new value proposition, discovered on a strategy canvas. But innovation could also be a by-product of a business process improvement. Such opportunities can only arise when business processes are tracked, measured and analyzed. Organisations that indulge in hypothesis driven product development or mass marketing could benefit by introducing  a data driven approach to the above processes, thereby uncovering the customer needs and product usage. Businesses may launch products with a certain outcome in mind, but sales, social media feedback and web analytics data may have another story to tell. It is in this story, that new opportunities can be unearthed. Understanding customer behavior is a way of discovering new marketing and/or product/service development opportunities.

Many organisations investing heavily in digitization, charting customer journeys, aimed at improving customer experience across all touch points, seemingly forget to make Analytics an integral part of this process.  The key to understanding  major business drivers like customer retention, ROMI, growth, customer engagement, monetization, finding new customer segments depend on deciphering the business data generated.

Analytics, therefore should be embedded in all business processes to capture the way the end customers perceive products or services or marketing and branding efforts made by any organisation. Analyzing the business data from existing processes could possibly give rise to future business prospects. To tread on a path of continuous improvement and innovation, companies will have to make Analytics a fundamental part of every business strategy.

Data integration is not a choice!

Every organization irrespective of industry has several business processes, each business process being supported by several IT products. Each of these IT products have an insurmountable amount of information that can generate insights which are paramount for any organization. Businesses that have been around for a while have obsolete processes and legacy systems that support the same. A typical organization independent of industry has transaction processing systems, CRM systems, ERP, billing and business analytics solutions. Each solution in itself is a silo if not integrated with the rest of the solutions. Granted that each of these solutions harbour valuable information but the the information residing in each system does not generate a holistic view of the business.

Integrating the silos is a Herculean task, or so it may seem, if the solutions are outdated and do not support APIs, plug-ins and adapters. Most CRM, ERP, Marketing automation products, lately are equiped with some form of connector, enabling data blending. If an organization has systems that do not support the above, then it is wise to migrate or upgrade the solutions to versions compatible with data extraction. Migrating legacy systems is a rocky road but the trade off being elimination of data silos. Often the implementation cycle of new software solutions are so long that the idea becomes outdated even before the roll out. Ofcourse there exist solutions with shorter time-to-market, for example data analytics platform that are run on Spark have a faster implementation cycle and are scalable, providing the flexibility that growing businesses need.

It was not long ago that marketing and data analytics borders got blurred due to new business needs. This has resulted in complex technological challenges. Not all businesses have the budget and resources to invest in migrating and upgrading most of the legacy systems. But in order to appease todays demanding customers, data integration is the key. No customer would like to remember or rummage through their homes to find old reciepts or mails when they call the customer care for a service or to complain. They would very much expect that on identifying themselves, the customer care representative not only solves their grievances but also comes up with suggestions to improve their customer lifecycle, which can be only attained by integrating data from disparate systems to gain a 360 degree view of the customer journey. Data integration, thus is a matter of being in business or out.

To start with, businesses should identify each data silo that exists and the function that each of them fulfill. (There maybe exist examples of one business process that is fulfilled by several software solutions. If an organization lacks data governance, then the number of redundant solutions and products can be plenty.) Listing and mapping business processes to softare solutions clarifies the current architecture. The next process is

• To identify the to-be roadmap
• Map solutions that support data blending, to each of the business process 

The solutions that are adapted for new age businesses require to embody the following characteristics:

• Easy to implement
• Short implementation time
• Compatability with a wide range of disparate systems
• Easy to implement data security and access rights
• Scalable
• Forward compatible

Businesses need technology that support business gain and growth and the ever changing rules of the game (read disrutption).

The Start-up Lifecycle

The start-up industry may appear very glamorous from the surface, but it entails endless meticulous planning and back breaking hard work, it’s anything but a cake-walk. Most success stories that come into limelight have already gone through testing times and been bitten by failure at some stage or the other and have survived the winds of change.

The typical stages in a start-up are more or less as below:

• Ideate – An idea that has been incubating for a while gets more concrete and is at a stage where it can be implemented. It has to be beyond the pen and paper stage, on the path to a more concrete objective.
• Feasibility study – validate your idea, think through all the possibilities, market demands, fall back plan and get an initial feedback from friends and family about the viability of the idea.
• Conceptualize the idea into a business case, planning the inception, the initial capital required, the source of fund, the launch of beta version of the product and the marketing of the same to acquire a customer base.
• While you’re at it, you’ll need to create an online presence in this age of digitalization, you will have to be visible. Brand building and audience buying begins even before the actual launch of the product. Send out teasers in the social media, engage your potential customers and get them interested in your product.
• To allure the venture capital funding, your product has to be foolproof in this age of competition. The VC firms have to see a potential market for the product to be convinced to invest in it. Be prepared to be grilled.
• Midway, if gets bumpy and you realize start-up life is just not your cuppa, you should return to your plan B, in case you give up. But if you have a heart made of steel, there’s no stopping you….
• Then slog even harder. Learn from mistakes. Curate and analyze:

1. consumer behavior both offline and online
2. abandoned baskets
3. competitive analysis
4. social media analysis
5. shopping history

Success is hard work and in order to acquire new customers and retain the existing, there’s no short cut but to get the above steps right, if it does not work at once – reiterate. When success does come by, in the form of reaching targeted number of customers or reaching the targeted revenue, all the hard work does pay off. The incredible feeling of having achieved something, which is your own, based on your accomplishments, is almost hedonistic. Handmade success!