Architecting Modern Data Platforms

As organisations struggle to capture and leverage multitudes of data, there is a surge of technological options to choose from. Well designed data platforms facilitate experimentation, have shorter time to markets, have faster adaptation to latest advancements in data technologies, promote self-service thereby accelerating data adoption.  Data being the key enabler for business transformations, it is vital to build platforms that accelerate validation of use cases and can handle scaling of use cases and users. Designing a platform which is elastic enough to embody all the above can be quite a daunting task.

The primary points to consider when architecting modern data platforms:

• Customer centric

Organisations battle immensely with legacy data technologies to deliver personalization, and customer experience, despite there being so much emphasis on hyper personalization. Thinking on the lines of creating 360 ° customer view helps align technological choices after business pain points.

• Cloud Native

Cloud solutions support elastic scaling, high availability  and secure fully managed services with integration to a range of enterprise security systems including LDAP, Active Directory, Kerberos and SAML. Cloud  solutions allow pluggable architecture – replacing components if better options are available with minimum reconstructing. Cloud platforms eliminate the time-consuming work of provisioning resources and infrastructure, thereby reducing time to market.

• Multi-platform architectures

Be it multi-cloud or multiple data storage patters, it should be the use cases that dictate the architectural patterns and not vice versa. Datawarehouses, datalakes and NoSQL databases can all co-exist on multi-cloud platforms if the use cases demand so. Organisations should avoid platform/vendor lock-ins, because then businesses are forced to make technology choices that are not in the best interests of the company.

• Microservice-enabled

It is critical to  envision data as not just a means for visualization like a diagnostic tool, data is critical to help organizations adapt to change, in evolving business environments and to innovate and every company wants to expedite the process to be the first ones to come up with innovative products and services. Data plays a key role in this aspect. Monolithic applications are a major bottleneck in this case. In microservices based design small decoupled services are developed completely independent of each other  to achieve business requirements, faster, generally through REST APIs or event streams.

• Flexible

Modern data platforms should be flexible enough to accomodate rapidly evolving business requirements. Be it integrating new data sources or feeding data into futurist data products. Modern data platforms should simplify testing new ideas on a small scale prior to making heavy investments in infrastructure.

Modernization continues to be a strong trend in data platforms, whether on Hadoop or RDBMS or multi-tenant solutions. It is the ease of integrating new data sources, TCO, prototyping functionalities, security and scaling that matter most in modern platform architectures.

 

Advertisement

Three reasons why Big Data projects fail

I have not been regular with my personal blog because I have been blogging elsewhere.

Here are the links to my latest blog posts about why Big Data projects fail and how to attract more women into tech.

Having worked extensively in the Big data & IoT space I have closely observed failures over and over again and the reasons for failure being repetitive :

• Wrong use cases
• Wrongly staffed projects
• Obsolete technology

Read the blog post for more details:

Three reasons why Big Data projects so often fail

Being a woman in tech or woman in data I am often the only woman in meetings, trainings and discussions which feels weird. With not many women in tech it gets easier to discriminate the few that do exist. Incidents of mansplaining, gaslighting are rampant and it’s the victim that gets labelled as drama queen while the abusers fo scot free. Organisations that are serious about increasing the number of women in tech need to address glass ceiling, gender wage gaps & bro-culture and cultivate an inclusive work atmosphere. Read my post on how to get more women into tech.

How to Get More Women in Tech

How to become big data – data analyst

Anyone who works in the tech industry is aware of the rising demand of Analytics/ Machine learning professionals. More and more organisations have been jumping on to the data driven decision making bandwagon, thereby accumulating loads of data pertaining to their business. In order to make sense of all the data gathered, organisations will require Big Data Analysts to decipher the data.

  Data Analysts have traditionally worked with pre formatted data, that was served by the IT departments, to perform analysis. But with the need for real time or near-real time Analytics to serve end customers better and faster, analysis needs to be performed faster, thereby making the dependency on IT departments a bottleneck. Analysts are required to understand data streams that ingest millions of records into databases or file systems, Lambda architecture and batch processing of data to understand the influx of data.

Also analysing larger amounts of data requires skills that range from understanding the business complexities, the market and the competitors to a wide range of technical skills in data extraction, data cleaning and transformation, data modelling and statistical methods.

Analytics being a relatively new field, is struggling to resource the market demands with highly skilled Big Data Analysts. Being a Big Data Analyst requires a thorough understanding of data architecture and the data flow from source systems into the big data platform. One can always stick to a specific industry domain and specialize within that, for example Healthcare Analytics, Marketing Analytics, Financial Analytics, Operations Analytics, People Analytics, Gaming Analytics etc. But mastering the end-to-end data chain management can lead to plenty of opportunities, irrespective of industry domain.

The entire Data and Analytics suite includes the following gamut of stages:

• Data integrations – connecting disparate data sources
• Data security and governance – ensuring data integrity and access rights
• Master data management – ensuring consistency and uniformity of data
• Data Extraction, Transformation and Loading – making raw data business user friendly
• Hadoop and HDFS – big data storage mechanisms
• SQL/ Hive / Pig – data query languages
• R/ Python –  for data analysis and mining programming languages
• Data science algorithms like Naive Bayes, K-means, AdaBoost etc. – Machine learning algorithms for clustering, classification
• Data Architecture – solutionizing all the above in an optimized way to deliver business insights

The new age data analysts or a versatile Big Data Analyst is one who understands the complexity of data integrations using APIs or connectors or ETL (Extraction, Transformation and Loading), designs data flow from disparate systems keeping in mind data security and quality issues, can code in SQL or Hive and R or Python and is well acquainted with the machine learning algorithms and has a knack at understanding business complexities.

Since Big Data and Analytics is constantly evolving, it is imperative for anyone aiming at a career within the same, to be well versed with the latest tech stack and architectural breakthroughs. Some ways of doing so:

• Following knowledgeable industry leaders or big data thought leaders on Twitter
• Joining Big Data related groups on LinkedIn
• Following Big Data influencers on LinkedIn
• Attending events, conferences and seminars on Big Data
• Connecting with peers within the Big Data industry
• Last but not the least (probably the most important) enrolling in MOOC (Massive Open Online Course) and/ or Big Data books

Since Analytics is a vast field, encompassing several operations, one could choose to specialise in parts of the Analytics chain like data engineers – specializing in highly scalable data management systems or data scientists specializing in machine learning algorithms or data architects – specializing in the overall data integrations, data flow and storage mechanisms. But in order to excel and future proof a career in the world of Big Data, one needs to master more than one area. A data analyst who is acquainted with all the steps involved in data analysis from data extraction to insights is an asset to any organization and will be much sought after!

Managing corporate innovation

The time is ripe for corporates to embark on a journey of innovation. Having said that it is a rocky road for enterprises that have been in existence for half a century or more, having made hay when the sun shined and thus gathering a good amount of legacy systems and processes on the way. Some organisations that have have chosen to invest in innovation prefer to run the innovation labs separately, far from the core business, the reason being that innovation should not get bogged down by corporate beauracracy. A big part of innovation is experimentation — experimentation with thoughts and ideas and prototyping. Business leaders have a herculean task of ensuring innovation for the businesses of the future while continuously improving the incumbent corporate machinery that generates the revenue necessary for future investments.

Innovation is about inventing new products or services that solve customer needs and can be monetised. Innovation involves trial and error and learning from the same, a structured method of experimentation leads to better tracking of ROI. The pentathlon framework of innovation articulates the methodology from ideation to market launch. The influx of ideas, which form the innovation backlog, can be either disruptive ideas or new ways of resolving existing business challenges.

Each part of the innovation funnel has to imbibe a fail fastand and an iterative approach for further ideation with feedback loops. Every new invention undergoes inception, improvements and adoption followed by stability and subsequent depreciation. If there are many trains of thought in the innovation projects pipeline, there has to be an order in the way of prioritizing the projects. Prioritization of innovation projects should depend on

• Marketability — is there a potential market for the product/service?
• Feasibility — is it possible to deliver the project with the resources the company can afford?
• ROI — How soon is the breakeven point?
• Time to market — how soon can the MVP be launched?

The implementation of innovation projects are not very different from other corporate projects, involving a portfolio selection based on the prioritization and business urgency. The next stage involves scoping and development, preceeded by prototyping before scaling. Post validation and launch, the impact has to be measured and analyzed to understand the product adoption and customer experience. The insights from measuring innovation efforts lead to newer ideas or incremental improvements to exisiting business processes and/or the innovative product under consideration. There has to be a relentless flow of insights into the innovation funnel to finetune the ideas being considered for prioritization, implementation and launch.

The very existence of an innovation foundry within a coporate house has to be justified to the investors and shareholders. As a means to create a process of accountability there should be KPIs defined, some examples being:

• Number of ideas considered for prioritization
• Number of ideas that were productionalized
• Number ideas that have lead to business process improvement
• Value added (Value = Accrued benefits — costs)
• Time to breakeven

The major factors that influence an innovative culture at a corporate level depend on the company culture – the ability to thrive during change and adaptability to new market conditions. It is of utmost importance to recognise and reward the people who contribute to innovation and showcase and communicate the succesful results. Innovation labs within corporates cannot be run separately forever, the outputs from the innovation exercises need to flow back into the day to day business and the challenges from the core business need to be worked upon in the innovation labs.

With innovation being a crucial area of focus for most organisations, there are a lot of ideas floating regarding innovation management. It can be rather chaotic with buzzwords like Big data, AI, Robotics, Augmented Reality etc. being thrown around callously. Technology is a great enabler for business strategy, but great ideas arise from understanding customer requirements and the lack of products or services that fulfill the same.The most significant point of focus in any corporate innovation should be customer centricity and business gain.

Chasing dopamine — The Neo-generalist

Dopamine a chemical released by nerve cells, which plays a major role in reward-motivated behavior. For some of us the adrenaline rush comes seeking and mastering new challenges and then moving onto the next.

Chasing dopamine

This habit of seeking new problems to solve goes beyond job titles, roles and responsibilities, educational background and age. These polymaths, knowledge seekers or autodidacts bloom where planted. They thrive when things need to be fixed, they use the knowledge gained from one industry on others, experience with one method leading to another. Monotonous tasks seem arduous.

“It still holds true that man is most uniquely human when he turns obstacles into opportunities.”–Eric Hoffer

I find it hard to grasp that organisations hire specialists to break down silos to facilitate continuous flow of information between different business units but individuals on the other hand are encouraged to be specialists within a certain discipline. Unless there are people who can handle multidisciplinary roles transcending departmental borders, I do not see a solution to organisational silos. The divide between tech and business is one such area of concern. People with tech skills are assumed to have little grasp of business acumen while people with strong business understanding are assumed to have limited IT proficiency and people with both business and tech expertise are perceived as average in both. That’s due to our obsession with specialism. Generalists are looked down upon, likened to jack of several trades and master of none. On the contrary, the monkey minds are an asset, being able to connect several dots and improvise solutions based on their creative thinking, envisioning paths beyond their job titles.

My constant dilemma

Hailing from India, Jugaad was part of the everyday vocabulary. Jugaad in Hindi means makeshift solutions which requires resourcefulness. Resourcefulness is not part of any syllabus, it comes naturally when there’s a scarcity of means. Being able to do more with less. Doing more has to do with understanding several disciplines to put together a solution beyond frameworks and recognized theories.

I came across a book The Neo-Generalist by Kenneth Mikkelsen and Richard Martin which was sort of narrating my mental state — the nomad state, in search of the next problem irrespective of domain, technological challenges or borders. We, the neo-generalists are happy as long as our brains are being harnessed and we are involved in something meaningful. If you identify yourself as neo-generalist then it is a must read. You’re not alone, there’s a whole tribe of us, restless souls, trying to juggle several disciplines at the same time and loving every moment of it!

Settle down — this word does not appeal to neo-generalists. Constant learning and treading on paths not previously traveled are our only focus.

“Listen baby, ain’t no mountain high,
Ain’t no valley low, ain’t no river wide enough”

Analytics – Implications on Digitization

Digital is all about data, contrary to the prevalent method of creating Analytics as a silo all by itself. Analytics should be seen as one of the fundamental underlying processes that support the core business processes like product development, marketing, sales, customer relationship, finance and innovation. Data and Analytics provide value to core processes, for continuous improvement.

Most organisations are keen on innovation. Innovation could entail new market opportunities and could be an entirely new value proposition, discovered on a strategy canvas. But innovation could also be a by-product of a business process improvement. Such opportunities can only arise when business processes are tracked, measured and analyzed. Organisations that indulge in hypothesis driven product development or mass marketing could benefit by introducing  a data driven approach to the above processes, thereby uncovering the customer needs and product usage. Businesses may launch products with a certain outcome in mind, but sales, social media feedback and web analytics data may have another story to tell. It is in this story, that new opportunities can be unearthed. Understanding customer behavior is a way of discovering new marketing and/or product/service development opportunities.

Many organisations investing heavily in digitization, charting customer journeys, aimed at improving customer experience across all touch points, seemingly forget to make Analytics an integral part of this process.  The key to understanding  major business drivers like customer retention, ROMI, growth, customer engagement, monetization, finding new customer segments depend on deciphering the business data generated.

Analytics, therefore should be embedded in all business processes to capture the way the end customers perceive products or services or marketing and branding efforts made by any organisation. Analyzing the business data from existing processes could possibly give rise to future business prospects. To tread on a path of continuous improvement and innovation, companies will have to make Analytics a fundamental part of every business strategy.

Four steps to becoming a Data-Driven organisation

Not a day goes by when our LinkedIn news feed is not flooded with the mentions of AI and Machine Learning benefitting and changing the ways of mankind, like never before. This hype surrounding AI, Machine learning has resulted in most organisations jumping on the bandwagon without proper evaluation. A couple of years ago, the term Big Data enjoyed a similar hyped status but it has been losing it’s lustre to all the talk about AI and Machine Learning, lately.

The truth, however, is that, AI and Big data need to coexist and converge. Merely collecting and storing data in huge amounts will prove futile, unless AI and Analytics are used to generate meaningful insights that help businesses, enhance customer experience or increase revenue influx.

Making an organisation Data-Driven will take time and will happen in stages. While there are no sure shot ways to create a Data-Driven organisation, below are some ways that could lead to a change:

1. Strategy – It all starts with a clearly defined strategy in place, stating the Whys, Hows, Whos and Whens. A clear strategy helps in raising awareness across the organisation, about the topic in focus (data in this case) and creates a sense of urgency around the change process. It is imperative that the entire organisation understands the importance and implications of a data-driven organisation, thus encouraging people to update their skill sets and raise their level of data awareness. An all round data strategy should not only include the technology required for execution but the kind of competence and people skills and the sort of conducive atmosphere required for a data-driven organisation to thrive.
2. People – Just as there are different kinds of skills required within a Marketing or a Software organisation, there are different skill sets for the different job roles within a data organisation. But due to the hype surrounding Machine Learning and AI while companies lack the practical knowledge in data know-how, the tendency is to either hire the wrong people or assign the wrong tasks to the right people! Not everyone has to be a data scientist in the data organisation. There will be people required to work on data architecture, data infrastructure, data engineering, data science and the Business Analysts. These could very well be the same person, if the organisation is lucky enough. But it is unfair to hire a data engineer and assign him/her the task of building Predictive models or hiring a data Scientist to be told to develop BI reports. Strategists will have to spend the time required to understand the nuances of skills and expertise required in a data organisation but it will be worth it, to retain and grown the talent pool required for a Data-driven organisation.
3.  Patience – Creating a Data-driven organisation will require ample amounts of patience and perseverance. If data has not been involved in the decision making process, earlier,  then the data is most probably not in a state that can be used readily or maybe there is no or not enough data to begin with! In that case, it has to start with gathering the data required to achieve the business goals. Transaction systems have a very different database design than the data storage mechanisms used for Analytics purposes, which entails a design and architecting process before being able to analyse the data. Moreover, as Analysts dig into the transaction data, they surely will encounter non-existence of relevant data, data retrieval issues and unearth data quality issues and data integration problems due to the existence of data silos. In a data-driven organisation, all data sources are integrated to provide a single enterprise version of truth, irrespective of Customer data or Sales or Marketing data. A data platform, integrating all business data sources, ensuring quality and data integrity and security is a time-consuming process. Organisations will have to take this lead time into consideration when strategizing a Data-driven decision making approach.
4. Organisational Culture – The purpose of a Data-driven organisation is to empower employees by means of data and information sharing to enable the organisation to collectively achieve the business goals. This approach requires employees to be data aware and not use gut feelings to make decisions and this could be a whole new approach for many. This new way of working requires organisational change management, educating people to use facts and figures to arrive at conclusions and make decisions. If an organisation is fairly data aware, in the sense that metrics are used to measure certain processes, in order to turn Data-driven , the organisation has to take steps to use data proactively (read Predictive Analytics) and not just summarise events that happened. The CDOs/ CMOs need to drive data awareness by showcasing quick wins and success cases of Data-driven approaches, as a means to use data as the foundation in every decision making process.

Some organisations may take longer to implement a Data-driven culture than others but there is no way an organisation can become Data-driven, just like that, one fine day! If the CDOs can gauge that the organisation has a longer incubation period then it is good to start with raising data awareness and introducing a BI/ Datawarehousing team. It is not recommended to directly leap on to AI, hiring data scientists, to be then left in a lurch if the organisation and the infrastructure are pretty rudimentary to handle their expertise.

A Data-driven organisation culture starts with the right strategy in place, followed by the right people and technology, evaluating and optimising the entire process, intermittently.

Growth Hacker’s Marketing

Marketing is being disrupted and no more run by only traditional non-technical marketeers. Marketing is supported by a wide range of – call it reporting, dashboarding, marketing analytics, marketing automation processes. Moreover, the startup scene is very exciting and a hot bed for innovation. Most startups spring into action sans a huge funding. The startups will have to grow exponentially, boasting a substantial customer base to be able to entice investors. Enter the growth hacker – with a single minded goal, growth!

Typically, the UX team designs the UX strategy, the product team develops the product, the coder codes in order to deliver the product and the marketeer tries selling the product. But with the new age disruptive marketing, the UX team, product team, code development team and the marketing team will have to work very closely, trying and testing every trick in the book to elevate growth. A growth hacker is a bit of all the above.

A growth hacker is more of a full stack employee armed with Swiss knife like multiple skill sets, analytical abilities being top rated. Growth hacking is primarily a focus within the startups, the budget being a constraint, lesser number of employees expected to contribute more. But with time, enterprise companies will adapt to growth hacking means of increasing revenue generation. Growth hacking is based on data, analyzing data to improve the business processes, to sell more, to convert more, to gain new customers and retain existing customers. Growth hacking does not entail data reporting only for the purpose of data visualization, it uses data to derive at hypotheses and reasoning to better understand and improve internet marketing.

So what’s growth hacking all about? Growth hacking is about

• Improving user experince by A/B testing to reduce bounce rate
• Content Marketing
• Designing, implementing and analyzing sales funnel to reduce drop rates
• Search Engine Optimization
• Channelizing all it takes to increase conversion rate
• Using analytics to track click stream data about consumer’s online behavior
• Analyzing past online or shopping behavior to be able to predict consumer’s probable behavior at the next visit
• Social Media marketing – paramount for startups on shoe string budgets. Using Facebook, Twitter APIs to analyze the demographics of consumers sharing and liking the products, consumer opinion in social media and competitor analysis
• Being able to analyze consumers that are likely to churn and the reasons behind, which can be addressed. Analyzing the response data from campaigns targeted at reducing churn, to measure campaign effectiveness.
• Improving omnichannel advertising and using analytics to analyze data to conclude the channel that yields most and finding potential market opportunities

From the above list, growth, data and analytics are evidently the point of convergence for growth hacking. Growth hackers have to be inherently curious, tenacious, analytical and above all innovative. Growth hacking is an an art, not just number crunching or coding. It is the ability to see beyond code, to be able to analyze the implications of new features or every change in any part of the business processes that drive growth.

As Sean Ellis says, a “growth hacker’s true compass is north.”