Combining Multiple Analysis Methods

Ever been in a situation where the numbers unravel one narrative while the words unravel another? Think of it more like putting together a jigsaw puzzle; you use a variety of approaches—surveys, interviews, statistics, observational data—that each offer partial information. By adding together the various approaches to data analysis, we gain a more complete, richer understanding.

In just a moment, you will learn about how to leverage multiple method approaches that reveal hidden insights, develop greater confidence in findings, and change the way you make sense of complex data. Focusing on the requirements for practical and actionable outcomes, this chapter will provide you with a step-by-step guide on how and when to combine analysis methods.

Why Combine Analysis Methods?

Combining analysis methods is more than just a matter of convenience or variety—it is a matter of efficacy, or synergy. One data analysis method may identify patterns, while others explain why those patterns emerged.

Together, processes confer each other's existence and amplify the meaning of the outcome. When we rely solely upon one method, we risk ignoring nuance, or ascribing meaning to the outcome that perhaps doesn't exist, or worse, misrepresenting the patterns being communicated. This is particularly problematic in business analysis practices, health care analysis, UX research, and policy analysis.

Common Frameworks & Approaches

There are several distinctive and structured ways in which to support the combining of analyses (as a generalization):

• Convergent Parallel Design: This is when you collect qualitative and quantitative data at the same time, and an analytical description, or later, quantitative and qualitative data and codes to explain the findings of your original records.

• Sequential Exploratory or Explanatory Designs: This is when you utilize the primary analysis method first (i.e. it might be interviews you want to work with), and you use the findings from those data to implement the second analysis method (i.e. it might be surveys).

• Embedded: This is when you use one data type as part of a larger data type. For example, in a numerical survey, collect qualitative open-text responses.

Each of these frameworks can help maintain clarity and logical progression of the analytical plan when utilizing disparate approaches. Mixed Methods: Combining Qualitative and Quantitative. We have a method that combines various measures (metrics, ratings, counts) with descriptive data (words, stories, observations).

The rationale is that numbers provide scale and general trends, and narratives provide contexts and meanings. You ended up with both the map and the travelogue, which is nice to have if you want the whole picture.

Meta-analysis and Data synthesis

In medical or social science research, many studies combine their results to achieve sufficient statistical power. This is often called meta-analysis. When you combine the data (effect sizes) from many experiments, you are creating an estimate that is much more meaningful than any study by itself. This gives you more confidence in your estimates and gives you an indication of patterns that may not have been clear.

Multimethodology, which is more than "mixed methods"

Mixed methods, in general, is imposed on mixing qualitative data/quantitative data. Multimethodology is broader: it distinguishes between various data sources, incorporates multiple perspectives on data analysis, and can even adopt separate research paradigms. Thinking pluralistically allows one to examine an issue through many data sources and methods, including surveys, experiments, case studies, models, and triangulation.

Statistical methods for integrating data

There is a lot that you can do statistically that integrates or combines datasets, or outcomes, especially when trying to integrate quantitative datasets; in particular:

• To combine multiple data sources as a general approach, think about some way of model-based integration or weighting of each data source (however you want to implement that) in a way that they can all filter into one analysis.

• As to so-called meta-analysis techniques. Take the effect sizes in the studies and combine them for a consensus.

• Combining p-values/statistical outputs: When raw data aren't available, there are options for aggregating evidence from separate tests.

These approaches involve some effort but can lead to quite powerful aggregation when done correctly.

Practical suggestions for preparing a hybrid analysis

If you're going to switch between qualitative and quantitative strategically:

• Identify the research purpose first, and allow it to dictate the approach, not the other way around.

• Identify your integration points in advance, and be clear about when and how the data will be integrated.

• Pay attention to method compatibility—use similar parallel structures (e.g., make survey and interview questions parallel).

• Secure sufficient resources—combined designs typically take longer and involve more resources than either approach alone.

• Be prepared to deal with different or conflicting results—be clear how you're going to make sense of, or resolve, the differences.

A case study vignette

Think about a product team that wants to find out why user engagement has dropped. They begin by examining analytics (low usage), followed by interviews to understand the reasons behind this, which ultimately leads to frustration with the poor onboarding process. Now that they know the issue, they redesign the workflow, survey users about their onboarding experience, and track usage and engagement over time. By the end of this sequence - quantitatively → qualitatively → - we have background, a direction for redesign, and evidence of impact.

Considerations for using combined methods

Even though combining methods has many advantages, there are challenges to note:

•     Cumbersome logistics: Juggling methods and methodologies can be challenging.

•     Skill gaps: It assumes fluency in multiple methods, and you can't be an expert in all methods—two rarely would be.

•     Integration challenges: Integrating results needs to be managed carefully, or you'll end up with mismatched or misinterpreted results.

•     Ambiguity: When the methods treat your results differently, it'll take care and thinking to synthesize what they mean about your original purpose, or how they fit together.

Regardless, if you are aware of the issues involved, and plan accordingly, and are reflexive and transparent, you can avoid the biggest pitfalls.

Summary: The Benefit of Integration

Using more than one way to analyze data is like taking photos using different lenses—even though each lens sees some things better than others. When combined, they provide sharper focus, improved depth, and clarity. Whether mixed methods, meta-analyses, or multimethodology, integrated modes of analysis provide more salient information, explain more deeply, and are more purposeful. What I have spectacularly failed to convey in the previous points, but want to emphasize with all my being, is to let your research purpose drive your methodology, plan integration points, and capture the richness that arises by integrating perspectives from your research.

Final remarks

If planned with intention and purpose, a combined analysis approach will transform your research findings from banal insights into empirical decision intelligence. It is not about making something complicated for complexity's sake. It is about realizing the whole picture, complete with the texture and context. In an era of abundant data, integrated approaches will not only be practical—they'll also be required to ensure meaningful and credible decision-making internally and externally.