Data Analysis Techniques in R

New Video: Applications and Limitations of Inferential Statistics Inferential statistics is a cornerstone of data analysis, helping us draw conclusions and make predictions from sample data. But how do you use it effectively while avoiding common pitfalls like sampling bias or misinterpreting p-values? In my latest video, I explore real-world applications of inferential statistics — like A/B testing, market research, and customer behavior analysis — while discussing the challenges and limitations every analyst should know. What You’ll Learn: • Practical use cases for inferential statistics in decision-making • Common mistakes and how to avoid them • How to design better studies and turn insights into action You will find the video here: https://bit.ly/3DxTGfr Art+Science Analytics Institute | University of Notre Dame | University of Notre Dame - Mendoza College of Business | University of Illinois Urbana-Champaign | University of Chicago | D'Amore-McKim School of Business at Northeastern University | ELVTR | Grow with Google - Data Analytics #Analytics #DataStorytelling

7 common mistakes to avoid in data analysis Data analysis plays a key role in making informed decisions, but it’s easy to make common mistakes. Here are 7 pitfalls to avoid to ensure your analysis is both accurate and insightful. 1. 𝐍𝐨𝐭 𝐃𝐞𝐟𝐢𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 𝐏𝐫𝐨𝐛𝐥𝐞𝐦 Jumping into data analysis without a clear problem can lead to irrelevant findings. A specific question helps focus your analysis and ensures you gather the right information. Without this clarity, you risk wasting time and resources on unnecessary data exploration. 2. 𝐈𝐠𝐧𝐨𝐫𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 Overlooking data quality can result in inaccurate conclusions. Checking for errors, duplicates, and inconsistencies is crucial for trustworthy insights. Poor data quality compromises the integrity of your analysis and can mislead decision-making. 3. 𝐎𝐯𝐞𝐫𝐥𝐨𝐨𝐤𝐢𝐧𝐠 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 Failing to understand the context of the data can lead to misinterpretations. Knowing the background and factors influencing the data is essential for accurate analysis. Context provides the necessary framework to draw meaningful conclusions from your data. 4. 𝐔𝐬𝐢𝐧𝐠 𝐈𝐧𝐚𝐩𝐩𝐫𝐨𝐩𝐫𝐢𝐚𝐭𝐞 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐚𝐥 𝐌𝐞𝐭𝐡𝐨𝐝𝐬 Applying the wrong methods can distort your findings. Choose analytical techniques that align with your data type and the questions you want to answer. Using suitable methods helps ensure that your conclusions are valid and actionable. 5. 𝐂𝐡𝐨𝐨𝐬𝐢𝐧𝐠 𝐭𝐡𝐞 𝐖𝐫𝐨𝐧𝐠 𝐆𝐫𝐚𝐩𝐡 Using inappropriate graphs can confuse your audience and misrepresent your data. Select visualizations that clearly communicate your findings. The right graph not only presents data effectively but also enhances audience understanding. 6. 𝐅𝐚𝐢𝐥𝐢𝐧𝐠 𝐭𝐨 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐞 𝐃𝐚𝐭𝐚 Neglecting to visualize data makes it harder to identify trends and insights. Visual representations like charts and graphs can clarify complex information. Effective visualization helps you and your audience grasp the story behind the data quickly. 7. 𝐍𝐞𝐠𝐥𝐞𝐜𝐭𝐢𝐧𝐠 𝐭𝐨 𝐃𝐨��𝐮𝐦𝐞𝐧𝐭 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐞𝐬 Forgetting to document your analysis can lead to confusion later on. Clear documentation allows others to understand and replicate your work effectively. Proper notes create a transparent record of your analysis, fostering collaboration and trust. By avoiding these pitfalls, you can make sure your analysis is more reliable and helps drive better decision-making. If you're planning to start your data analytics journey, here are a few courses/boot camps I'd recommend: 1. Alex' Courses on Analyst Builder - https://lnkd.in/eJaEC6qe 2. Dhaval and Hemanand' Data Analytics Bootcamp 4.0 on Codebasics - https://lnkd.in/ew3UN2JF 3. DataCamp Courses - https://lnkd.in/ejJYGp4D What else would you add? ♻️ Save it for later or share it with someone who might find it helpful! #dataanalytics #commonmistakes

How to understand that your statistical analysis might be garbage Sometimes significant results in omics don’t reflect underlying biology — they come from noise, suboptimal design, or misinterpretation. These common warning signs help you catch issues early. ✅ Sign 1: you have no biological replicates — only technical ones Technical replicates measure instrument variability. Biological replicates capture true system-level variability. Without biological replication, statistical significance becomes unreliable. ✅ Sign 2: your PCA clustering contradicts your experimental design If samples cluster by batch, processing date, or operator instead of biological groups, you’re primarily seeing batch effects rather than biological differences. ✅ Sign 3: you forgot about multiple testing Omics involves thousands of comparisons. Without FDR correction, the number of significant proteins is heavily inflated. ✅ Sign 4: you adjusted filters after seeing the results This falls under data snooping or p-hacking. Changing fold-change cutoffs, filtering rules, or removing outliers after inspecting the volcano plot can introduce bias. ✅ Sign 5: your results do not pass robustness checks If significance disappears: - under different filters, - after removing outliers, - after applying multiple testing correction, then the conclusions may not be stable. ✅ Sign 6: you have no independent biological validation If candidates are not supported by orthogonal methods, they remain hypotheses rather than confirmed findings. Statistics cannot compensate for poor design, missing replicates, or unaddressed batch effects. Building a solid analysis strategy before the experiment — and not relying on a bioinformatician to “fix it afterwards” — leads to far more reliable and interpretable omics results. #omics #proteomics #transcriptomics #bioinformatics #datascience #dataanalysis #analysis #massspectrometry #FDR #PCA #reproducibility #statistics #research #biology #researchdesign #phd

If you want to level up your research rigor and reduce common statistical pitfalls, this is one of the most practical papers you can read this year. “Common Mistakes in Biostatistics” https://lnkd.in/d5hHVZSP This review highlights issues such as: ✔ Using the wrong metrics to describe data ✔ Misinterpreting P-values and confidence intervals ✔ Ignoring sample size planning ✔ Confusing correlation with causation ✔ Mishandling confounders and mediators ✔ Coding errors and bias from study design assumptions PMC If you’re writing a thesis, preparing a journal submission, or doing applied health research, understanding these pitfalls can dramatically improve your results’ credibility and reproducibility. Highly recommended for students, clinicians, data analysts, and early career researchers. #ResearchMethods #Biostatistics #DataScience #Epidemiology #PublicHealth #Statistics #RStats #ClinicalResearch #ScientificWriting #ReproducibleResearch

A common data scientist's dilemma - It's 2am, you’re knee-deep in a dataset, caffeine-deprived, and suddenly you come across a data point so absurd, so statistically unhinged, that it defies all laws of physics, logic and expectations. Outliers are the drama queens of datasets. They scream for attention, derail models, and keep you up at night. The twist is, some outliers are typos (one who entered “$999,999” instead of “$99.99”) while others can be the next penicillin or the next Higgs boson moment. The problem? Both look identical at first glance. For example imagine the first time Netflix discovering an outlier in user behavior where a customer is binge-watching entire seasons in a day wasn't an error—they revealed a new normal for streaming. On the other hand a healthtech startup once celebrated a groundbreaking discovery: a patient with a heart rate of 0 bpm for 48 hours, later realising the sensor fell off! As data practitioners, our job isn’t to delete outliers—it’s to investigate with curiosity and humility. Some things to keep in mind - context is the king (If a 12-year-old spent $2M on a e-commerce site, it’s probably fraud), reproducibility check can help (can one reproduce the outlier?), think 'what if' (what if the outlier is actually real), do outlier analysis (instead of brushing under the carpet). Go find an outlier today. If it’s a typo, laugh it off. If it’s a Nobel Prize discovery, cut me in for 10%. Deal? #TheInsightEdge #DataScience  

The pitfalls of class prediction in omics 🧵 1/ You think you’ve built the perfect omics predictor. The accuracy is high. The p-value is low. But is it real—or just a story your data whispered back? 2/ High-dimensional data is a double-edged sword. Thousands of genes, hundreds of samples. With enough features, even random noise can look predictive. That’s the curse of dimensionality. 3/ Statistically, you can always draw a hyperplane to separate two classes. Even if the labels are random. That’s overfitting. And omics is a playground for it. 4/ So we add regularization: LASSO, Ridge, Elastic Net. They penalize complexity, reward simplicity. But it’s not enough. Because the real danger is how we validate. 5/ Cross-validation (CV) is standard. But do you select your features before the CV folds? That’s data leakage. And it gives inflated performance. Always. Every time. 6/ Nested CV is your friend: Inner loop: tune hyperparameters Outer loop: estimate error It’s slower. But it’s honest. 7/ Still confident? Let’s talk confounders. Batch effects. Age. Ethnicity. Study site. If they’re correlated with outcome, they fake predictive power. 8/ Confounding doesn’t go away with random splits. It hides in the noise. And only shows itself when your model fails in an external dataset. Validate on independent cohorts. 9/ Want to compare your model to an existing one? Do it on a neutral dataset. Using your training set to favor your model is bias by design. 10/ So you beat the baseline by 2%. Is it statistically significant? Not unless you test it across multiple datasets. Ideally 5–6. Meta-analysis helps. 11/ Unsupervised pitfalls: If you cluster samples using features chosen with the labels in mind, You’ll rediscover your labels—not biology. Clustering must be unsupervised in every way. 12/ Many retractions in omics come from these mistakes: Data leakage Confounding Overfitting Unvalidated results Because story-telling is easier than science. 13/ To get it right, you need more than code. You need humility. Statistical discipline. Curated metadata. And rock-solid validation. 14/ Key takeaways: Overfitting loves high dimensions Never pre-select features across folds Use nested CV Validate externally Watch for confounders Simplicity > complexity 15/ Omics is powerful. But power needs control. Guard your models from yourself. The truth is out there—but only if you earn it. I hope you've found this post helpful. Follow me for more. Subscribe to my FREE newsletter chatomics to learn bioinformatics https://lnkd.in/erw83Svn

*** Blind Spots in Statistics *** Blind spots in statistics are a rich topic, and it’s more interesting than just “common mistakes.” These are the places where even smart analysts, researchers, and data‑driven people routinely mislead themselves without realizing it. Below is a map of the major blind spots. Major Blind Spots in Statistics 1. Confusing Correlation, Causation, and Mechanism • People often treat a statistical association as if it reveals the underlying mechanism. • Even when analysts know correlation ≠ causation, they still slip into causal language. • Blind spot: forgetting that most datasets are observational, not experimental. Example: A model predicts that people who buy diapers also buy beer. The blind spot is assuming a psychological cause rather than recognizing the structural mechanism (parents running errands). 2. Overtrusting Models Without Checking Assumptions Many statistical tools rely on assumptions that are rarely verified: • Normality • Independence • Linearity • Homoscedasticity • Random sampling Blind spot: analysts often treat these assumptions as “default truths” rather than hypotheses that need checking. 3. Survivorship Bias We see the winners, not the failures. • Companies that succeed look like they followed a formula. • Athletes who “made it” seem to validate a training method. 4. The Base Rate Fallacy People ignore the underlying prevalence of an event. • A test with 95% accuracy doesn’t mean a 95% chance the result is true. • Rare events produce many false positives. 5. Misinterpreting p‑values • p < 0.05 does not mean “there’s only a 5% chance the result is due to chance.” • p-values don’t measure effect size or importance. • They’re sensitive to sample size. 6. Overfitting Disguised as Insight Models can memorize noise and present it as structure. • Especially common in machine learning. • Humans then interpret the noise as a meaningful pattern. 7. Ignoring Measurement Error Every variable is a shadow of the real thing. • Self-reported data • Sensor drift • Survey wording • Proxy variables 8. Simpson’s Paradox A trend appears in subgroups but reverses when groups are combined. • Happens when a lurking variable shifts group sizes. Blind spot: assuming aggregated data tells the same story as disaggregated data. 9. The Multiple Comparisons Problem If you test enough hypotheses, some will appear significant by accident. • A dataset with 1,000 variables can produce dozens of “significant” results by chance. Blind spot: forgetting that searching for patterns creates patterns. 10. Human Pattern‑Seeking Even with perfect math, humans: • Overinterpret randomness • See trends in noise • Prefer simple stories • Anchor on first impressions --- B. Noted

📊 The Hidden Bias in #Sampling — Why Data Scientists Should Care We often talk about model accuracy, but what if the real issue isn’t the model… it’s the data we trained it on? One of the most overlooked pitfalls in #datascience is sampling bias — when the data we collect doesn’t truly represent the population we want to predict. Think about it: - Your #churnmodel only uses data from active users — you’ve already lost insight into those who left. - A marketing A/B test is run only on high-intent users — your lift estimate is inflated. - Credit risk data excludes declined applicants — your model is biased toward “safe” customers. Each of these cases leads to a false sense of confidence. The metrics look great… until the model hits the real world. Here’s how advanced analytics teams mitigate these issues: 🔹 Stratified or importance sampling — ensure proportional representation across key segments. 🔹 Reweighting or post-stratification — adjust sample weights to match the population. 🔹 Inverse propensity scoring — correct for selection bias when randomization isn’t possible. 🔹 Out-of-sample validation — test performance on untouched, representative data. The takeaway? 👉 Even the most sophisticated model can’t fix a biased sample. 👉 #Statistical rigor isn’t just about algorithms — it’s about the data story before modeling even begins. As data scientists, our credibility depends not only on #predictivepower - but on representativeness. #DataScience #Statistics #MachineLearning #SamplingBias #Analytics #CausalInference

In data analysis, missing values are the darkness, while observed values are the light. Let’s shed some light on common challenges. 1️⃣ Causing a Forest Fire from Trying to Get Warm 🌲🔥 Indiscriminately replacing missing values with medians can invalidate analysis. First understand why data is missing—skipped, ineligible, or non-response—and handle accordingly. Darkness can’t be painted over; it must be understood. 2️⃣ Mistakenly Bringing a Light Source That Assumes Power Is Present Methods like multiple imputation are wrong if the underlying assumption is violated. For example, people might skip a question about sexual orientation because it feels invasive. The decision to skip isn’t random—it’s tied to the question’s sensitivity. Imputation in such cases introduces bias and invalidates results because imputation assumes data is Missing at Random (MAR) 3️⃣ Mixing Day and Night 🌞🌙 Combining missing data with valid categories, like grouping non-responses with smokers, leads to misclassification bias. Combining bad data with good data results in bad data—always. 4️⃣ Treating Darkness as Light 🌑✨ Indeterminate responses should not be forced into categories where they don’t belong. For example, if a survey asks, “Do you smoke?” with responses: 'Yes,' 'No,' or 'Not sure,' combining “Not sure” with “No” creates misclassification bias. It’s better to treat “Not sure” as missing. 5️⃣ Misreporting the Length of Daylight 🕰️ When conducting analyses like regression, missing values are automatically excluded. Always report the actual analytical sample size, not the raw dataset size. 6️⃣ Ignoring Patterns in Shadows 🕵️♀️ Patterns in missingness, like younger respondents skipping income questions, often signal biases. Use visualizations or tests to explore patterns in missing data before deciding how to handle them. 7️⃣ Not avoiding the shadows altogether The best way to manage missing values? Avoid them from the start with proper questionnaire design. Use clear, concise questions, provide “prefer not to answer” options for sensitive topics, and pilot-test your surveys to detect issues. Prevention is better than cure! 8️⃣ Overlooking Flickering Lights 💡 If results change drastically depending on the method used to handle missing data, it’s a red flag. Sensitivity analysis is your firefighter friend. 9️⃣ Treating All Shadows the Same 🔀 Recognize that not all datasets can be salvaged. Sometimes the data are missing to such an extent that the data is no longer fit for use or fit for purpose. The only thing worse than no data is bad data. 🔟 Ignoring the Context of Shadows 🔍 Sometimes, what’s missing tells us more than the data itself. Patterns of missingness can reveal societal attitudes or highlight populations that need targeted approaches for sensitive issues. Join our community and stay tuned for a special webinar series on data analysis, including handling missing values! 🔗 https://bit.ly/3EkddQV #Chisquares #VillageSchool #Research