Data Science in Practice

A place to talk about the challenges and exciting parts of applying data science to real world problems

  • 19 Topics
  • 39 Replies

19 Topics

SorchaGilroy
SorchaGilroyCommunity Manager
published in Data Science in Practice

Purpose of this section

Welcome to the Data Science in Practice part of the Community 👋Some ideas for discussion for this section:How to approach different types of problems - e.g. optimisation problem vs classification problems Dealing with difficult stakeholders  Driving actions from your outputs Data science problems that come up in particular industries - e.g. retail or pharmaceuticals Mapping out business processes in a way that can feed into your solution design What questions to ask business users to best come up with a solution How to check that the solution you’re proposing is the right one for your end users. And anything else you can think of related to data science in practice!

120
SorchaGilroy
3 months ago
E
EvolvingRichieActive Contributor
asked in Data Science in Practice

Should parquet be the default for storing large data?

Mostly people store data in CSV, but sometimes the data can be so large it takes a long time to read in. I hear that because they are smaller, parquet files are faster to read in, letting you get into the data faster. the downside I guess is that you can’t quickly open the file in excel to check something, but for most big data you’ll rarely do that anyway. Assuming:I have some large data files that I use regularly they are about 3 million rows so can be slow to read in I never need to open them in excelWhen shouldn’t I be using parquet?

1
ChrisBillingham
23 days ago
D
darianraadFrequent Visitor
asked in Data Science in Practice

What are the greatest challenges or blockers to implementing AI solutions in businesses?

We all want to build data science solutions that get used by customers to make substantial improvement to their organisational processes. But sometimes even the best solutions with incredible technical fundamentals can fail at the acceptance or implementation phasesWhat are the biggest challenges and blockers to implementation?And what (if anything) can be done about them?One example is when the major stakeholder just doesn’t agree with or understand what you’re doing, and no amount of explanation is helping (yes, models are imperfect representations of reality. No, prediction outliers are not as important as you’re making them out to be: look at the typical behaviour and overall KPI uplift). It sometimes helps if you can find another senior stakeholder that this person trusts to buy into the project. Another potential route is to get the end users to fall in love with how much more productive and happy the solution is going to make them. There is a huge opportunity cost in trying to c

1
liammccaffrey
29 days ago
BenLawson
BenLawsonActive Contributor
posted in Data Science in Practice

How do People organise their code?

Hey Community, Just something I’m curious about, how does everyone go about organising their code when they’re working on a solution?I’m currently using a system where I have a R script that will never see the light of day🚮. I write code in it to the point it functions correctly. Then when it achieves the desired effect, I implement it into a package for the project, which I then use for testing/putting into production. I was wondering if anyone else had other ways of working I could try, or any weird ways of making code 

3
M
30 days ago
BenLawson
BenLawsonActive Contributor
asked in Data Science in Practice

How do you add vertical gridlines between columns in the formattable R package?

I’m trying to make a copy of a visualisation of a table from Excel in R so that it can be automated as the original version is very labour intensive. From a google it seems formattable is the best package to achieve this, but I can’t find any documentation for adding lines between columns. Does anyone have experience with this package, and know how to do that?

5
M
30 days ago
A
AndyLauCommunity Member
posted in Data Science in Practice

When to make use of Open Source data?

Many customer projects could benefit from additional data from other sources. However, this comes with its own complications such as:How is it maintained? How is it stored? Are the customers OK with this approach? How is it sourced?

2
SorchaGilroy
1 month ago
E
EvolvingRichieActive Contributor
posted in Data Science in Practice

Regression discontinuity for estimating the causal effect of marketing campaigns

I had this idea the other day, dunno if its been done before:Say you want to know how effective your marketing campaign is for VIP customers. However, marketing won’t let you randomly assign VIPs to ‘gets campaign’ or ‘doesn’t get campaign’ for a true AB test. What do you do?Well, if VIPs get the marketing campaign and non VIPs don’t, naively you could compare VIPs to non VIPs to estimate the effect of the campaign. The problem is that VIPs and non VIPs differ in all sorts of ways and so you'd have lots of confounds. How do we address these confounds? Enter regression discontinuity. The idea is that if VIP status is determined by a customer exceeding certain thresholds (e.g. ‘average purchase value above £35’) then you could compare VIPs to customers who just missed the VIP boundary (e.g. average purchase value of £34). Assuming that whether a customer spends £34 or £35 on average is more or less random chance, there will be basically no difference between these groups on average. So b

0
E
1 month ago
C
Cinzia BragliaCommunity Member
posted in Data Science in Practice

Explainable Machine Learning

I recently came across this paper Principles and Practice of Explainable Machine Learning.In my personal experience as a Data Scientist working with different stake holders in different industries, being able to explain and motivate what is happening under the surface of a model enables the user to better understand the decisions suggested and consequently trust them! What to you think about explainable Machine Learning? Why is it important when working in industry? Is it always possible to create a model that is explainable?

1
lara
1 month ago
M
Magdalena NowakowskaActive Contributor
posted in Data Science in Practice

How do you use Exploratory Data Analysis (EDA) to inform your problem formulation?

One of the first steps in tackling a data science problem is exploring and understanding you data. Initial analysis can discover important trends and patterns as well as identify areas where you can deliver the most value.How do you use EDA for problem formulation? Do you adjust your plans based on the outcomes of you initial analysis? How do you communicate to your customer that you need to change your plans after you had a look at the data? 

1
E
1 month ago
EmmaBellamy
EmmaBellamyActive Contributor
posted in Data Science in Practice

What are your top tips for tackling Data Science problems beyond code?

Here are a few of my top tips:  Work with all stakeholders involved to improve poor quality data Don’t forget the importance of setting goals and clearly defining the scope especially with difficult stakeholders Build trust in the models by ensuring that model outputs are reliable and accessible to end users  Would be great to hear about your top tips or experiences!

6
lara
1 month ago
F
FranBakerCommunity Member
posted in Data Science in Practice

How can we use ai for good?

Hi everyone, We often hear the phrase of AI for good but I would be interested to know what examples everyone has seen and what ideas you have for doing more.Here at Peak we do a lot of work with Fareshare, who redistribute surplus food to charities that turn it into meals. We are helping them to understand their data better, for example to get better engagement with their suppliers by creating a portal allowing Farshare to produce personalised impact reports so that suppliers can see the difference that they are making.Looking forward to hearing your stories….

7
E
2 months ago
L
LeeWhittakerFrequent Visitor
published in Data Science in Practice

Commercial applications of data science

Since the turn of the millennium, there has been an explosion in the number of commercial  companies using data science to optimize their business strategies. As companies – even smaller ones – accumulate masses of data, it’s clear that leaving this data to stagnate is a waste of useful information that could potentially give a business a competitive advantage.  As data amasses and computational power increases, how we view data is changing; it’s no longer a means by which to assess the past, but a predictive tool that could be used to aid decision making.  Gartner predicts that a third of all large organizations will employ some level of artificial intelligence (AI) to aid their decision making before 2023.  Here we discuss why data science is so attractive, what it is that data scientists actually do and how a new, emerging discipline within data science is changing how businesses approach AI.  Data science: why do we need it? Using the masses of data generated by a company, it is po

830
L
2 months ago
tomtom
tomtomActive Contributor
asked in Data Science in Practice

What data science trends do you anticipate for 2022?

Ahead of the in person ensemble event today (did someone say pizza?!)https://community.peak.ai/events/data-science-trends-for-2022-7What trends have people seen in data science this year and what trends do you expect to see over the next few years?

4
HelenCraven
2 months ago
Wes Raynbird-Tilbury
Wes Raynbird-TilburyActive Contributor
asked in Data Science in Practice

What's the first thing you do when you start a new project?

Personally, I like to get straight into the data and do some EDA (Exploratory Data Analysis). This allows me to explore the type of data available, how the data tables fit together and the wider context of the project. Following this, I feel ready to start formulating the problem, creating solution diagrams and defining deliverables.

1
SorchaGilroy
2 months ago
gareth_york
gareth_yorkCommunity Member
asked in Data Science in Practice

Whats worse, bad data, no data or sparse data?

We all would love to have mountains of high quality data with perfect coverage of the problem at hand but the real world is grimy and full of corrupted CSVs maintained by one legacy member of the customer team. If you had to choose your poison what would it be?

6
Wes Raynbird-Tilbury
2 months ago
H
HarryJacksonFrequent Visitor
posted in Data Science in Practice

How do you approach different types of problems - e.g. optimisation vs classification?

It would be interesting to hear how approaches vary in terms of formulating an optimisation problem vs classification problem vs any other types of problems people solve in data science?

1
gareth_york
2 months ago
H
HarryJacksonFrequent Visitor
posted in Data Science in Practice

How do you design a good solution diagram?

A good solution diagram can help map out data inputs, workflows, dependencies and outputs in a clear and concise way. Often these diagrams can be more interpretable to customers. How can we create solution diagrams in a consistent way that ties in nicely to the Peak platform?

0
H
2 months ago
Wes Raynbird-Tilbury
Wes Raynbird-TilburyActive Contributor
posted in Data Science in Practice

How to understand your customer, their constraints and what success looks like

This topic is for discussions surrounding understanding your customer:Understanding your customers, their market and the context of the problem What constraints does your customer have? What limitations exist? What does success look like? What are their KPIs? What does the customer really care about?

0
Wes Raynbird-Tilbury
2 months ago
Liz
LizActive Contributor
published in Data Science in Practice

Turning data into actions

"Data, data everywhere..." but it's not useful unless we can actually use it! Data insights need to be actionable to be valuable, but more than that – they need to be impactful, they should direct action and decision making. ‘Nice to know’ won’t move the dial and drive meaningful results for your business. What is an actionable insight? An actionable insight is a compelling truth that has a meaningful impact on a course of action. It is often a fact that would have gone unnoticed without in-depth analysis. These insights will engage your audience; they should be powerful enough to influence their decisions and implementable enough to drive positive change.  Why does it matter? Businesses are producing more data than ever before, and most now understand that it can give them a competitive edge. Actionable insight is how companies unlock that potential quickly and effectively, they provide a specific solution to a business problem and a firm grounding for an evidence based decision.  Whe

550
Liz
3 months ago

Badge winners

Show all badges
Terms & ConditionsCookie settings