Ask Me Anything: Atul Sharma


Userlevel 4
Badge +2

Join Atul Sharma, chief technology officer (CTO) at Peak

for our next Ask Me Anything (AMA) session

 

When: Thursday 19 May 2022 at 12:00 BST

Where: Questions and answers posted in this discussion

 

Introducing our guest: Atul Sharma

 

Atul’s experience and skills make him the ideal person to take your questions on career and all things tech. Starting as a software engineer, Atul gained a wealth of experience as a data engineer and consultant before joining forces with Richard Potter and David Leitch in 2016 to form Peak, the Decision Intelligence company.

 

Submit your questions

 

Got questions about business, career, data engineering, tech? Whatever your question, post it below and Atul will answer as many as he can.

liammccaffrey 2 months ago

The AMA has closed now.

Thanks so much for everyone who took part in the discussion and to Atul Sharma for joining us today.

Look out for more AMAs other Community events coming up on our Events page.

Have a great day 😊

View original

21 replies

Userlevel 4
Badge +2

Hi Atul! I have a few questions around tech and culture: 

  1. Data is everywhere, we’re living in the Data Age, but not everyone is ‘data ready’! What cultural change do you think companies that want to embrace the full power of Decision Intelligence and AI need?
  1. I maybe biased, but Peak are an excellent example of a tech company who are getting talent attraction and retention right. In your opinion, what does it takes to attract and retain the top tech talent across the world?

Finally, here’s a more personal question about your career:

  1. Do you still like to program? What was last thing you programmed, what did it do and what language was it written in?
Userlevel 6
Badge +2

What are the biggest changes in the data industry you’ve noticed since you joined it? 

Userlevel 3
Badge +1

Hi Atul,

What can Data Scientists learn from Data engineers and vice versa?

Related question: Is there a rule-of-thumb or method for how do Data engineers prioritise front-end development (e.g. feature enhancements, user requests) with back-end development?

 

Userlevel 3
Badge +1

Do you have a memorable “best bit of code I’ve ever written”?

Do you have a memorable “worst bit of code I’ve ever written”?

Userlevel 2

Hi Atul! 
 

Thanks so much for taking the time to answer our questions. It’s very kind of you and great for our curious minds.

 

My question for you is:

to what extent does Peak view other data science and AI parties as potential sources of collaboration vs purely competition? Would Peak ever consider acquisitions or just partnerships? Is this something likely to develop over the coming years?

 

What sorts of other opportunities for cross-pollination of creativity are part of the Peak strategy to spearhead the data science/AI movement worldwide, whilst balancing IP and keeping cutting edge?

 

Thanks again!

Diane

Userlevel 2

What do you see as the main barriers to traditionally technically ‘conservative’ sectors such as housing, charities, and public health joining the AI revolution?

What steps is Peak taking currently and which roles should someone with passion about getting these sectors to maximise their potential with data science and AI look into to apply to work at Peak?

Userlevel 2

What is *the* big picture problem you look forward to seeing data science and AI tackling in the next 10 years?

What, if anything, do you feel is over-promised in the next 10 years?

Userlevel 1

What is an immutable data layer and what are the benefits of implementing one? 

What’s the difference between a data lake and data warehouse? 

What are the major differences between AWS Redshift and Snowflake?   

What are the core skillsets you see being required to be a data engineer going forwards?

Where does data engineering end and data science start, and what areas cross over?  

What schema approach best suits the work done at Peak? 

Userlevel 3
Badge +1

What was the thing you were most proud of early in your career (pre-Peak)?

Userlevel 3
Badge

What is *the* big picture problem you look forward to seeing data science and AI tackling in the next 10 years?

What, if anything, do you feel is over-promised in the next 10 years?

There are too many big picture problems AI will solve but the biggest change I see coming in the next 10 years is that AI going mainstream and helping organisations to run more efficiently and help humans to have better quality of life. 

The way steam power, electricity and computers have an impact on us, AI will have the same impact if not bigger. This will create different kinds of systems and jobs. 

If I have to pick a field, I will pick medical science and AI will help us release new drugs faster and diagnose diseases earlier. 

Artificial General Intelligence will still be a gimmick at least in the next 10 years unless new AI models start to train/create better AI models and the cycle becomes a lot faster.

Userlevel 3
Badge

What do you see as the main barriers to traditionally technically ‘conservative’ sectors such as housing, charities, and public health joining the AI revolution?

What steps is Peak taking currently and which roles should someone with passion about getting these sectors to maximise their potential with data science and AI look into to apply to work at Peak?

Main barriers are data availability, “fear of unknown”, inertia of not doing anything and industry not being able to come up with the right use case for AI. We also need to be very careful and sensitive towards industries such as public health and have a healthy debate around ethics before jumping on using data and AI. 

Peak is sharing a lot of knowledge using blogs and events. We work very closely with some local universities and have a great network. Peak is working on a product which will allow more and more organisations to use decision intelligence to grow and optimise their business. 

It will be hard for me to answer questions on a role. Each person/sector/organisation is different but happy to talk further or put you in touch with the right person.

Userlevel 3
Badge

Hi Atul! 
 

Thanks so much for taking the time to answer our questions. It’s very kind of you and great for our curious minds.

 

My question for you is:

to what extent does Peak view other data science and AI parties as potential sources of collaboration vs purely competition? Would Peak ever consider acquisitions or just partnerships? Is this something likely to develop over the coming years?

 

What sorts of other opportunities for cross-pollination of creativity are part of the Peak strategy to spearhead the data science/AI movement worldwide, whilst balancing IP and keeping cutting edge?

 

Thanks again!

Diane

The answer is, it depends. We already have a strong partners network including AWS, Snowflake and few others. They all have offerings very similar to Peak but there is a lot of synergy which allows us to work with a lot more organisations than we can do on our own. 

We currently have commercial, technical and implementation partners and we look for more if there are common grounds to help each other.

Userlevel 3
Badge

What are the biggest changes in the data industry you’ve noticed since you joined it? 

Biggest challenge I faced in the data industry was focus on technology, not outcomes.  Data industry spent time and money to solve technology issues, not business outcomes. For me, It doesn't really matter if GCP or AWS is used to build a data lake, as long as business is benefiting from it in a matter of weeks not years.

Another challenge is converting data into insights. Technology can help to gather and organise a lot of data. But converting it into insight and action is very different. A good buy in from stakeholders and robust business processes are pre requisite for it. This is where Peak can real help :).

Userlevel 2

Thanks again for your thought provoking responses I’m so happy to see how balanced and considered the Peak approach is

I actually have a conversation tomorrow with a member of your team so I will be sure to put my questions about roles to Charlie then

all the best 

Di

Userlevel 3
Badge

What was the thing you were most proud of early in your career (pre-Peak)?

I worked for a company and their one customer for almost 5 years. I helped my company to grow that account by more than 400% (that is the proudest achievement) and delivered some complex projects. And yes, I wrote loads of SQL too. It was really great for me as even as engineers our good work was allowing my company to grow the account. Upsell is the best sell :).

Userlevel 3
Badge

Hi Atul! I have a few questions around tech and culture: 

  1. Data is everywhere, we’re living in the Data Age, but not everyone is ‘data ready’! What cultural change do you think companies that want to embrace the full power of Decision Intelligence and AI need?
  1. I maybe biased, but Peak are an excellent example of a tech company who are getting talent attraction and retention right. In your opinion, what does it takes to attract and retain the top tech talent across the world?

Finally, here’s a more personal question about your career:

  1. Do you still like to program? What was last thing you programmed, what did it do and what language was it written in?

First two are really hard to answer but below are my answers. We can talk more about them if you are interested as it is a very interesting topic.  

  1. Data is everywhere, we’re living in the Data Age, but not everyone is ‘data ready’! What cultural change do you think companies that want to embrace the full power of Decision Intelligence and AI need?

Atul : There can be few things which can set any organisation on this journey in right way. 

  1. Buy in and right signals from top execs.

  2. Culture of continuous improvement. This will make sure the right questions are asked and data will be required to answer those questions. 

  3. Basic training for everybody.

  4. Champions in each department and initiative to identify problems/opportunities which can be solved using Decision Intelligence. 

  1. I may be biased, but Peak is an excellent example of a tech company who are getting talent attraction and retention right. In your opinion, what does it take to attract and retain the top tech talent across the world?

Atul : Opportunity to learn Cloud + AI + Latest tech!! I think we are in a great place to attract and retain data science. We can do better in engineering and product. I am not sure we are there yet so it makes it harded to answer. Attracting talent requires brand recognition and talking about complex problems that we solve. Retaining talent is all about communicating “what” and “why” and letting people find how. As more and more people join, they start to influence “What” as well which is great. As an engineer, I never changed a job if I had loads to do and found it challenging or interesting. I try to keep it that way for engineering and product too (not always successfully though). 

 

  1. Do you still like to program? What was the last thing you programmed, what did it do and what language was it written in?

Atul : No, I don’t code these days. I wrote my last production grade code in 2010 and moved to a data architecture role. Most of my coding days were spent as Oracle ERP consultant and then data engineering. So Oracle, SQL Server, Pro C, SQL, PL/SQL, Oracle Forms/Reports, Unix scripting and dabbling in ETL tools such as ODI, SSIS. My code is still running for a customer called PCF in Peak as it was in the early days. I enjoy the product and startup side of things more than coding nowadays. Having said that, I think I am still the best SQL dev in the UK if not the world :). 

Userlevel 3
Badge

Do you have a memorable “best bit of code I’ve ever written”?

Do you have a memorable “worst bit of code I’ve ever written”?

I can spend hours on this one :). 

I have loads of examples. I am always more interested in making $$ for the organisation paying my bills so here are the top two :). 

  • I was working for a vehicle leasing company. It was growing very fast and overnight batches (invoice generation, printing, accounting, BACS collections etc.) were over spilling into working hours. I managed to rewrite some of them in 2-3 months. I reduced running time by more than 50%. It saved us a lot of money by using the same hardware for the long term (no cloud those days). Tech as Oracle, PL/SQL and SQL. 

  •  For a data warehouse project I wrote a DBT type of tool on Oracle. It can be used by business users for data sync between upstream and downstream systems. This stopped the need for a lot of data engineers (very expensive contractors like me) to write code for each new data source. Business users could just add an extra line in the config file for new tables/files.  It is still being used. Tech was C#, Oracle and PL/SQL. 

“worst bit of code I’ve ever written”

 

My worst one was being overconfident and running a script on the Production database without test and review. It created tables in different schemas (where we used to have synonyms) and that means data getting loaded in wrong tables for some time. I have to spend more than 12 hours with my colleagues in front of the terminal to fix my sins. 

Being an over confident developer, I made this mistake a few times before but not after this massive incident.  Even if you write 1 line of code, make sure it gets tested and reviewed is my advice before going to production.

Userlevel 3
Badge

Hi Atul,

What can Data Scientists learn from Data engineers and vice versa?

Related question: Is there a rule-of-thumb or method for how do Data engineers prioritise front-end development (e.g. feature enhancements, user requests) with back-end development?

 

Data Engineering is all about 

 

  1. Moving huge amounts of data in batches or streams from one source to another in the most efficient and fault tolerant way while handling volume, velocity, veracity and variety.

  2. Transform data for grouping, aggregation, cleaning, organising, formatting and a lot more.

 

From my experience DS can learn a lot of things listed above in an efficient and organised way. They can also learn what to push down to SQL (if used) and what to do in the data frames. Data Engineers are generally better at using native features of a specific tech too.

 

Data Engineers can learn a lot about feature engineering and different ways data needs to be prepared compared to a BI solution or integration solution. Data Engineers can also learn new types of transformation which are not required in their day to day work but needed for DS work. Last but most important, learn to convert data into insights and actions from data scientists. 

 

I will not be able to give a good answer on getting priorities right as data engineering was behind in adopting agile methodology compared to full stack engineering. Best way is to go for an iterative and incremental approach with limited scope/testing instead of massive projects. 

Most of the data pipelines were multi-layered and “write on” schema. In this changing business, ready on schema and better data lineage was an option which should be given priority.

Userlevel 3
Badge

What is an immutable data layer and what are the benefits of implementing one? 

What’s the difference between a data lake and data warehouse? 

What are the major differences between AWS Redshift and Snowflake?   

What are the core skillsets you see being required to be a data engineer going forwards?

Where does data engineering end and data science start, and what areas cross over?  

What schema approach best suits the work done at Peak? 

 🤣 I was not expecting interview questions. But hey answer coming up!! Good that nobody asked what is linear regression 😃 as I can’t answer that. 

Userlevel 3
Badge

What is an immutable data layer and what are the benefits of implementing one? 

What’s the difference between a data lake and data warehouse? 

What are the major differences between AWS Redshift and Snowflake?   

What are the core skillsets you see being required to be a data engineer going forwards?

Where does data engineering end and data science start, and what areas cross over?  

What schema approach best suits the work done at Peak? 

What is an immutable data layer and what are the benefits of implementing one? 

 

Immutable data lake is the only way to build lakes before the lake house concept came in and I still think is the best way.  In immutable data lake data is always sharded and loaded in INSERT mode with a timestamp compared to UPSERT which traditional staging techniques or lake house can support. It is great as when dealing with massive data, loading/sharding data is more important than lookup and update and increasing point of failures. It alway keeps all historical data. It works best in ELT mode and UPSERT can be considered a transformation step once data has been handed. 

What’s the difference between a data lake and data warehouse? 

 

Data Lake is better suited for ML, keeping tons of data in expensive data stores and running deep dive analysis using READ ON Schemas. Data lakes can work as staging too. DW are for specific reporting requirements and need data organised in dimensions and facts to make self serve reporting easy. They are not suitable for ML at all and may use ML Model outcomes as one of the data sources. If you are building an EDW, then conformed dimensions make it more extendable for future needs. 

What are the major differences between AWS Redshift and Snowflake?   

 

Redshift is old technology which is being changed for cloud computing, snowflake is built for cloud from ground up. 

 

What are the core skillsets you see being required to be a data engineer going forwards?

 

Skills for data engineering : Any cloud, SQL, Python, Apache Spark and good handle of data stream and API as data source, publisher. Old days of proprietary ETL tools are probably gone. 

Where does data engineering end and data science start, and what areas cross over?  

 

Data Engineering is a sub field of data science now so they kind of merge. 

 

What schema approach best suits the work done at Peak?

 I am still not sure. I think it needs a new database technology or data modelling technique required for AI. When I have the answer, that will be the second startup. 

Userlevel 4
Badge +2

The AMA has closed now.

Thanks so much for everyone who took part in the discussion and to Atul Sharma for joining us today.

Look out for more AMAs other Community events coming up on our Events page.

Have a great day 😊

Reply