Blending Data Science and Software Projects at Microsoft

As more and more customers adopt AI and go deeper with data science, we see a shift in the composition of project teams along with a shift in how we conduct projects in general. We've been dealing with this paradigm shift here at Microsoft, as well as when working with our customers for over a year now and this article attempts to capture my learnings.

Data Science products require a different approach. There is some overlap and there are stark differences and learning to structure the team and their effort makes all the difference. Let's start by looking at similarity's and overlap between software engineering and data science across stages of product development.

Comparison and Overlap

1. Requirement Analysis:

   • Data Science: Focuses on understanding the problem, defining objectives, and identifying data sources.

   • Software Development: Involves gathering requirements from stakeholders and defining the scope and objectives.

   • Overlap: Both start with understanding the problem, defining clear goals, and accounting for responsibility and security.

2. Planning:

   • Data Science: Plans for data collection, cleaning, and analysis.

   • Software Development: Develops a project plan with timelines, milestones, and resource allocation.

   • Overlap: Both require detailed planning to ensure project success.

3. Design:

   • Data Science: Involves designing data pipelines, feature engineering, and model selection.

   • Software Development: Focuses on architectural design, UI/UX design, and system models.

   • Overlap: Both involve creating a blueprint for the project, though the specifics differ.

4. Development:

   • Data Science: Involves coding for data processing, model training, and evaluation.

   • Software Development: Involves writing and testing code to implement features and functionalities.

   • Overlap: Both require coding and testing, though the nature of the code differs.

5. Testing:

   • Data Science: Focuses on model evaluation using metrics and validation techniques.

   • Software Development: Involves various testing types (unit, integration, system) to ensure software quality.

   • Overlap: Both emphasize testing to ensure the quality and reliability of the output.

6. Deployment:

   • Data Science: Deploys models into production environments and sets up monitoring.

   • Software Development: Deploys software to production and conducts final validation.

   • Overlap: Both involve deploying the final product and ensuring it works in the live environment.

7. Maintenance:

   • Data Science: Regularly updates models with new data, monitors performance, and model drift.

   • Software Development: Provides ongoing support, updates, and enhancements.

   • Overlap: Both require ongoing maintenance to ensure continued performance and relevance.

Conclusion

Both fields are integral to modern product development and often intersect, especially in projects involving data-driven applications. A key learning early on was to divide the team's effort into two broad stages, experimentation (unclear path forward) and product development (clear path forward).

I talk about this in my blog https://www.drjoeshepherd.com/blog/leading-ai-projects but the gist of it is when you are experimenting the team should adopt a hypothesis driven development approach that is led by the data science team and supported by engineering. Once you have a product defined and a clear path to get there engineering takes the lead, supported by data science. 

As always, knowing your team and knowing your customer are critical factors but understanding how to employ your team and their respective talents in a just-in-time manner can make things a whole lot easier.