Our developers Aya Stead, Alan Chodyniecki, and Sam Collard recently joined a DataKind UK DataDive weekend in London, hosted by Softwire. We’re proud to support this data and AI-for-good initiative, helping the DataKind UK charity to support social sector organisations with data science so they can make better decisions and have greater social impact.
The event’s aim was to aid Starlight, which supports children’s experiences of the power of play and boosts their well-being and resilience during treatment, care, and recovery from illness. The challenges our engineers faced included collaborating with a diverse group, handling new data, and working to a tight deadline. How did they tackle these challenges, and what did they learn?
The problem that Starlight wanted to solve
While the team was briefed under an NDA, we can reveal that Starlight wanted support conducting an end-to-end exploration of its various data sources. Our engineers’ task was to discover and understand what was possible in the future so that the charity could plan for the most beneficial allocation of its resources.
The ten challenges they faced and how our data engineers addressed them
At the beginning of the weekend, our engineers joined a mixed team and got to work, connecting closely with the ambassadors of Starlight. Along the way, they identified ten challenges that impacted their journey and shared their insights on how they tackled each one, advising how other engineers can respond if they come across similar problems.
- Negotiating a limited timeframe
With less than two days to work on the project, the data engineers had to design, implement, and refine complex data pipelines and systems, so it was important to get started quickly and have a unified approach.
Sam’s top tip for overcoming this challenge was to have an open, frank conversation to find out what everyone felt comfortable working on and then start there. As small goals were reached, the task list was refined. They said, “This involved a lot of asking and answering questions to ensure that everyone was on the same page and making progress from the get-go”.
- Working with diverse experience levels
Being part of a group with varying skill levels in data visualisation, data science, and data engineering can be challenging because there are different ways of working. Our data engineers aimed to understand each participant’s working processes, bridge the gaps in technical understanding, and maintain effective communication of their work to the rest of the team.
After grasping the team’s backgrounds, our engineers evaluated their strengths and weaknesses, and how to leverage them for the weekend’s goals. Alan particularly found his experience with data science and visualisation enthusiasts to be enriching: “I found it very interesting learning about the different methods and goals and found that there were complementary skills that helped make better results when combined.”
- Unfamiliarity with a new team
With a new team of six participants, data engineers could face challenges in establishing effective teamwork, understanding team dynamics, and assigning tasks based on strengths. It’s normal for engineers to work with new people on each project, but it might not be normal for people in other fields.
Aya outlined the team’s approach: “Initially, we discussed our interests and assessed each person’s skills. We prioritised tough tasks first, ensuring fair distribution. Then, lessons from these tasks aided our progress on the rest of the tasks, which maintained team morale and momentum. It was a highly effective strategy.”
- Real-time problem solving
In any project, unexpected problems like data quality issues or tool compatibility can arise. In a time-bound project, teams must quickly adapt and maintain quality by involving the client for feedback along the way.
Although our engineers encountered minimal problems, Sam recalls a standout issue: “We found that certain data didn’t align as we desired. We improvised by creating a new dataset and integrating it strategically. This was an unforeseen challenge, though your job is to find a proactive solution to make progress.”
- Data integration
Integrating varied data sources can be complex, especially when dealing with limited documentation or inconsistent data and formats. Our data engineers had to develop strategies to efficiently clean, transform, and combine diverse datasets.
Aya notes that DataKind UK’s well-prepared datasets saved a lot of time: “Raw data can often be messy. Addressing this during a weekend wastes time as it needs context from the client. Pre-cleaned datasets made in advance are better, and results and lessons learned from these can be applied to subsequent datasets by the client in the future.”
- Choosing appropriate tools
Selecting the right tools and technologies in a tight timeframe needed careful consideration because do-overs wouldn’t be possible, and the tools would need to align with the project’s needs.
Our engineers were able to adapt their skills and use tools that other data science and data visualisation group members were already familiar with to avoid downtime and support the team.
Sam expands on this: “We had that conversation early to ask what tools everyone is used to working with. From our experience working with different technology stacks in our Softwire project work, we could be more flexible and get up to speed with any technology quickly. This avoided a learning curve for others to adapt to our tools and made for a better working experience for all.”
- Ensuring the tools can handle the volume of data
Managing large volumes of data can raise concerns about whether the available software and hardware being used can process the information at speed. To avoid this issue, our data engineers came armed with a suitable laptop for coding and access to their installed and online tools, which enabled adequate storage and processing power for their tasks.
Also, Alan suggested a great way to cut down on the volume of inputs: “The key is to work out what we need to identify that will specifically help us with achieving the project’s end goal. Dedicate your time to this first. Then save even more time by giving it a quick look-over – you’d be surprised how many times I find oddities or duplicated data that will cause problems later.”
- Documentation and handover
With limited time, there’s always a risk that documentation could be overlooked, but our data engineers knew the importance of good documentation. Aya explains: “The documentation is a record for the client to understand how we took data and statistics and took them all the way to the result. We need to explain what worked and what didn’t, where all the information came together, and the structure of the solution. It also serves as a handover, covering what to do next so that the Starlight team can rerun the data and pick up where we left off.”
The group decided to set aside an amount of time to write up the activities, working on their own sections before collecting all the parts together. Aya talked about her really positive experience of how this went: “Everyone was so supportive and helpful. When someone had finished their tasks, they offered to help pick up the documentation side, self-organising themselves and taking initiative.”
- Creating an innovative solution
Under the pressure of an upcoming deadline and adapting to changing circumstances, making the outcome as innovative as possible feels like an added challenge.
Sam explains how they innovated along the project’s length: “We considered the charity’s specific needs and constraints, suggesting innovative and effective ways to make a meaningful impact, like presenting novel ideas and variation possibilities of the client’s questions, which the client hadn’t considered before. It was a proactive way to provide a bit more value to add to what was already important work.”
- Meeting the charity’s goals
Did the technical outcomes align with the charity’s social impact goals? Balancing technical excellence with the real-world needs of Starlight’s projects took some delicacy to achieve.
Feedback proved pivotal, and the group used the charity’s ambassador for progress validation and to talk about our next steps. Alan said the key question he liked using was ‘Is this useful to you?’. “As you go through the event, you start to see the insights that can be taken away”, he says. “It’s a continuous feedback loop that shared what we were doing and what the client responded with – a core agile process that we employ in all our projects.”
Now that the weekend has finished, would our engineers do it again?
Aya: “Data science is something I’ve been relatively adjacent to for quite some time; however, I had never really dabbled in applying it until taking part in the DataDive weekend. The skillsets required to use code for data science are very different from the more typical software development I do in my day-to-day, and it felt like I was exercising whole new muscles. It has helped me understand the strength behind being able to quickly apply queries to datasets that shed light on new information.
“Ultimately, I feel I have a new tool in my kit, supporting my pre-existing skills. I have developed an understanding that software development and data science have something of a symbiotic relationship, and I hope to indulge in both sides of that more in the future!”
Sam: “It was really, really tough, but the client was extremely happy. I didn’t think I had much interest in data science or engineering, but I had a great time, and I’ve gotten more involved with it at work since. It was so good to be working with people who really cared about the charity. The ambassadors did a fantastic job cleaning up the data. The whole experience was wonderful.
“Our skills were very apparent when we got the code for processing the data into a useful format for the client to rerun it later. We don’t work in a way where we write a one-time thing, and the result is all you receive. We work to structure code so that it is maintainable, reusable, and friendly. Those skills served us well when we had to then share that code with other people who will need to run it against updated data in the future.”
Alan: “My honest personal takeaway was that it was really good to understand what data scientists do as part of their day job. There were loads of different backgrounds on our team, so I could go around discussing the data and understanding how they use it in their work. I’m very interested in identifying places where data could be used more effectively to help make decisions – how you’d go from ‘I kind of am interested in knowing more about this’ to actually getting some useful business insights at the end.”
What did the client say?
Suzy East, Head of Volunteer Programmes at DataKind UK, said: “We loved collaborating with Softwire. As well as sharing their office space and providing support so the event ran without a hitch, their team showed tons of enthusiasm and really got involved! We appreciate their clear commitment to supporting charities and championing responsible data use in the third sector, and hope to partner with them again in future.”
David Amos-Reeves, Head of Business Intelligence and IT at Starlight, said: “We’re so grateful for the support of Softwire, Datakind UK and the volunteers who helped us with this project. We learnt so much about improving our data quality and how we can use and link our data in creative ways to enhance the value it can add to us. We got some interesting insights into how we can better target our services and fundraising. Thank you!”
How can you do more with your data?
Thank you to the DataKind UK team, who did a fantastic job setting up and organising the event. It was an exciting challenge that used our engineers’ skills in technology, people management, and collaboration.
You can find out more about our upcoming events and see past event summaries on our Softwire Events calendar. If you’re interested in learning about the data projects that Softwire gets involved with, check out our Data Engineering page for all the details.
