Fixing slow data processing

"My data processing runs too slow on my computer"-problem. This is a problem many researchers face during data processing. But while the problem is clear and may affect the researchers productivity and quality of results, the solution is seldom straightforward.

Last modified: 23.10.2018

The core issue(s) may be numerous, and the possible solutions to address the issue(s) are just as diverse. However, often the solutions may be categorized into two groups 1) change of existing approach 2) keep existing approach, but there is a need to scale. In the first group we often observe solutions such as

1. Change of data processing methods
2. Change of implementation and computing methodology
3. Change of data representation
4. Change of numerical methods

If these changes can address the issue, the researcher may still use the workflow on the computer that he/she is familiar with, and often these changes can be put into place on a short term time scale. These types of solutions are often the best choice, if possible, as they allow the researcher to keep using the workflow he/she prefers and knows is in line with scientific methods practiced the respective academic field. On the other hand, more extensive problems do occur. In such cases, there is a need for scaling:

5. Scale to cluster computing
6. Scale to high performance computing
7. Scale to hardware specific computing (e.g. GPU computing)

Solutions using scaling often call for a significant change in the underlying codebase and is more time consuming. Furthermore, it is likely to cause a significant change in the workflow. But if scaling is the only method to conduct the research, then we should scale. 

Together with the researcher, we work on a  case-to-case basis to identify the right solution for the specific computing problem. After analysis and clarification of the issue(s), CLAAUDIA may assist in providing the necessary code and workflow changes to alleviate the "my data processing runs too slow on my computer"-problem.

Book a meeting

Book a meeting


We would love to talk to you, because research is complex and we know from experience that the devil is in the detail. Our team holds a wide range of competencies, and we will set the consultancy team on a case-to-case basis.

A meeting is free and without obligations. We do not always have to meet in person - but we would like to come and visit you, if possible. But we can also meet over Zoom, MS Teams or similar. 

Request a meeting at claaudia@aau.dk

Day-to-day support

Day-to-day support


Is it just a simple question, or are you unsure if you need a meeting? We also provide day-to-day support and advice by e-mail or phone on matters that do not require a meeting.

Questions for currently hosted services: support@its.aau.dk 
Other requests: claaudia@aau.dk