A short update from the ARP team, including details about the changes they are making to their storage system.
Background
The Africa Rainfall Project aims to run computer simulations of rainstorms in sub-Saharan Africa to improve weather forecasts for the region. Providing accurate weather prediction data to the local farming community is crucial for agriculture and self-sufficiency. Comparing results obtained through computing rainfall data from various sources using the World Community Grid, scientists can create increasingly accurate forecasts and improve future simulations and in turn weather predictions. To learn more about the specific details of the project, check out our ARP update from March 2022.
As of December 2022, the ARP team has paused releasing new work units, until they resolve the capacity limitations of their storage system. More about this pause can be read in our work unit status forum update.
The last 100 days for the African Rainfall Project
We asked the ARP team about their recent developments and this is what they shared with us:
“It has been really quiet for the African Rainfall Project, at least on the front end. Behind the scenes, necessary changes now have been made to start going again. In previous news items, we talked a lot about the data volumes involved, which are indeed impressive. It is, however, also a major problem in terms of storage and moving them from one place to another. In the case of ARP, the results from the volunteers are sent by the WCG to computers at SURF, the collaborative organization for IT in Dutch education and research. The first 200TB could be stored on disk, which is relatively easy to access. That was, however, also the maximum amount allowed. There is also “work storage”, where results can be analyzed and compared. Officially, the maximum amount allowed there is 50TB but ARP had sent 150TB, so the data needed to be moved elsewhere. In this case, “elsewhere” was tape storage, which is actually still the cheapest way to store data but also involves physically moving tape-reels to and from reading and writing units. Even though these tapes are moved by a robot, one can imagine that it takes some time and one would only move data on or off tape once every now and then. One could probably give the command to simply copy every file drive to tape but that would probably get stuck somewhere and one may also have transcription errors. Instead, we moved the data one generation at a time (i.e., two days of simulation)and used a so-called hash function to make sure the bits on the original medium were the same as on the new medium.
This has now all been solved so there is room for continued activity by volunteers in the African Rainfall Project. We now have about 265 days of results and we would really like to fill the complete year with the final 100 days because of the very seasonal character of rainfall in Africa. Close to the equator, there are typically two rainy seasons and two dry seasons as the Inter-Tropical Convergence Zone moves with the sun. To the north and south, there is typically one rainy season and one relatively long dry season. So a full year really adds scientific value to the dataset. Support by SURF for large storage will come to an end but a different long-term solution has been found to ensure that scientists can continue to analyze the outcomes to better understand weather in Africa and its predictability."
Heavy rain in Africa. Photo: Jan Friesen
Thank you to the ARP team for their continued partnership with WCG and their help creating this update. If you have any comments or questions, please leave them in this thread for us to answer. Thank you for your support.
WCG team