On the programming of CumulusUtils

  • HansR 

Introduction

Having developed theoretically a Fire Weather Index (see my pwsFWI) and creating a first implementation of it in C for analytical purposes, I needed to take the software a bit further to make it more robust and useful. I had some feedback which made me realize it was urgent to have a good Fire Weather Index (FWI) for Personal Weather Stations (PWSs).

This blog is about the context and the actual programming requirements.

Feedback and context

My  interest and experience in wildfires (or bush fires in Australia) comes from my forestry studies and my life in the south of France where forest fires are abundant during summer season. But noting prepared me for the experience of Phil Porritt from Inverell, Australia who provided me with feedback, when I used his weather station data for my first analysis of the pwsFWI.

He explained to me that the weather in NSW is changing and the drought period(s) are getting longer and specifically this year (2019) the drought start earlier than normal. He provided links to newspaper articles[1] and in one of his mails he describes the impact of the wildfires (bush fires) on his family. His grandfather, who was a  fire fighting captain at one point, died fighting a bush fire when in his early eighties. In the area around Inverell, Phil says,  farmers have neither feed or water for their livestock so both food crops and live stock are diminishing. He continues:

My uncle,  grandfather’s son , now 74 still runs that property and has had to reduce his stock by 70% and is hand defining what he has remaining as well as casting water on a daily basis.  One of the other issues he will gave if the drought does break is that he will no longer have breeding stock to increase his stock holding again. He normally runs (stocks) 15,000 head of sheep on his 20,000 acres but I believe he is now down to 6,000.

This feedback is important, it contains an almost live report of the environment changing, very probable because of climate change. Then at the 4th of September – early spring in Australia – a warning for bush fires was given by the government of New South Wales (NSW) and the 6th fires broke out and according to the site are still burning, new fires breaking out every day.

This is the context in which the first testsite of my Fire Weather Index is placed with Phil as site owner and local tester and contact. Such an extreme environment – I have no other qualification for it – should make clear what my theory is worth and what changes to the implementation may be required.

With changing climate, increasing temperatures and drought, Fire Weather Indicators are a necessity to make public aware of what is going on. Personal Weather Stations need to have an FWI available which relates well to reality.

 

Development requirement

Phil offered to be a test site, but he had no Raspberry Pi (for which I wrote the original software in C) and was running CumulusMX on Windows.

Looking around the world of Cumulus today – I first installed my PWS in 2008 and changed to Cumulus in 2012 – a lot has changed. Originally the windows version of Cumulus 1.9.4 was a standalone program which took care of data gathering (of multiple kinds of weather stations), data storage and displaying info on the data, tabular as well as with graphs.

Soon the world changed because people wanted to display their weather on the internet and websites were created. Cumulus had – already very early – a standard website and a method to send near realtime data to any site. People picked it up and started creating PHP scripts to easily create sites around the data it could get from applications like Cumulus (it isn’t the only one around). Attention shifted from data gathering to data display. The display layer was – and is – basically on the websites where a diversity of PHP templates is alive.

But the effect was that the local data needed to display were (partially) brought to the website either via SQL to a database on the websites or by transferring the actual data files to the website. In both cases a complexity of data transfer and pluriform location of data was added to the system. I regard this, as a dead-end-road. In addition, I do not think PHP is a language well suited for data handling, it is not a general purpose language. So the effort and complexity of the programming increases with the use of PHP (and possibly SQL). Also every page request requires a data request. This means website inefficiency added to the increased complexity.

So, although I have ideas about how to set up Cumulus for the future, at this moment I can only make a choice, taking  CumulusMX as a starting point. I did not look into similar programs, Cumulus is a very open and easy access program, for data and for code. It is likeable.

 

Actual implementation road

Having thought it all over, giving the above, I decided to make and addition to CumulusMX, which could run independently and on any frequency, but which had direct access to the datafiles. The output had to be in a format usable by any website. So, I took the following decisions:

  1. The utility had to run on any platform CumulusMX does. This means it was best to use C# (CumulusMX is programmed in C# so other languages were not considered);
  2. It has to run from the directory where CumulusMX is installed with the same rights as the CumulusMX-user. The data are directly accessible in the data-directory. Every file opened for use will be copied first to prevent lock interaction with CumulusMX as is advised in the Wiki. For running, one can either use the scheduler of the operating system or the scheduler by CumulusMX itself, which provides the realtime, ftp and daily frequencies.
  3. Any output for the user will be written in plain text format. My current approach is that I create functional HTML tables with rudimentary formatting. The inclusion and the website itself are for the user. The output files can be to the required destination by the extra webfiles feature of CumulusMX, choosing the frequency.

Having worked on this, I managed to create a working system within a bit more than a week (learning C# on the go [2]), which I named CumulusUtils. Anybody using CumulusMX can run this program. I implemented my pwsFWI for a start, which is currently under test, and a top10 record list. On a short term I expect  a system status page to be implemented for Linux/Unix. Other functionalities may follow.

 

Finally

I started out with the top10 list in C, just for my own pleasure. With the pwsFWI, I needed the outside world and therefore I needed to change to C# (which btw is a blessing). The feedback from Phil, made me want to accelerate the test, because they are now approaching the fire season in Australia and climate change is not waiting. Therefore I pressed on with some  programming.

My added goal with this exercise, CumulusUtils, is to create an easy interface between the data of Cumulus and the display layer, bypassing PHP and SQL as a functional data handling tool.

My choice for HTML output may change in future, depending on user  feedback. This does not hold for complete functionalities like pwsFWI where the theoretical background is too important. I present pwsFWI as a complete product embedded in the utility.

The utility ‘cumulusutils.exe’ is downloadable from support forum of Cumulus. Updates and support go through there. Here you may find additional information and changes in theoretical background. Some discussion on the issue and the theory can also take place on the forum.

 

Thanks to Phil Porritt for his feedback.

 


[1] ] See eg : february 2019, en of january 2019 and while I was publishing beginning of September 2019 there was a warning and there were actual fires (the info on live fires in NSW can be found here).

[2] I have been a professional programmer until 2000, principally using C. Picking up C# was a good experience. I left C++ because I thought it too complex. C# has all the good things of object oriented life, but the ease and the simplicity of the early languages. Not saying it is all easy. Many pitfalls and many libraries to learn.

Geef een reactie