On the programming of CumulusUtils (2)


More than 6 months ago, I wrote a blog  on the how and why of the programming of CumulusUtils. It is time to do that again because in 6 months, over the winter and now deep into the Corona Virus crisis, a lot has happened with CumulusUtils. When I wrote the first blog, it was mainly about the inspiration and development of pwsFWI. And it still is the most important module of the whole program (at least, it is for  me). But while working on pwsFWI and seeing where it was deployed  triggered something. Beside having the pwsFWI, one needs to think about the weather. It is not just a black box, it is about understanding the weather and it’s effect on vegetation and animals (including humans). To understand the weather you need to look at the parameters, the measurements.

So I started out creating modules which made it easier to get an overview without switching pages. The original Cumulus website for example – let’s call it the seagull site – has several pages of weather info which actually should be beside each other for comparison ( today, yesterday, this month, this year, records, monthly records ) or at least easily switchable. The fact that the info was limited is understandable in the time it was created and it did trigger a large development of tools. However, that sprawl of techniques and tools, ingenious as they often were, made the weather website a real technical issue. Just being a weather enthusiast was  not enough, you had to be an ICT enthusiast as well.

Partly understandable: you need to be able to handle wireless, computers, programs. Sure. But does that need to be to a level of programming, configuring in  PHP, javascript  etc… I don’t think a lot of that is needed and I doubt it is wanted. Many weather enthusiasts no doubt live through it just to have their gadget, but they would love to do without. And beside that, the confusion between the data layer and presentation layer did not lead to elegant solutions (more layer distinctions can be made, but there is no need for more abstractions now).

I assume.
I did not do market research.

Nevertheless, it gave me the idea of the website generator. Primarily because I wanted another site for myself. And that idea, I will describe here. Some design facts, some algorithms, some programming technique. If you don’t like the technique to handle it, maybe you appreciate reading about it. Especially when you use the Website Generator, you may be wondering what’s inside.

The Design

When the idea of the Website Generator emerged, it was clear by  me it had to comply to some goals:

  1. A limited amount of tools. As CumulusUtils had already been written in C# and was based on direct access to the datafiles of Cumulus, I would not break that liaison. But then the choice for the client. Lean and mean I wanted it. So I went for a javascript driven site which would use data selected at generation time. This meant I could use the modules which existed already and expand that by the same method into the future. I skipped all existing tools like the SQL database, PHP (which for me is kind of incomprehensible, but that’s probably taste) and any form of scripting. The result of this would be that the basic site exited of two files: index.html and cumulusutils.js, the javascript library. That was basic enough for me.
  2. Recognisable. The resulting site had to be recognisable as a descendant from the Cumulus stable. So I chose the Bootstrap template as basis for the user experience. The block based grid which is known by all Cumulus users from the management interface, would be recognisable present. I also decides to maintain the original Cumulus graphs and gauges and embed these into the site. It had to run on the ‘realtime’ features provided by Cumulus.
  3. New features. It had to have new features which nobody else had to make the statement that it was new. So I embedded my modules into the system, gave it an expandable menu, a large screen design which should be responsive (not very successful in the first release) to scale down acceptable to smaller screens and gave it standard interface for the sun and moon. And finally, a feature which I designed actually towards the end of the first implementation: the switch of the dashboard between the gauges and the text based standard Cumulus dashboard like it exists on the standard interface.
  4. No touching of the Cumulus Environment. CumulusUtils has to blend in. It must not touch the functionality of Cumulus itself, nor disrupt it’s performance. So it copies the datafile when using, takes as less CPU/Disk as possible. One should hardly notice it is there.  Only the footer of the site will actually show a program has run to update the site with the latest data..

The Interface

The user interface consists of the following parts, al of which are defined in Bootstrap HTML :

  1. The header which contains the name of the site. The name is taken from Cumulus.ini, parameter: locname;
  2. The NavBar which contains the menu. Part of that menu is fixed i.e. the user cannot touch  it. It contains the defining items for CumulusUtils like pwsFWI, Graphs and some other reports. the NOAA reports belong to Cumulus so they also are embedded into the menu, even if the  user does not use it, they belong there. Aside the fixed menu, there are two things the user can modify here:
    1. There is a user menu which the user can name and expand at will. An example of how to do that is supplied.
    2. There is an About menu which gives a Cumulus Utils About and a License description. But there is also a default entry where the contents of a file ‘CUserAbout.txt‘ is displayed. This is where the user can tell his story of his site, its components and what he wants with it.
  3. Dashboard Left. The left part contains the ‘realtime’ data of what is happening with the weather as measured by the weather station. Its data comes from realtime.txt and realtimegauges.txt, both of which are timer based read (Ajax1) and displayed. This is basically standard  technique as used by all templates. The realtime part of course is handled by javascript. What is new to this dashboard, is that the user can toggle between Text display and Gauge display. Apart from small differences, the information shown to the user is the same. It is a matter of taste, but it gives a real nice feeling and the performance is much better than the classical page load.
  4. Dashboard Right. This is actually the display with most freedom. In this area, all standard reports and graphs, the map, the system info is displayed to the user choice. But there is more. The user can through a menu choice of his own actually display any file here and as far as I have tried, HTML is interpreted because it is simply an innerHTML load into a Div-element. This means the user can create almost everything and display  it in this area. I am curious for what this will bring. The javascript function to use in the user menu is ‘LoadUtilsReport(‘filename’)‘.
  5. The Footer. The footer is a bit like the header, there is little functionality. It shows the identification of the CumulusUtils version, the copyright notice and the Cumulus identification. In times of trouble I can show there the messages of the Ajax functions, but currently that is configured away.

That is all  about the user interface.
The leading technique – apart from Bootstrap – is javascript / Ajax: display files which are output of the  modules. Straight text files.
The essential thing of a website as Tim Berners-Lee invented it :)

The Sun and the Moon

While making the dashboard, I asked myself: should I take in the sun and moon times or should I make a separate almanac and astronomical pages. Many sites have additional tooling for [earth] sciences like earthquakes, volcanos, astronomy etc… sometimes the weather is hard to find between all planets, satellites, earth disasters  like quakes and eruptions. And I agree, those are often the site I spend more time than I thought I would. Nevertheless, that is not the staring point for this Website Generator. The starting point is the weather as registered by the user on his weatherstation with the help of Cumulus. So I stick to that.

But  having said that, the sun and moon are the most important drivers of life on earth and actually they are the drivers of the weather. All energy which raises temperature, blows the wind, raises the tides…. it all comes from the sun and the moon. So, yes, I wanted the sun and the moon incorporated in the dashboard.

And when you make that decision, it comes down to 1) how do I calculate, 2) how far do I go in accuracy and 3) how do I display.

  1. The calculations are quite complex but very well  described and many people have implented those equations. I would like to point to two sites which I used for testing: suncalc.org and mooncalc. For the actual calculations in CumulusUtils, I use a javascript library suncalc.js by Vladimir Agafonkin. I am not sure how that goes, Beteljuice claims there are some issues with that library but the results are satisfactory so far.
  2. Accuracy is an issue. All those equations are  approximations and the library I use is of course no exception. In general, I don’t care too much. It is a weather site. If I have the sunrise a few minutes wrong, I don’t care. Seen from the remarks made by Beteljuice suncalc.js may be too inaccurate, in which case I will look for another way of calculation. I will not create the library myself though. That is too much, too many have already done that, why throw away good work? So, I’ll check and maybe try another library. Some time.
  3. Presentation or how do I display?
    1. For the sun it was evident, I would use the interface of the unbeatable Beteljuice. But I was not going to use his code – which is in PHP – because that would contradict my design. So I rethought: how to create this image. And that is how I came to use d3.js. d3 is a  library to create datadriven documents. So I represented the duration of the different phases of the day (day, night, dawn etc…) as a fraction of the 24 hours the day takes and put that in pie chart with d3 as an SVG image. The hand of the clock is just a line with a rotation transformation which is calculated once per minute. If I don’t count the d3 library, it is all done in 60 lines of code. That satisfied me.
    2. But than the moon. Those calculation were a bit difficult, as they have been through the ages. The moon is a confusing, hard to understand astronomical body. It was so hundreds of years ago, it still is now if you really want to understand what’s going on. So, imaging the moon on the basis of the calculation (moon age, angle to the earth or fraction visible are mostly used) is mostly done the easy way out: take an image which fits the data and display. Especially when Cumulus offers moon images and moon calculation. But that is not what I wanted: no external files, apart from what CumulusUtils calculates itself. I had the calculation but I needed to abstract that to an understandable image. It took me some time to understand there are actually three moons to use to create the image:
      1. A background moon
      2. A half moon which overlays 1) on the right side when waxing or waning, changing colour as required
      3. An ellipse moon of which the x-radius and colour is determined by the phase

The algorithm to display the moon with d3 as an SVG image consisting of a circle, a semicircle and an ellipse is 70  lines of code and leads – I think – to an [easy?] understandable image of the phase  of the moon, a nice abstraction. I like my own work. The image is updated once per hour (if you leave the line open).

The Gauges

As far as I know CumulusUtils is the first tool to actually use a switch between displays or dashboards as I call them. The point is that both dashboard contain wind gauges and that appeared to be rather difficult. As the switching technique consists of creating two document sections for the same area, the gauges library cannot handle two gauges of the same type on one page. So when switched, the old gauges are no longer known in the DOM under their ID and instead the same ID is created on another place. As a result, the gauges library need to be reinitialised. And that does not exist in the library itself. The only way to reinitialise the loaded javacode is by reloading it. Reloading forces that specific library to reset and start again.

So I reloaded gauges.js which did the trick but unfortunately did not work for the windrose. After some days fruitless trying and debugging, I improved a lot of things, but not the reinitialisation of the gauges. In the end I asked Mark Crossley and he came with the call that did the trick : $(‘#rosePlot’).parent().remove(); which than is followed by a true reloading of the library : $.getScript(‘lib/gauges.js’);

But then it’s not good enough so I made a pause procedure to stop all timer interrupts and create a kind of synced getScript with a simulation of the Document Load Ready function. As follows:

1) The Pause function

Pause = function () {
  PauseInEffect = true;

  if ($.active > 0) { jqXHR.abort(); jqXHR = null; }

And 2) The call to getScript:

         $.Deferred(function (deferred) { $(deferred.resolve); })
       ).done(function () {
         gauges.config.realTimeUrl = 'realtimegauges.txt';
         gauges.config.realtimeInterval = 15;
         RT_timer = setInterval(loadRealtimeTxt, 15000);

That is it. Relatively easy with some minor difficulties, solved with a little help.
If anybody needs more detail let me know. The code is not public domain at the moment because it is not really in that phase of the lifecycle yet, but it’s no secret either.

Anyway, these are my considerations and experience while creating CumulusUtils.
And now it’s live.

1] Ajax: Asynchronous JavaScript And XML, a technique to read external files. The content of those files is usually used to modify the webpage.

Geef een reactie