git push and other copy commands

The usual pattern of a copy command:

copy source destination

The pattern of git push command

push destination source:destination

the first destination is the destination repository - the second is the branch (and 'source' is the source branch). I must have skimmed the linked manual dozens of times over many years. There was something that just did not make sense and I imagined that I would have to read some book or something to eventually understand the magic 'ref specs'. It never dawned on me that it was that simple.

The additional difficulty is that the destination repository is usually shortened to 'origin' (because it was the origin for the clone command).

Old code and survivor bias

If code is well designed, decoupled and has good test coverage there are chances that it will be regularly updated to new technologies and new programming techniques. The other code will probably scare programmers and have much chance to stay in it's sad state until decommission.

Extending the 'Template Method' design pattern

There is this notion that you should not put semantics into names, or maybe that what your program do should not change in alpha-conversion. But of course everyone is breaking that rule if she is using the Template Method design pattern (if we agree that the library they use is somehow outside of the program and is not alpha-converted together with it). There the names of methods have a lot of semantics.

In Advanced Search in web DBIx::Class based applications I proposed an extension to DBIx::Class that would treat all methods with names search_for_* as predicate builders. This way programmers could easily extend the query language used by DBIx::Class with new predicates by adding methods to the ResultSet class. Now in WebNano all *_action controller methods also have a special meaning - these are the methods that can be called from outside via automatic dispatching from an url. Taken literally sub my_method_action is not that different from sub my_method : Action which is the way to specify that in Catalyst. The difference is only ' : A' versus '_a' (and I could use *_Action if I wanted to golf it even more) - but yeah - I am cramming the semantics into the name while Catalyst reserves it for the code attributes. Catalyst way is cleaner - but my goal was a 'minimal' framework. The 'Extended Template Method' design works in a few lines while MooseX-MethodAttributes is 13 packages with who knows how many lines of very subtle code.

What surprises me is that I have never seen a description of this 'Extended Template Method' design pattern. It looks rather obvious, maybe it is called by some other name in the literature.

Object::Tiny:RW and MooseX::NonMoose

There are conflicting requirements for WebNano. On one hand I would like to build big heavy web applications based on it, and of course I would like to use Moose as the object framework for them. On the other hand I would like to give it a chance to work in the restricted environments of most shared web hosting serving CGI scripts - so heavy dependencies and XS and big startup times are out. And I would also like to make it suitable for serving huge amounts of simple Ajax requests - so it needs to be fast.

First I tried using Any::Moose and let people decide if they used Moose or Mouse. By using Mouse people could solve the startup problem and also there is some way to use Mouse without XS.

But Mouse was still huge dependency - so next I asked myself it it would be possible to subclass in Moose a non Moose base classes. That proved to be rather simple with the aptly named MooseX::NonMoose. Using it in WebNano I can build my objects in what ever way I choose and as long as I stick to the standard hash based objects I can extend the WebNano classes in my Moose based applications with no problems. Or at least it seems to be with no problems for me now - if someone knows more about disadvantages of this solution - please leave a comment.

After discovering MooseX::NonMoose I switched to use Class::XSAccessor. It small and very fast, but unfortunately it was XS based. I was thinking about the possibilities of bundling the software as a PAR package or so and using it in the restricted shared hosting environments and relying on a C compiler seemed to complicate things a lot.

Next I tried Object::Tiny, it was tiny, fast and did not use XS, there was just one but - it did not produce the attribute setters. I could live with that and it did not take much time for me to port WebNano to use Object::Tiny, but still it seemed like an extreme choice. Fortunately just browsing CPAN I discovered a fork of O::T - Object::Tiny::RW - this one is small, fast, does not use XS and does produce attribute setters. I am still a bit worry about why Adam would not add that feature to his module - given that it is such a trivial change (compare the sources of Object::Tiny::RW and Object::Tiny - its just one line changed), he must have had a good reason for that?

Technically right, socially not so

Imagine that you let anyone use your hard work by publishing your code as Open Source and then instead of thank you notes you receive angry emails about purported bugs in it.

Imagine that you noticed a problem about some Open Source code and as a good citizen you analyze it and report your findings and instead of a thank you note you receive a reply explaining how much confused you are for thinking that that could be a bug.

Imagine that this happens in a public space like an mailing list, open bug tracker, etc. This is like the setting of a Greek tragedy - both sides have their own rights and values, both sides can be technically right - and yet the conflict spreads and escalates. People copy the emotions of each other, even perceived emotions and especially anger, when displayed publicly leads to more anger. This is a vicious circle of positive feedback loop. When it starts everyone gets irritated and everyone wants to know who started it, who is responsible for that.

But go back to the start - it's not anyone's fault - you cannot point the finger on one person and say 'he is responsible'. Looking for the main culprit is really looking for a scapegoat - a surrogate victim that would heal everyone's consciousness. This is not especially geek thing overall - it is a universal trait - but sadly inventing fodder for the scapegoat search is a prominent part of geek culture.

One of the many things that I admire about Larry Wall is how he can neutralize the poisons of angry reactions. People scorn Perl for being a 'scripting' language - he talks about Ada Lovelace:

Suppose you went back to Ada Lovelace and asked her the difference between a script and a program. She'd probably look at you funny, then say something like: Well, a script is what you give the actors, but a program is what you give the audience. That Ada was one sharp lady...

Installing dependencies - continued

I don't know what I am doing wrong - but after installing Perl 5.12.1 under perlbrew "cpan -m ." stopped to install the dependencies. It seems that for making and testing it adds the libs in '.cpan/build' to the PERL5LIB search path so both can go on without really installing the dependencies:

Running make for /home/zby/progs/ravlog/.
Prepending blib/arch and blib/lib of 108 build dirs to PERL5LIB; for 'get'

"cpan -t . " works in the same way and if the tests fail then "cpan ." stops there - so I cannot tell if it would install the dependencies in the case when the tests pass.

I don't really know how to investigate it further - but I am noting that as it is very frustrating. Eventually I had to resort to cpanminus ("cpanm --installdeps ." is really handy as it addresses the problem directly).

Implementing file upload progress bar

At work I was assigned the task of replacing the previous hodgepodge of tools that provide the progress bar functionality for forms with file uploads. At first glance this seems like a trivial thing to do - you periodically observe how much of the file you've got and update the progress bar, but there are details that make it hard to do with the current tools. I report here how I solved that problem - not because I think this is the optimal way - but rather to open the discussion and because I could not find any description of solving it when I googled for it.

First of all the common programming tools (like CGI.pm or Mason that we use here) assume that the page handler receives the whole request as input - and that whole request is not available until after the file is uploaded. So for example 'my $q = CGI.pm->new' will not finish until it is too late to measure the upload progress. The solution to that is to use another page to report the upload progress and call that page via Ajax from Javascript code updating the progress bar. This would work great - but the file is normally uploaded to a temporary file with a random name and the other script would not have any chance to guess it. We need to generate a new random file name in the form page and then pass that name to the form handler script so that it would save the data to that file, and in parallel to the Ajax scripts that would check the size of that file.


To save the data into a specified filename I used the CGI.pm callback feature:

my $q = CGI->new( \&hook, $fh, undef );
...
sub hook {
my ($filename, $buffer, $bytes_read, $fh) = @_;
print $fh substr($buffer, 0, $bytes_read);
$fh->flush();
}

It is described in the subsection called "Progress bars for file uploads and avoiding temp files" of the CGI.pm documentaion, but actually it is a great leap of thought to say that it supports progress bar implementation, you still cannot use it directly to get the progress bar from the CGI object on the form landing page, you still need the separate scripts measuring the progress. For my solution all I needed was to pass the target file name to the code saving the data, this could be easier than writing this callback above. And the callback is still not everything - I yet need a way to pass the generated filename from the form page to that script - and not via form parameters, remember they are not available at that stage. So how can that be done? Simple - as PATH_INFO - which is available in the %ENV hash even before the params are parsed by CGI.pm.

This is the skeleton of the solution - there are a few more details in the actual implementation - but the code will be published soon as Open Source - so I hope everyone will be able to look them up there.

When those micro-seconds matter

Catalyst is the only web framework that has a mailing list I subscribe to - but I am sure that it happens at others too. In a recurring pattern someone someone posts a benchmark showing that Catalyst for some trivial operation is many times (or many hundred times) slower than some other web framework or for that matter PHP. That does not fail to generate a heated debate - but eventually the seasoned framework developers gain the upper hand with the argument that for all the, often big, web sites they worked on, those few micro-seconds lost in the Catalyst dispatcher never mattered much because the application spent hundred times more in other code fragments and mostly in business logic parts, so shaving off some part of the few micro-seconds would not improve the overall speed more than 1%. This is a great argument, perfectly reasonable and rational but it is biased towards the status quo. It might be true that everywhere where Catalyst is currently used it works great but it is also not hard to imagine an application with very simple business logic that needs to serve millions of users, Twitter anyone? Or an app that does many simple Ajax callbacks. Sure you can always code the speed requiring, simple parts in PHP and keep Catalyst only for the other more heavy-weight tasks but having a universal solution would be so much more convenient.

Templates and inheritance

In every web site project I have been working on there inevitably came that moment that someone wanted to have a copy of the site with some small changes, or that someone wanted to use a piece the website in something else. In both cases it was all about (initially) some small changes, what they call as skin or a branded site. With programming part we know what to do then - we design base classes for the core undifferentiated functionality and inherit from them in the targeted classes. Most templating engines do have some notion of search path - so this should be doable with templates - but it always proves to be much more messy than it initially seems to be. I'll try to catalogue here all the use cases and how they compare to the standard inheritance in object oriented programming languages and describe them in terms of operations on the search path.

Initial setup

In 'normal' programming languages the atomic piece that can be inherited and overloaded and that is searched on the inheritance path is a method - in case of templates the atomic part is a whole template. The granularity is defined differently, but we can do analogues things with both. Here I will concentrate on the dispatching model familiar for users of most of the MVC frameworks with controllers and their methods as the endpoints (called actions). In this setting all the methods could have a matching template file and inheriting and overriding a template could work on the same level as inheriting and overriding an action.

Basic case

Lets say I have a controller 'MyApp::Controller::Book' with a few actions like 'view', 'list', 'edit' (a CRUD controller) and matching templates in 'templates/Book'. I would like that when calling the render from that controller I did not need to specify the full path of the templates - but just call them by name 'view'. Also when they have a common subcomponents in the same directory they should be able to [% INCLUDE date_formatting %] them by name. This is just like calling methods in a class and it can be accomplished by adding the 'templates/Book' directory at the start of the search path.

Library component

Sometimes we would like to use ready made components from some published library, maybe a CRUD controller for our Books. We inherit from the library class in our 'MyApp::Controller::Book' and we would like its templates to be inherited in a similar way. That means adding '/usr/lib/something/something/CRUD/templates' to our search path somewhere after 'templates/Book'.

Subclassing our own controllers

If our 'MyApp::Controller::Book' is a subclass of our own ''MyApp::Controller::Item' class - this is a very similar case to the above. Here as well we want to insert the search path of the parent class somewhere after 'templates/Book. But in this case what we insert is 'template/Item' - a path relative to our application root.

Application wide components

This is solved simply by adding an application wide element in the search path. It would be analogues to adding stuff to a common base class in standard programming - much more rare there.

Subclassing a whole site

This is the case of 'branded' sites where we would like to subclass everything at once. It is maybe the most common case - but also the most difficult because there are no good analogies/solutions in standard object oriented programming. If 'MyNewApp' inherits from 'MyApp' that does not automatically mean that 'MyNewApp::Controller::Book' springs to existence and can serve book information in the new app (but I've heard there are existing Object Oriented languages that work like that and some people are thinking about implementing this in Perl). So I think we need to first think over how this is handled in the main framework and then we can deduce something about how to treat templates from that. For the limited case of application wide components this can be solved by adding the new template root to the beginning of the search path - but it gets more complicated for all the other cases described above.

Conclusions

I mostly wrote that to have it all in one place - but I think it does show the need to make the template search path dependent on the controller used.

A subject for mst's talk

I did not really intended to vote in that silly poll - but I think I have found a subject for the talk that could be highly beneficial for the Perl community: How using the insights from cognitive science (and what else is needed) I am going to make HTML::Zoom actually usable for mere mortals.

WebNano - what I did not say at the London.pm tech meeting

About a month ago I had a presentation at the London.pm tech meeting about the design ideas I plan to experiment in WebNano. There are a few things that I did not have time to tell - and I plan to blog about them. First of all I think I should have admitted that I stole the both the name and the idea from Simon Cozen's MicroMaypole (I hope he will treat that in the terms of the well known quote about imitation and not intellectual property trespassing:).

At the presentation I mentioned that there is not much code in WebNano currently - but maybe this was not the best way to put it. I actually don't plan to ever have much code in it and the example of MicroMaypole shows that perhaps such a minimal web framework can be a practical thing, but I do plan to build some more examples using that minimal code so that I can be sure that even if minimal it does have all the desired features. What are those features will be the subject of other blog posts soon.

Bread::Board

Lots of packages in the Catalyst namespace are simple adaptors of external libraries. In many cases what you need from that adaptor layer is only passing initialisation values from the config file to the appropriate call to the object creator. By using the Dependency Injection pattern you can move that initialisation code out of that "deep in the framework guts" place into something that can be called another config file - containing values that are complex objects instead of simple constants. This way your view object can be constructed there and then injected into the application which does not need to know how to construct it. You still might prefer to have a different config file with simple literal configuration values using the standard YAML or Config::General syntax.

If that was too vague for you - don't worry - Stevan promised some more docs for Bread::Board soon. The real take away from all of this is that Bread::Board and future sub-classes of it spares you from the need to writing adaptor classes for all the view or model classes of your yet another web framework. Maybe you'll not need a framework at all - and assemble your application using Bread::Board out of a web dispatcher and other ready made parts.

Frameworks are not yet well refactored libraries.

At first we had Catalyst. After some refactoring we got HTTP::Engine and now we have PSGI/Plack and the tools around that abstraction that provide a lot of functionality that previously required using Catalyst. Now all of that is in libraries that can be used independently from each other, there is no coupling between the components other than the well defined and pretty minimal PSGI interface.

The title of this blog post might not be universally true, it is possible that there are some kernel parts in some frameworks or even most of the frameworks that could never be in any practical way reduced into sets of libraries. But what I observe is that frameworks tend to grow and only in very exceptional cases spawn libraries that are refactored out of their main body. It is just so much easier to just write new extensions, plugins or things that use the whole framework than find the right abstraction that would let to split some functionality out of the main code base. Certainly I would never guess that the simple PSGI abstraction would let people refactor so much into separate libraries, I bow to it's author. But we should always try:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Antoine de Saint-Exupery

Scope and Immutability

I am sold on the immutability thing. I started to think about various designs and about what immutability of objects would mean to them and I think the results are promising - they tend to have much more clear structure - and just thinking about immutability gives you a chance to see an additional structure in the code. One quite obvious rule of such design is that you cannot keep data of narrower scope in objects with wider scope - and this forces you to think about the scope of various data pieces in your programs. For example in web oriented programming you have (usually) three scopes:

• Application - i.e. all data that is persistent to the whole web server process (like configuration)
  
• Session - i.e. data saved into the session variables (there is an opinion, and I mostly subscribe to it, that you should keep that minimal).
  
• Request - i.e. the data you receive from the web client and that you send back to it
  

An example of framework that does not respect this division is known to everyone - it's Catalyst - fortunately the app/context split is mostly done.

Acknowledgements: Stevan thanks for the web apps scope break down :)

External dependencies once again

My proposal to add a 'lib_requires' keyword to the META file - with identical semantics as Devel::CheckLib did not make at this round of META spec changes. Now I think that maybe a better strategy would be to start with adding this to the documentation. I am sure many packages already do that - but I have encountered others that don't. What I have in mind is a standard POD section that would be generated by all the module scaffolding generators (like Module::Starter) and which would contain all the information needed to install the external lib in common environments. After this becomes a common thing we would be able to make more informed decisions about including it into the META file.

Warszawa.pm - Biblioteczka (The Library)

W czwartek odbyło się spotkanie Warszawa.pm. Warta zanotowania jest inauguracja naszej biblioteczki - pożyczyłem od Piotrka Czynnik Ludzki. Dzięki!


Last Thursday there was a Warszawa.pm meeting. Worth noting is the inauguration of our library - Piotrek lended Peopleware: Productive Projects and Teams to me. Thanks!

Role that adds Traits to multiple classes

The DBIC model stuff naturally spans the whole FormHandler class tree - there are things that need to be added to the main form class, but also to the Field classes. An obvious example of the latter is the lookup_options method which loads data from the db table (ResultSet object more precisely) appropriate for the field into the field's options array. It does not really touch any global form stuff and:

$field->form->lookup_options( $field, ... )

is like reaching with your left hand to your right pocket. I mean really awkward. So what we need is a form role that would add other roles to the fields classes. I imagined that this could be possible, in theory a role could be a hierarchy that would be applied to a class hierarchy - but I don't think you can do that with current Moose, so instead I used a hook in the method that builds fields and added there code to inject an appropriate Trait.

The code is in new branches in the main HTML::FormHandler repo and in the HTML::FormHandler::Model::DBIC repo. I am waiting for commetns. I would think that this is not something unusual - so I guess others must have already solved similar problems before - I would like to hear what they did.

We have a similar problem with rendering. It is a role now so that it can be easily replaced, but it is naturally recursive, fields should know how to render themselves so we need a way to apply a whole set of roles in one sweep to the whole form class structure. It looks like there really is some abstract super-role in there.
How much is it monkey patching, how much is it action at a distance?

Controller immutability

On IRC I've got an answer from t0m and others about why they would not use controller attributes for storing data related to the current request - they want to keep their controller object immutable for the request phase. The idea is that if you have an object that is being reused then it would be nice if after each use of that object it is still in the same clean state. That is it is nice to always have the changes contained in a minimal area.

I'll leave it to the experts to list all the advantages of immutable data structures, but I've seen why we needed a similar compartmentalisation in HTML::FormHandler. The FormHandler form object is reusable, you can use it to validate many parameter hashes. We started coding it as a mutable object and storing all the changing values in the form object itself, the first problem was that we kept forgetting to reset them at the start of processing of new data. This lead to many subtle bugs, sometimes difficult to track so we started to just recreate the form object for each processing run. This worked OK, because the form object itself was not that heavy to create until someone had a form object that contained other heavy object and recreating it would require recreating them. So eventually we separated all the state that changes during the processing into a separate Result object, now to clean the form state we need to recreate only that separate object.

It needs to be stated clearly that this separation of immutability is a trade-off (as always) - it is an additional requirement so it increases the complexity of the code and means for example that the $self object in controller actions is underused - because actions should not change it's state (including storing something in it's attributes). Certainly it is not feasible to make such change now and this also would be a trade-off but for the sake of exercising imagination: maybe actions could be methods on the changing part (and not on the controller object) and only use the controller object in some way - this way all the basic code (like that from the Manual) would use $self object in the way common to object oriented code.

Catalyst::Component::InstancePerContext

How often do you use the controller object? It is the first, most important parameter passed to all the controller methods - but it is nearly never used in the body of these methods. In the code samples in the whole Catalyst::Manual distribution $self is assigned to 117 times - but is used only 38 times (and not all code samples are controllers there). This sounds like a strange case for object oriented code which bases it's definition on the usage of the $self object.

There is much discussion about the stash and how to replace it. The main problem with stash is that it is global - there is no use in replacing it with another global thing. Just like global variables it needs to be replaced by many specialized local things like parameter passing and attributes, the controller object attributes, and for that to be entirely clean we need the a new controller object per response.

HTML::FormHandler - a few notes

This is not an introduction to HTML::FormHandler - the docs there are very good, but I wanted to write down the main ideas I had when contributing to the design of it.

Let's look at the functions of a form processor:

1. processing the input into internal representation
   
2. checking the values
   
3. saving the internal representation into the persistence layer and loading values from it
   
4. rendering the form (with input, error messages, etc)
   

The 'best practice' of rendering the HTML view of an application page is to do it in the templates. Unfortunately there is just too much logic needed for the correct rendering of forms (mostly with the display of errors) to leave that for the templates mini language (like that of TT) - that's why this needs to be done by the form processor. But because of the diversity of the rendering requirements it also needs to be easily replaceable, ideally part by part. Then the application could start with a crude but working HTML rendering, introduced with minimal effort from the programmer since it is built into the processor and later gradually refined and replaced.

CGI.pm and after it Catalyst and other libraries parse the url-encoded and body parameters into a hash of scalar values but internally we would like to manipulate objects like DateTime instead of lists of values for year, month and day. In HFH this conversion is done in two phases - first the input is turn into a deeper structure built of hashes and arrays - then then the processor goes recursively from the leaves and at each node applies a conversion if it is defined for that node.

There is always a question when we should do the checks - sometimes it is easier do do them on the input data sometimes on the internal values. A good example is DateTime - we can check if the day field is a number between 1 and 31, but we also need to check if the whole date is correct and the business rules could add another requirement that the day of the week is Monday or some other operation that is most convenient on the DateTime object. In FormHandler all of this is possible:



How this works? As explained above the processor starts with the leave nodes - and checks the day, month and year fields (here they don't define any transformations only checks, but they could). Then it goes to the parent node date_time and there it has the hash of values from the leave nodes ready for the next transformation and checks - it is passed to:

sub { DateTime->new( shift ) }

If that subroutine triggers an exception, for example when someone passes

{ year => 2009, month => 2, day => 30 }

to it - then the error is recorded with the message This is not a correct date. If it works - the value returned from it, a DateTime object, is then passed to the next check, and later presumably saved to the database. The checks and transformations are taken from a single list - so you can interweave them as you need.
Instead of transform and check in the apply attribute we can also use the Moose types and coercions and also we can mix these two styles (for example coerce some value using Moose types and then check it again). If someone needs even more power - he can also write his own field class with it's own validation method - and do both validation and transformations in there, that method would receive the same hash of child values.

To save the full data structure of values that can contain many inter-related records HTML::FormHandler::Model::DBIC uses
DBIx::Class::ResultSet::RecursiveUpdate. I separated it so that HFH is not related to the hacks I made there :) Or maybe I hoped that perhaps other form processors could reuse my module. Using it you can update a company record together with all it's related addresses in one form.