Technically right, socially not so

Imagine that you let anyone use your hard work by publishing your code as Open Source and then instead of thank you notes you receive angry emails about purported bugs in it.

Imagine that you noticed a problem about some Open Source code and as a good citizen you analyze it and report your findings and instead of a thank you note you receive a reply explaining how much confused you are for thinking that that could be a bug.

Imagine that this happens in a public space like an mailing list, open bug tracker, etc. This is like the setting of a Greek tragedy - both sides have their own rights and values, both sides can be technically right - and yet the conflict spreads and escalates. People copy the emotions of each other, even perceived emotions and especially anger, when displayed publicly leads to more anger. This is a vicious circle of positive feedback loop. When it starts everyone gets irritated and everyone wants to know who started it, who is responsible for that.

But go back to the start - it's not anyone's fault - you cannot point the finger on one person and say 'he is responsible'. Looking for the main culprit is really looking for a scapegoat - a surrogate victim that would heal everyone's consciousness. This is not especially geek thing overall - it is a universal trait - but sadly inventing fodder for the scapegoat search is a prominent part of geek culture.

One of the many things that I admire about Larry Wall is how he can neutralize the poisons of angry reactions. People scorn Perl for being a 'scripting' language - he talks about Ada Lovelace:

Suppose you went back to Ada Lovelace and asked her the difference between a script and a program. She'd probably look at you funny, then say something like: Well, a script is what you give the actors, but a program is what you give the audience. That Ada was one sharp lady...

Installing dependencies - continued

I don't know what I am doing wrong - but after installing Perl 5.12.1 under perlbrew "cpan -m ." stopped to install the dependencies. It seems that for making and testing it adds the libs in '.cpan/build' to the PERL5LIB search path so both can go on without really installing the dependencies:

Running make for /home/zby/progs/ravlog/.
Prepending blib/arch and blib/lib of 108 build dirs to PERL5LIB; for 'get'

"cpan -t . " works in the same way and if the tests fail then "cpan ." stops there - so I cannot tell if it would install the dependencies in the case when the tests pass.

I don't really know how to investigate it further - but I am noting that as it is very frustrating. Eventually I had to resort to cpanminus ("cpanm --installdeps ." is really handy as it addresses the problem directly).

Implementing file upload progress bar

At work I was assigned the task of replacing the previous hodgepodge of tools that provide the progress bar functionality for forms with file uploads. At first glance this seems like a trivial thing to do - you periodically observe how much of the file you've got and update the progress bar, but there are details that make it hard to do with the current tools. I report here how I solved that problem - not because I think this is the optimal way - but rather to open the discussion and because I could not find any description of solving it when I googled for it.

First of all the common programming tools (like CGI.pm or Mason that we use here) assume that the page handler receives the whole request as input - and that whole request is not available until after the file is uploaded. So for example 'my $q = CGI.pm->new' will not finish until it is too late to measure the upload progress. The solution to that is to use another page to report the upload progress and call that page via Ajax from Javascript code updating the progress bar. This would work great - but the file is normally uploaded to a temporary file with a random name and the other script would not have any chance to guess it. We need to generate a new random file name in the form page and then pass that name to the form handler script so that it would save the data to that file, and in parallel to the Ajax scripts that would check the size of that file.


To save the data into a specified filename I used the CGI.pm callback feature:

my $q = CGI->new( \&hook, $fh, undef );
...
sub hook {
my ($filename, $buffer, $bytes_read, $fh) = @_;
print $fh substr($buffer, 0, $bytes_read);
$fh->flush();
}

It is described in the subsection called "Progress bars for file uploads and avoiding temp files" of the CGI.pm documentaion, but actually it is a great leap of thought to say that it supports progress bar implementation, you still cannot use it directly to get the progress bar from the CGI object on the form landing page, you still need the separate scripts measuring the progress. For my solution all I needed was to pass the target file name to the code saving the data, this could be easier than writing this callback above. And the callback is still not everything - I yet need a way to pass the generated filename from the form page to that script - and not via form parameters, remember they are not available at that stage. So how can that be done? Simple - as PATH_INFO - which is available in the %ENV hash even before the params are parsed by CGI.pm.

This is the skeleton of the solution - there are a few more details in the actual implementation - but the code will be published soon as Open Source - so I hope everyone will be able to look them up there.

When those micro-seconds matter

Catalyst is the only web framework that has a mailing list I subscribe to - but I am sure that it happens at others too. In a recurring pattern someone someone posts a benchmark showing that Catalyst for some trivial operation is many times (or many hundred times) slower than some other web framework or for that matter PHP. That does not fail to generate a heated debate - but eventually the seasoned framework developers gain the upper hand with the argument that for all the, often big, web sites they worked on, those few micro-seconds lost in the Catalyst dispatcher never mattered much because the application spent hundred times more in other code fragments and mostly in business logic parts, so shaving off some part of the few micro-seconds would not improve the overall speed more than 1%. This is a great argument, perfectly reasonable and rational but it is biased towards the status quo. It might be true that everywhere where Catalyst is currently used it works great but it is also not hard to imagine an application with very simple business logic that needs to serve millions of users, Twitter anyone? Or an app that does many simple Ajax callbacks. Sure you can always code the speed requiring, simple parts in PHP and keep Catalyst only for the other more heavy-weight tasks but having a universal solution would be so much more convenient.

Installing dependencies for your packaged for CPAN library

Update: See the comments, apparently the warning against auto_install is outdated information. I don't have any opinion of myself here.

It always puzzled me what is the 'canonical' way to install dependencies for an unpacked distribution, be it downloaded from CPAN or your own in-house product packaged the CPAN way for convenience. Module::Install provides an 'auto_install' option that you add to your Makefile.PL and then perl Makefile.PL; make (or perl Makefile.PL; sudo make) should do that trick. This seemed like a good solution that could be a standard, but now I see that the FAQ of that module says:

Do I really have to avoid auto_install()?

auto_install() has a long history of breaking CPAN toolchains. Lots of people had a bad feeling on it, and have said it should be strongly avoided. In fact it was deprecated and removed once.

Although most of the known problems have been fixed and you can use it more safely than ever, the use of auto_install() is still discouraged, especially if what you are writing is a module to be uploaded on the CPAN. auto_install() does lots of things itself, thus does not always do the same things as other toolchains do (including extra attribute handling, etc; which can be fixed somehow but that's not too DRY). It only supports CPAN and CPANPLUS as its backends. If you use other tools to install, it may still cause a trouble.

Besides, now you can do what auto_install() does with other means. If your CPAN module is new enough, you can pass a dot to the cpan command it provides, and it will install all the required distributions from the CPAN:

$ cpan .

So apparently this is the way to go - using the cpan shell. There is a minor problem with this when used for testing - it would automatically install the package from the directory you are in. Sure you can suppress that you need by adding another command switch (-m), but it would be better if the default action was the safe one.

Templates and inheritance

In every web site project I have been working on there inevitably came that moment that someone wanted to have a copy of the site with some small changes, or that someone wanted to use a piece the website in something else. In both cases it was all about (initially) some small changes, what they call as skin or a branded site. With programming part we know what to do then - we design base classes for the core undifferentiated functionality and inherit from them in the targeted classes. Most templating engines do have some notion of search path - so this should be doable with templates - but it always proves to be much more messy than it initially seems to be. I'll try to catalogue here all the use cases and how they compare to the standard inheritance in object oriented programming languages and describe them in terms of operations on the search path.

Initial setup

In 'normal' programming languages the atomic piece that can be inherited and overloaded and that is searched on the inheritance path is a method - in case of templates the atomic part is a whole template. The granularity is defined differently, but we can do analogues things with both. Here I will concentrate on the dispatching model familiar for users of most of the MVC frameworks with controllers and their methods as the endpoints (called actions). In this setting all the methods could have a matching template file and inheriting and overriding a template could work on the same level as inheriting and overriding an action.

Basic case

Lets say I have a controller 'MyApp::Controller::Book' with a few actions like 'view', 'list', 'edit' (a CRUD controller) and matching templates in 'templates/Book'. I would like that when calling the render from that controller I did not need to specify the full path of the templates - but just call them by name 'view'. Also when they have a common subcomponents in the same directory they should be able to [% INCLUDE date_formatting %] them by name. This is just like calling methods in a class and it can be accomplished by adding the 'templates/Book' directory at the start of the search path.

Library component

Sometimes we would like to use ready made components from some published library, maybe a CRUD controller for our Books. We inherit from the library class in our 'MyApp::Controller::Book' and we would like its templates to be inherited in a similar way. That means adding '/usr/lib/something/something/CRUD/templates' to our search path somewhere after 'templates/Book'.

Subclassing our own controllers

If our 'MyApp::Controller::Book' is a subclass of our own ''MyApp::Controller::Item' class - this is a very similar case to the above. Here as well we want to insert the search path of the parent class somewhere after 'templates/Book. But in this case what we insert is 'template/Item' - a path relative to our application root.

Application wide components

This is solved simply by adding an application wide element in the search path. It would be analogues to adding stuff to a common base class in standard programming - much more rare there.

Subclassing a whole site

This is the case of 'branded' sites where we would like to subclass everything at once. It is maybe the most common case - but also the most difficult because there are no good analogies/solutions in standard object oriented programming. If 'MyNewApp' inherits from 'MyApp' that does not automatically mean that 'MyNewApp::Controller::Book' springs to existence and can serve book information in the new app (but I've heard there are existing Object Oriented languages that work like that and some people are thinking about implementing this in Perl). So I think we need to first think over how this is handled in the main framework and then we can deduce something about how to treat templates from that. For the limited case of application wide components this can be solved by adding the new template root to the beginning of the search path - but it gets more complicated for all the other cases described above.

Conclusions

I mostly wrote that to have it all in one place - but I think it does show the need to make the template search path dependent on the controller used.

A subject for mst's talk

I did not really intended to vote in that silly poll - but I think I have found a subject for the talk that could be highly beneficial for the Perl community: How using the insights from cognitive science (and what else is needed) I am going to make HTML::Zoom actually usable for mere mortals.

PSGI and Object Oriented Programming

There is one thing that seems pretty simple - but it took me a bit of mulling over to fully understand

A PSGI application is a Perl code reference.

It is a reference to a sub that expects a hash ref representing the HTTP request passed to it as the one and only parameter and which returns an array ref representing a HTTP response. Hmm - yeah that's what web apps do - take the request and return the response.

But then you think - OK, but I like Object Oriented code and I want my application to be an object with attributes and stuff - so what can I do? Well - that is simple. Have your app as object as you like, and to the PSGI layer pass a closure referring to that object. For example if you application object normally handles the requests with a ->handle method then

would return a suitable callback. Which you can then use in your app.psgi file:

And this is exactly what I do in WebNano.

Web Applications and Dispatching

Historically the operation of finding the code to handle the web request was trivial - the server just matched the path part of the web request into the filesystem to find the CGI program producing the requested page. But then it got more complicated when we moved from the CGI 'one program equals one page' way into the realm of web applications that serve many pages.

The tree (or DAG) data structure is the underlying concept of both the path part of URL and many programming language library (or class) structures - so there is a natural way of mapping between them. To have code handling requests to related pages in the same file we can extend it so that the last part of the path is interpreted as a subroutine name.

That idea gets a bit more complicated when we realize that we don't want to expose all of our often internal methods (and libraries/classes) to be callable from outside - we need a way to mark them as 'external' and expose only these.

Of course sometimes there are reasons that we don't want to use such a literal mapping, it can be some security (by obscurity) need to hide internal code structures, need for more elegant URLs or other requirements. So sometimes we need to extend it or completely replace it with something else.

The Perl web frameworks (that I know the best) in respect to usage of that mapping can be roughly divided into following styles (many use more than one style so the boundaries are fuzzy):

1. The 'old' CGI (or PHP) paradigm of one address one program
   
   
2. Using the mapping of paths to libraries, selecting 'externally callable' methods by placing them on a list ( run_modes in CGI::Aplication ).
   
   
3. Using the mapping of paths to libraries, but dispatching to methods configured via a hash (also run_modes in CGI::Application).
   
   
4. Using the mapping of paths to libraries (but not to methods) and calling the 'get' or 'post' method in the landing library (like in Tatsumaki).
   
   
5. Configuring the dispatching by code attributes of the methods (like in Catalyst).
   
   
6. Not using methods but anonymous subroutines. This way it is easy to assign the dispatching configuration to the code by using DSLs (like in Dancer).
   
   
7. Dispatching configuration external to the classes that are dispatched to.
   

Having the dispatching configuration close to the subroutine code let's you avoid switching between two places in the source code when adding a new action or changing an existing one. I think one of the greatest conveniences of Catalyst was that that configuration was not only in the same file but directly in the subroutine definition.

WebNano - what I did not say at the London.pm tech meeting

About a month ago I had a presentation at the London.pm tech meeting about the design ideas I plan to experiment in WebNano. There are a few things that I did not have time to tell - and I plan to blog about them. First of all I think I should have admitted that I stole the both the name and the idea from Simon Cozen's MicroMaypole (I hope he will treat that in the terms of the well known quote about imitation and not intellectual property trespassing:).

At the presentation I mentioned that there is not much code in WebNano currently - but maybe this was not the best way to put it. I actually don't plan to ever have much code in it and the example of MicroMaypole shows that perhaps such a minimal web framework can be a practical thing, but I do plan to build some more examples using that minimal code so that I can be sure that even if minimal it does have all the desired features. What are those features will be the subject of other blog posts soon.

Frameworks are framing, libraries are liberating

It seems that the season is open and everyone is writing his new web framework again (all authors of the dozens of other recently created Perl web framework - please accept my apologies - I could not possibly link to all of your babies). If you ask me why is that? My answer is: it is easy now. We have libraries that do nearly everything that the traditional web frameworks provide. You don't need to write much - you just saw together the available patches and voila: your own web framework. You can be proud.

For some time I was thinking: OK, so now we have that cambrian explosion of new frameworks, that's OK, it will all settle down, people will select the best of them and the rest will die off. All we need to do is waiting. We could also speed it up a bit and make it a bit more smooth and rational by learning how to evaluate the frameworks. To do that we need to talk about them, study their functions and compare designs, project goals, project management styles and all of that. This will not be easy, but once we get past the awsome depth of common analyses of the 'this sucks' style and start to think about 'this is good for that and that other thing is better if you need something else', the rest will be incremental.

And I still believe that we need to talk more about design solutions and expand our vocabulary and use less of 'this is stupid' authoritative evaluations. This is something beneficial in any circumstances. But now I also think that since we have the libraries that do the all the work that previously was done by frameworks - then why should we build any more frameworks? Frameworks are coupling, they tie together the libraries that do the actual work and send that loose, uncohesive package to the programmer, who he has no choice, but has to use all of it and in the way prescribed by the framework. But every educated programmer knows that coupling is bad and cohesion is good. So why don't we just use the available nicely decoupled and internally cohesive libraries instead? Yeah, the libraries don't yet do all the 100% of stuff traditionally provided by frameworks, they can do 80% or even 90% of it but not yet the whole 100%, it is still ahead of us to extract the other 10% of functionality from the existing frameworks and turn it into a nicely decoupled and internally cohesive libraries.

Historically the trend is clear - the first frameworks did everything, the MVC frameworks included all Model, View and Controller components, then came Catalyst which is mostly just the Controller part. But that was not the end of decoupling - the Controller part still can be broken down further. And it was, with libraries like HTML::Engine budding off it, splitting and heading for an independent CPAN life. Now it is time to have a look at Catalyst and other frameworks and think what are the exclusive features that it provides that are not yet available as independent CPAN libraries. We have Plack, that provides all the abstraction needed for easy deployment and testing, we have packages like Bread::Board that provide a generic way of initialising application components, and I am sure there are also other interesting libraries at CPAN that provide other parts of the framework functionality. So what is left?

Maybe I am missing something, but the only important functionality of Catalyst that I cannot find any generic libs for is dispatching - selecting the subroutine (or method) that will handle a given HTTP request. I don't see any reasons why this should be hard to do. In fact I have already started working on that. OK - to be frank, what I started to work on was another web framework. I decided that I would like to have a mini framework and started coding it. Only now, after having the first CRUD example written in it, I can see that all I wrote is just a dispatch system, the other parts are freely available at CPAN. I can use Plack for the web engine abstraction and Plack::Test for testing, I can use Bread::Board for application components creation and Moose as the object framework - no need to reinvent the wheel.

MST explains the ideas lying behind his 'being a bastard'

It always seemed a bit weird when I saw mst kick himself out of an IRC channel - now we have it all nicely explained. Next time we'll have a self tickling machine, I am sure.

Bread::Board

Lots of packages in the Catalyst namespace are simple adaptors of external libraries. In many cases what you need from that adaptor layer is only passing initialisation values from the config file to the appropriate call to the object creator. By using the Dependency Injection pattern you can move that initialisation code out of that "deep in the framework guts" place into something that can be called another config file - containing values that are complex objects instead of simple constants. This way your view object can be constructed there and then injected into the application which does not need to know how to construct it. You still might prefer to have a different config file with simple literal configuration values using the standard YAML or Config::General syntax.

If that was too vague for you - don't worry - Stevan promised some more docs for Bread::Board soon. The real take away from all of this is that Bread::Board and future sub-classes of it spares you from the need to writing adaptor classes for all the view or model classes of your yet another web framework. Maybe you'll not need a framework at all - and assemble your application using Bread::Board out of a web dispatcher and other ready made parts.

Frameworks are not yet well refactored libraries.

At first we had Catalyst. After some refactoring we got HTTP::Engine and now we have PSGI/Plack and the tools around that abstraction that provide a lot of functionality that previously required using Catalyst. Now all of that is in libraries that can be used independently from each other, there is no coupling between the components other than the well defined and pretty minimal PSGI interface.

The title of this blog post might not be universally true, it is possible that there are some kernel parts in some frameworks or even most of the frameworks that could never be in any practical way reduced into sets of libraries. But what I observe is that frameworks tend to grow and only in very exceptional cases spawn libraries that are refactored out of their main body. It is just so much easier to just write new extensions, plugins or things that use the whole framework than find the right abstraction that would let to split some functionality out of the main code base. Certainly I would never guess that the simple PSGI abstraction would let people refactor so much into separate libraries, I bow to it's author. But we should always try:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Antoine de Saint-Exupery

Catalyst app/context split

This week I've received a bounty from the Enlightened Perl Organisation for my work on the app/context split. As I understand they have doubled the private bounty from groditi who was too busy to keep his promise to do that job and wanted someone to free him from the shame of broken promises. Anyway I am very happy with the outcome. I am not sure if this can be a stable way of founding work on Open Source projects - but it this time it worked :)

Scope and Immutability

I am sold on the immutability thing. I started to think about various designs and about what immutability of objects would mean to them and I think the results are promising - they tend to have much more clear structure - and just thinking about immutability gives you a chance to see an additional structure in the code. One quite obvious rule of such design is that you cannot keep data of narrower scope in objects with wider scope - and this forces you to think about the scope of various data pieces in your programs. For example in web oriented programming you have (usually) three scopes:

• Application - i.e. all data that is persistent to the whole web server process (like configuration)
  
• Session - i.e. data saved into the session variables (there is an opinion, and I mostly subscribe to it, that you should keep that minimal).
  
• Request - i.e. the data you receive from the web client and that you send back to it
  

An example of framework that does not respect this division is known to everyone - it's Catalyst - fortunately the app/context split is mostly done.

Acknowledgements: Stevan thanks for the web apps scope break down :)

External dependencies once again

My proposal to add a 'lib_requires' keyword to the META file - with identical semantics as Devel::CheckLib did not make at this round of META spec changes. Now I think that maybe a better strategy would be to start with adding this to the documentation. I am sure many packages already do that - but I have encountered others that don't. What I have in mind is a standard POD section that would be generated by all the module scaffolding generators (like Module::Starter) and which would contain all the information needed to install the external lib in common environments. After this becomes a common thing we would be able to make more informed decisions about including it into the META file.

Warszawa.pm - Biblioteczka (The Library)

W czwartek odbyło się spotkanie Warszawa.pm. Warta zanotowania jest inauguracja naszej biblioteczki - pożyczyłem od Piotrka Czynnik Ludzki. Dzięki!


Last Thursday there was a Warszawa.pm meeting. Worth noting is the inauguration of our library - Piotrek lended Peopleware: Productive Projects and Teams to me. Thanks!

What are the features that Perl shares with Lisp?

The author of the famous Higher-Order Perl book (it's available for free download - so no excuse for those that have not yet read it) writes:

Perl is much more like Lisp than it is like C. If you pick up a good
book about Lisp, there will be a section that describes Lisp’s good features. For example, the book Paradigms of Artificial Intelligence Programming, by Peter Norvig, includes a section titled What Makes Lisp Different? that describes seven features of Lisp. Perl shares six of these features; C shares none of them. These are big, important features, features like first-class functions, dynamic access to the symbol table, and automatic storage management.

Dear lazy web: what are the other three features?

On naming things

One benefit that I have hope will be the result of the IronMan blogging challenge is that Perl people will get more expressive in describing their design. What I can see now is that often there is a strong belief that some code has a good or bad design but any inquiry into why said design is good or bad ends in a 'because you are stupid and you don't understand' argument. While I agree that coding can be sometimes more productive than discussing, I think this rule should not be treated absolutely. All design is trading one thing for another one, having this trade-off stated explicitly gives us the power to weight it intelligently and appropriately to circumstances. In contrast an unnamed intuition can be only an article of our faith in the designers skills.

There is power in naming things as the popularity of Design Patterns shows.