Web Applications and Dispatching

Historically the operation of finding the code to handle the web request was trivial - the server just matched the path part of the web request into the filesystem to find the CGI program producing the requested page. But then it got more complicated when we moved from the CGI 'one program equals one page' way into the realm of web applications that serve many pages.

The tree (or DAG) data structure is the underlying concept of both the path part of URL and many programming language library (or class) structures - so there is a natural way of mapping between them. To have code handling requests to related pages in the same file we can extend it so that the last part of the path is interpreted as a subroutine name.

That idea gets a bit more complicated when we realize that we don't want to expose all of our often internal methods (and libraries/classes) to be callable from outside - we need a way to mark them as 'external' and expose only these.

Of course sometimes there are reasons that we don't want to use such a literal mapping, it can be some security (by obscurity) need to hide internal code structures, need for more elegant URLs or other requirements. So sometimes we need to extend it or completely replace it with something else.

The Perl web frameworks (that I know the best) in respect to usage of that mapping can be roughly divided into following styles (many use more than one style so the boundaries are fuzzy):

1. The 'old' CGI (or PHP) paradigm of one address one program
   
   
2. Using the mapping of paths to libraries, selecting 'externally callable' methods by placing them on a list ( run_modes in CGI::Aplication ).
   
   
3. Using the mapping of paths to libraries, but dispatching to methods configured via a hash (also run_modes in CGI::Application).
   
   
4. Using the mapping of paths to libraries (but not to methods) and calling the 'get' or 'post' method in the landing library (like in Tatsumaki).
   
   
5. Configuring the dispatching by code attributes of the methods (like in Catalyst).
   
   
6. Not using methods but anonymous subroutines. This way it is easy to assign the dispatching configuration to the code by using DSLs (like in Dancer).
   
   
7. Dispatching configuration external to the classes that are dispatched to.
   

Having the dispatching configuration close to the subroutine code let's you avoid switching between two places in the source code when adding a new action or changing an existing one. I think one of the greatest conveniences of Catalyst was that that configuration was not only in the same file but directly in the subroutine definition.

WebNano - what I did not say at the London.pm tech meeting

About a month ago I had a presentation at the London.pm tech meeting about the design ideas I plan to experiment in WebNano. There are a few things that I did not have time to tell - and I plan to blog about them. First of all I think I should have admitted that I stole the both the name and the idea from Simon Cozen's MicroMaypole (I hope he will treat that in the terms of the well known quote about imitation and not intellectual property trespassing:).

At the presentation I mentioned that there is not much code in WebNano currently - but maybe this was not the best way to put it. I actually don't plan to ever have much code in it and the example of MicroMaypole shows that perhaps such a minimal web framework can be a practical thing, but I do plan to build some more examples using that minimal code so that I can be sure that even if minimal it does have all the desired features. What are those features will be the subject of other blog posts soon.

Frameworks are framing, libraries are liberating

It seems that the season is open and everyone is writing his new web framework again (all authors of the dozens of other recently created Perl web framework - please accept my apologies - I could not possibly link to all of your babies). If you ask me why is that? My answer is: it is easy now. We have libraries that do nearly everything that the traditional web frameworks provide. You don't need to write much - you just saw together the available patches and voila: your own web framework. You can be proud.

For some time I was thinking: OK, so now we have that cambrian explosion of new frameworks, that's OK, it will all settle down, people will select the best of them and the rest will die off. All we need to do is waiting. We could also speed it up a bit and make it a bit more smooth and rational by learning how to evaluate the frameworks. To do that we need to talk about them, study their functions and compare designs, project goals, project management styles and all of that. This will not be easy, but once we get past the awsome depth of common analyses of the 'this sucks' style and start to think about 'this is good for that and that other thing is better if you need something else', the rest will be incremental.

And I still believe that we need to talk more about design solutions and expand our vocabulary and use less of 'this is stupid' authoritative evaluations. This is something beneficial in any circumstances. But now I also think that since we have the libraries that do the all the work that previously was done by frameworks - then why should we build any more frameworks? Frameworks are coupling, they tie together the libraries that do the actual work and send that loose, uncohesive package to the programmer, who he has no choice, but has to use all of it and in the way prescribed by the framework. But every educated programmer knows that coupling is bad and cohesion is good. So why don't we just use the available nicely decoupled and internally cohesive libraries instead? Yeah, the libraries don't yet do all the 100% of stuff traditionally provided by frameworks, they can do 80% or even 90% of it but not yet the whole 100%, it is still ahead of us to extract the other 10% of functionality from the existing frameworks and turn it into a nicely decoupled and internally cohesive libraries.

Historically the trend is clear - the first frameworks did everything, the MVC frameworks included all Model, View and Controller components, then came Catalyst which is mostly just the Controller part. But that was not the end of decoupling - the Controller part still can be broken down further. And it was, with libraries like HTML::Engine budding off it, splitting and heading for an independent CPAN life. Now it is time to have a look at Catalyst and other frameworks and think what are the exclusive features that it provides that are not yet available as independent CPAN libraries. We have Plack, that provides all the abstraction needed for easy deployment and testing, we have packages like Bread::Board that provide a generic way of initialising application components, and I am sure there are also other interesting libraries at CPAN that provide other parts of the framework functionality. So what is left?

Maybe I am missing something, but the only important functionality of Catalyst that I cannot find any generic libs for is dispatching - selecting the subroutine (or method) that will handle a given HTTP request. I don't see any reasons why this should be hard to do. In fact I have already started working on that. OK - to be frank, what I started to work on was another web framework. I decided that I would like to have a mini framework and started coding it. Only now, after having the first CRUD example written in it, I can see that all I wrote is just a dispatch system, the other parts are freely available at CPAN. I can use Plack for the web engine abstraction and Plack::Test for testing, I can use Bread::Board for application components creation and Moose as the object framework - no need to reinvent the wheel.

MST explains the ideas lying behind his 'being a bastard'

It always seemed a bit weird when I saw mst kick himself out of an IRC channel - now we have it all nicely explained. Next time we'll have a self tickling machine, I am sure.

Bread::Board

Lots of packages in the Catalyst namespace are simple adaptors of external libraries. In many cases what you need from that adaptor layer is only passing initialisation values from the config file to the appropriate call to the object creator. By using the Dependency Injection pattern you can move that initialisation code out of that "deep in the framework guts" place into something that can be called another config file - containing values that are complex objects instead of simple constants. This way your view object can be constructed there and then injected into the application which does not need to know how to construct it. You still might prefer to have a different config file with simple literal configuration values using the standard YAML or Config::General syntax.

If that was too vague for you - don't worry - Stevan promised some more docs for Bread::Board soon. The real take away from all of this is that Bread::Board and future sub-classes of it spares you from the need to writing adaptor classes for all the view or model classes of your yet another web framework. Maybe you'll not need a framework at all - and assemble your application using Bread::Board out of a web dispatcher and other ready made parts.

Frameworks are not yet well refactored libraries.

At first we had Catalyst. After some refactoring we got HTTP::Engine and now we have PSGI/Plack and the tools around that abstraction that provide a lot of functionality that previously required using Catalyst. Now all of that is in libraries that can be used independently from each other, there is no coupling between the components other than the well defined and pretty minimal PSGI interface.

The title of this blog post might not be universally true, it is possible that there are some kernel parts in some frameworks or even most of the frameworks that could never be in any practical way reduced into sets of libraries. But what I observe is that frameworks tend to grow and only in very exceptional cases spawn libraries that are refactored out of their main body. It is just so much easier to just write new extensions, plugins or things that use the whole framework than find the right abstraction that would let to split some functionality out of the main code base. Certainly I would never guess that the simple PSGI abstraction would let people refactor so much into separate libraries, I bow to it's author. But we should always try:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Antoine de Saint-Exupery

Catalyst app/context split

This week I've received a bounty from the Enlightened Perl Organisation for my work on the app/context split. As I understand they have doubled the private bounty from groditi who was too busy to keep his promise to do that job and wanted someone to free him from the shame of broken promises. Anyway I am very happy with the outcome. I am not sure if this can be a stable way of founding work on Open Source projects - but it this time it worked :)

Scope and Immutability

I am sold on the immutability thing. I started to think about various designs and about what immutability of objects would mean to them and I think the results are promising - they tend to have much more clear structure - and just thinking about immutability gives you a chance to see an additional structure in the code. One quite obvious rule of such design is that you cannot keep data of narrower scope in objects with wider scope - and this forces you to think about the scope of various data pieces in your programs. For example in web oriented programming you have (usually) three scopes:

• Application - i.e. all data that is persistent to the whole web server process (like configuration)
  
• Session - i.e. data saved into the session variables (there is an opinion, and I mostly subscribe to it, that you should keep that minimal).
  
• Request - i.e. the data you receive from the web client and that you send back to it
  

An example of framework that does not respect this division is known to everyone - it's Catalyst - fortunately the app/context split is mostly done.

Acknowledgements: Stevan thanks for the web apps scope break down :)

External dependencies once again

My proposal to add a 'lib_requires' keyword to the META file - with identical semantics as Devel::CheckLib did not make at this round of META spec changes. Now I think that maybe a better strategy would be to start with adding this to the documentation. I am sure many packages already do that - but I have encountered others that don't. What I have in mind is a standard POD section that would be generated by all the module scaffolding generators (like Module::Starter) and which would contain all the information needed to install the external lib in common environments. After this becomes a common thing we would be able to make more informed decisions about including it into the META file.

Warszawa.pm - Biblioteczka (The Library)

W czwartek odbyło się spotkanie Warszawa.pm. Warta zanotowania jest inauguracja naszej biblioteczki - pożyczyłem od Piotrka Czynnik Ludzki. Dzięki!


Last Thursday there was a Warszawa.pm meeting. Worth noting is the inauguration of our library - Piotrek lended Peopleware: Productive Projects and Teams to me. Thanks!

What are the features that Perl shares with Lisp?

The author of the famous Higher-Order Perl book (it's available for free download - so no excuse for those that have not yet read it) writes:

Perl is much more like Lisp than it is like C. If you pick up a good
book about Lisp, there will be a section that describes Lisp’s good features. For example, the book Paradigms of Artificial Intelligence Programming, by Peter Norvig, includes a section titled What Makes Lisp Different? that describes seven features of Lisp. Perl shares six of these features; C shares none of them. These are big, important features, features like first-class functions, dynamic access to the symbol table, and automatic storage management.

Dear lazy web: what are the other three features?

On naming things

One benefit that I have hope will be the result of the IronMan blogging challenge is that Perl people will get more expressive in describing their design. What I can see now is that often there is a strong belief that some code has a good or bad design but any inquiry into why said design is good or bad ends in a 'because you are stupid and you don't understand' argument. While I agree that coding can be sometimes more productive than discussing, I think this rule should not be treated absolutely. All design is trading one thing for another one, having this trade-off stated explicitly gives us the power to weight it intelligently and appropriately to circumstances. In contrast an unnamed intuition can be only an article of our faith in the designers skills.

There is power in naming things as the popularity of Design Patterns shows.

Stuff

Only remotely related to Perl.

Socio-Technical Systems

I like the definition from The Social Requirements of Technical Systems, now I have what CPAN realy is. I have not yet read the whole Handbook of Research on Socio-Technical Design and Social Networking Systems but the rest seems rather dissapointing.

von Hippel

I recently discovered that most of his work is available on-line: Collected Papers, Downloadable Books. I recommend it to people trying to understand Open Source. He is much more 'technical' than Yochai Benkler. I've just finished reading Open Source Software and the 'Private-Collective' Innovation Model.

Rene Girard

I am working on a paper about on-line conflict in the light of mimetic theory.

Role that adds Traits to multiple classes

The DBIC model stuff naturally spans the whole FormHandler class tree - there are things that need to be added to the main form class, but also to the Field classes. An obvious example of the latter is the lookup_options method which loads data from the db table (ResultSet object more precisely) appropriate for the field into the field's options array. It does not really touch any global form stuff and:

$field->form->lookup_options( $field, ... )

is like reaching with your left hand to your right pocket. I mean really awkward. So what we need is a form role that would add other roles to the fields classes. I imagined that this could be possible, in theory a role could be a hierarchy that would be applied to a class hierarchy - but I don't think you can do that with current Moose, so instead I used a hook in the method that builds fields and added there code to inject an appropriate Trait.

The code is in new branches in the main HTML::FormHandler repo and in the HTML::FormHandler::Model::DBIC repo. I am waiting for commetns. I would think that this is not something unusual - so I guess others must have already solved similar problems before - I would like to hear what they did.

We have a similar problem with rendering. It is a role now so that it can be easily replaced, but it is naturally recursive, fields should know how to render themselves so we need a way to apply a whole set of roles in one sweep to the whole form class structure. It looks like there really is some abstract super-role in there.
How much is it monkey patching, how much is it action at a distance?

Controller immutability

On IRC I've got an answer from t0m and others about why they would not use controller attributes for storing data related to the current request - they want to keep their controller object immutable for the request phase. The idea is that if you have an object that is being reused then it would be nice if after each use of that object it is still in the same clean state. That is it is nice to always have the changes contained in a minimal area.

I'll leave it to the experts to list all the advantages of immutable data structures, but I've seen why we needed a similar compartmentalisation in HTML::FormHandler. The FormHandler form object is reusable, you can use it to validate many parameter hashes. We started coding it as a mutable object and storing all the changing values in the form object itself, the first problem was that we kept forgetting to reset them at the start of processing of new data. This lead to many subtle bugs, sometimes difficult to track so we started to just recreate the form object for each processing run. This worked OK, because the form object itself was not that heavy to create until someone had a form object that contained other heavy object and recreating it would require recreating them. So eventually we separated all the state that changes during the processing into a separate Result object, now to clean the form state we need to recreate only that separate object.

It needs to be stated clearly that this separation of immutability is a trade-off (as always) - it is an additional requirement so it increases the complexity of the code and means for example that the $self object in controller actions is underused - because actions should not change it's state (including storing something in it's attributes). Certainly it is not feasible to make such change now and this also would be a trade-off but for the sake of exercising imagination: maybe actions could be methods on the changing part (and not on the controller object) and only use the controller object in some way - this way all the basic code (like that from the Manual) would use $self object in the way common to object oriented code.

Catalyst::Component::InstancePerContext

How often do you use the controller object? It is the first, most important parameter passed to all the controller methods - but it is nearly never used in the body of these methods. In the code samples in the whole Catalyst::Manual distribution $self is assigned to 117 times - but is used only 38 times (and not all code samples are controllers there). This sounds like a strange case for object oriented code which bases it's definition on the usage of the $self object.

There is much discussion about the stash and how to replace it. The main problem with stash is that it is global - there is no use in replacing it with another global thing. Just like global variables it needs to be replaced by many specialized local things like parameter passing and attributes, the controller object attributes, and for that to be entirely clean we need the a new controller object per response.

OpenID support in LoginSimple

This week I've added some OpenID support to LoginSimple. Please clone the git repository and try it out! The hard part of the whole LoginSimple idea is to make it universal enough. We need people to try to apply it in many different scenarios and give us some feedback.

The test application is in t/lib/TestAppOpenID.pm, it has some tests in t/07-openid-live.t, but you can also run it with perl -Ilib t/lib/script/testappopenid_server.pl and then go to the http://localhost:3000/login page and test it manually.

HTML::FormHandler - a few notes

This is not an introduction to HTML::FormHandler - the docs there are very good, but I wanted to write down the main ideas I had when contributing to the design of it.

Let's look at the functions of a form processor:

1. processing the input into internal representation
   
2. checking the values
   
3. saving the internal representation into the persistence layer and loading values from it
   
4. rendering the form (with input, error messages, etc)
   

The 'best practice' of rendering the HTML view of an application page is to do it in the templates. Unfortunately there is just too much logic needed for the correct rendering of forms (mostly with the display of errors) to leave that for the templates mini language (like that of TT) - that's why this needs to be done by the form processor. But because of the diversity of the rendering requirements it also needs to be easily replaceable, ideally part by part. Then the application could start with a crude but working HTML rendering, introduced with minimal effort from the programmer since it is built into the processor and later gradually refined and replaced.

CGI.pm and after it Catalyst and other libraries parse the url-encoded and body parameters into a hash of scalar values but internally we would like to manipulate objects like DateTime instead of lists of values for year, month and day. In HFH this conversion is done in two phases - first the input is turn into a deeper structure built of hashes and arrays - then then the processor goes recursively from the leaves and at each node applies a conversion if it is defined for that node.

There is always a question when we should do the checks - sometimes it is easier do do them on the input data sometimes on the internal values. A good example is DateTime - we can check if the day field is a number between 1 and 31, but we also need to check if the whole date is correct and the business rules could add another requirement that the day of the week is Monday or some other operation that is most convenient on the DateTime object. In FormHandler all of this is possible:



How this works? As explained above the processor starts with the leave nodes - and checks the day, month and year fields (here they don't define any transformations only checks, but they could). Then it goes to the parent node date_time and there it has the hash of values from the leave nodes ready for the next transformation and checks - it is passed to:

sub { DateTime->new( shift ) }

If that subroutine triggers an exception, for example when someone passes

{ year => 2009, month => 2, day => 30 }

to it - then the error is recorded with the message This is not a correct date. If it works - the value returned from it, a DateTime object, is then passed to the next check, and later presumably saved to the database. The checks and transformations are taken from a single list - so you can interweave them as you need.
Instead of transform and check in the apply attribute we can also use the Moose types and coercions and also we can mix these two styles (for example coerce some value using Moose types and then check it again). If someone needs even more power - he can also write his own field class with it's own validation method - and do both validation and transformations in there, that method would receive the same hash of child values.

To save the full data structure of values that can contain many inter-related records HTML::FormHandler::Model::DBIC uses
DBIx::Class::ResultSet::RecursiveUpdate. I separated it so that HFH is not related to the hacks I made there :) Or maybe I hoped that perhaps other form processors could reuse my module. Using it you can update a company record together with all it's related addresses in one form.

Barriers to entry

Chromatic calls us to Remove the Little Pessimizations, to sand off the tiny, rough corners of our software that individually are small, ignorable distractions. I've read somewhere that each barrier to entry reduces the number of users by some percent - and it is a good model to think about it. Each one reduces the number of possible uses and users and the effect is cumulative - hundreds of small barriers make the tool useful in only a narrow niche, but even just one small barrier can make a tool not usable in a particular setting. The question is when fixing these little pessimizations becomes needlesly pedantic.

I believe that fixing CPAN installation issues, each one usually a small issue occurring in some special cases is needed (at least for the most popular modules) if we don't want to confine CPAN users, and since CPAN is the killer application of Perl, then it follows that also Perl users, to a narrow niche.

Another thing is how off-putting are the pessimizations that seem gratuitous, you immediately feel neglected and you imagine how the arrogant developers just don't care about your case. Then there are also pessimizations that look small if you have a big project - but are impenetrable barriers for small projects - this is how Perl lost the place of the most used language for WWW programming to PHP.

CPANHQ

There is a project to build a Catalyst based CPAN search engine, with the idea to make it more hackable and ultimately incorporate some more 'social' features into it. The original effort was started by Brian Cassidy, he created a cpanhq github repository and wrote some initial code, later there were some features added by Shlomi Fish (in his repo) and now I am working on it. In my repo you'll find a version that finally has a search page and it even works. It uses the SQLite FTS3 full text search engine.

The biggest problem for me is that I had to modify the database quite extensively and now the loading scripts don't work any longer. To show something I decided to load the database into the repo - so now it can take awful long time to download the software, but in return right after the download you should be able to try out the searching. It is not at end user quality - you need to know some internals to use if effectively (like using 'me.name desc' in the order field to sort the packages by name), but it should be reasonably fast. Now I am thinking what are the most useful search strategies.

warszawa.pm

Witam,

Co prawda zwykle piszę tutaj w języku angielskim - ale reklamówkę naszej grupy Perl Mongers postanowiłem napisać po polsku, żeby było widać, że u nas też się coś dzieje. Mamy już swoją wiki: http://warszawa.pm.org/. Jeszcze nie wszystko działa, ale miejmy nadzieję, że administrator sie postara i naprawi. Żeby dostać konto na wiki trzeba poprosić na liście dyskusyjnej, niestety ze względu na spam nie mogliśmy wiki zrobić zupełnie otwartej. Zapraszamy na wiki, listę i spotkania.

Optional Dependencies Are Going Out of Fashion

I was reading the Changes document for the latest DBIC release and I spotted:

- Remove the recommends from Makefile.PL, DBIx::Class is not
supposed to have optional dependencies. ever.

I am noting this so that it will not missed by the general public that optional dependencies are going out of fashion.

The problem here is that if an optional dependency enables some useful feature (and otherwise why would you add it?) - then you should expect that someone writes a library using this feature and uploads that library to CPAN. What that other guy should add to his dependencies? How is he supposed to know which of your optional prerequisites are needed for his module? And what if this is not about just two modules in dependency relation but rather a whole chain of them? Then it becomes a mess.

How they do it in Python (core libraries).

I don't read much from the Python blogosphere - so maybe it is not representative at all - but it seems that Python 'batteries included' approach to core libraries is also approaching its limits. I am sure that debate will happen in any popular Open Source language after reaching some maturity level. It is interesting what kind of compromise is proposed by the author of that piece:

I would first prune the standard library down to a more core focus of libraries that require cross-platform expertise, using widely accepted standards, and stuff needed to simply get a script to run. In terms of cross-platform stuff, that would mean os, multiprocess, etc. These are modules that a typical developer would have a hell of a time getting working on all the OSs that CPython runs on.

How to install Catalyst::Authentication::Credential::OpenID

This post is meant to be both a quick instruction and another data point for the CPAN installation problems thread of my blog posts.

For those impatient the winning sequence of commands (for Debian/Ubuntu) is:

sudo apt-get install libgmp3-dev
cpanp -i Math::BigInt::GMP

and now you can:

cpanp -i --force Catalyst::Authentication::Credential::OpenID


Now the story - at my pristine installation of Perl 5.10.1 on my Ubuntu box I discovered two problems with C:A:C:O installation. First was a trivial POD error, detected with the new pod tests - this is why you need to do that --force (it is fixed in the dev version). The second is more interesting - installing one of the prerequisites - Crypt::DH seemed to take forever. Fortunately by chance I found a good explanation of the problem in Crypt::DH::GMP docs - apparently for any reasonable performance Crypt::DH needs Math::BigInt::GMP which is not in the prerequisites. Without it it still works - but it can take serveral hours to pass the tests (reported first 3 years ago). So next thing is install Math::BigInt::GMP - unfortunately this is not automatic either - you need to install gmp.h header file first (on Debian/Ubuntu it is the libgmp3-dev package). This is why you need to execute the first two commands.

I hope that will help someone.

Change Management and FOSS authors obligations

The two recent controversies making rounds in the Perl blogging community made me thinking about what obligations FOSS authors have to those using their code. The traditional answer is none, but on the other hand there are some customs to the contrary - like maintaining API compatibility to at least one version back. Some people have expectation that software not marked upfront as 'experimental' will follow that custom. This is a good custom, because complete freedom for the authors can mean too much chaos for those using their code, but it is both not general enough and too radical in rejecting the traditional no obligations rule. It is not enough - because not all API changes can be made with a backcompat layer, and as shows the example of Dave porting Array::Compare to Moose - there are also non API changes that can make downstream software unusable. And it is too radical because of being an implicit, 'by default' rule. And it is also too ambiguous in it's formulation because it does not state when the new releases can be published. What we need is more general and explicite policies.

Unexpectedly changing a library's API can be like pulling a rug from under the feet of authors of software using that library. In the past that was a rare event - and usually there was time to warn all the parties involved and let them fork the project if needed. Now in CPAN we have such huge dependency tries that even if it is still rare for the individual leaf nodes - it becomes quite frequent for the root nodes where all the changes cascade from the leaves. These deep dependency hierarchies also makes forking a lot more problematic as it can mean forking not just one library - but a whole path of them leading from the leaf up to the root. CPAN might be still rather unique in it's size among distributed library projects - but with the overall exponential growth of FOSS software we can expect that to happen in other projects soon. That's why if we want to continue the process of lego-like assembling software components out of more basic libraries we need those libraries to adhere to some change management policies.

I am inviting FOSS authors to formulate their change management policies (and in particular API deprecation policies) and voluntarily add them to the manuals. I want to stress it - it is not about a top down rule forcing authors to do something - this is an invitation to voluntarily give up a bit of the authors freedom in leading the project in order to make it more useful. The bulk of FOSS can still be published as a gift to the humanity with no strings attached. I just want to show that additional option and also create some pre-canned policies that could be cut and pasted into the documentation just like the FOSS licences are.

What really fails in installation?

I've finally installed Perl 5.10.1 and I had to reinstall most of the modules I use. I set PERL_MM_USE_DEFAULT=1 and did make for a module that requires half of the modules on my computer. Everything was installed automatically - with no additional intervention from me. The morale of this is that if you are using Ubuntu and if you have installed all the external prerequisites - then the installation is really rather automatic. The problem with systems other than a special flavour of Linux is both hard to solve and rather unsophisticated - we just need more people working on those systems and reporting and solving bugs. Or at least that is the solution I can imagine. But the problem with external prerequisites is where we could do better with just a bit of programming.

I remove memcached from my system and try

zby@zby:~/progs/Enzyme$ cpanp i Cache::Memcached::Managed
Installing Cache::Memcached::Managed (0.20)
Running [/home/zby/local/bin/perl /usr/local/bin/cpanp-run-perl /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/Makefile.PL]...
No executable memcached found: No such file or directory
Running [/usr/bin/make test]...
PERL_DL_NONLAZY=1 /home/zby/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
# Looks like you planned 176 tests but ran 184.
t/001basic.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 176 subtests passed
(less 177 skipped subtests: -1 okay)
t/002inactive.t .. ok
t/003multi.t ..... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
t/010fork.t ...... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server(s)
t/020grab.t ...... ok

Test Summary Report
-------------------
t/001basic.t (Wstat: 65280 Tests: 184 Failed: 8)
Failed tests: 177-184
Non-zero exit status: 255
Parse errors: Bad plan. You planned 176 tests but ran 184.
Files=5, Tests=36330, 18 wallclock secs ( 6.12 usr 0.03 sys + 3.86 cusr 0.20 csys = 10.21 CPU)
Result: FAIL
Failed 1/5 test programs. 8/36330 subtests failed.
make: *** [test_dynamic] Error 255
[ERROR] MAKE TEST failed: Bad file descriptor PERL_DL_NONLAZY=1 /home/zby/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
# Looks like you planned 176 tests but ran 184.
t/001basic.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 176 subtests passed
(less 177 skipped subtests: -1 okay)
t/002inactive.t .. ok
t/003multi.t ..... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
t/010fork.t ...... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server(s)
t/020grab.t ...... ok

Test Summary Report
-------------------
t/001basic.t (Wstat: 65280 Tests: 184 Failed: 8)
Failed tests: 177-184
Non-zero exit status: 255
Parse errors: Bad plan. You planned 176 tests but ran 184.
Files=5, Tests=36330, 18 wallclock secs ( 6.12 usr 0.03 sys + 3.86 cusr 0.20 csys = 10.21 CPU)
Result: FAIL
Failed 1/5 test programs. 8/36330 subtests failed.
make: *** [test_dynamic] Error 255



The tests for 'Cache::Memcached::Managed' failed. Would you like me to proceed anyway or should we abort?

Proceed anyway? [y/N]:
[ERROR] Unable to create a new distribution object for 'Cache::Memcached::Managed' -- cannot continue

*** Install log written to:
/home/zby/.cpanplus/install-logs/Cache-Memcached-Managed-0.20-1251915361.log

Error installing 'Cache::Memcached::Managed'
Problem installing one or more modules

zby@zby:~/progs/Enzyme$


Yeah - the information is there - but now imagine that this was a deep dependency tree with lot's of similar external prerequisites. You try to install the module - it fails, you view the log, locate the problem, install the external dependency, run the installation again and it fails again. Everytime you need to parse the really long installation log and try to understand what happened. It is not that complicated (although it can become complicated when there aren't so explicite error messages, or when there are other installation failures mixed together) - but it is intimidating and really time consuming. Definitively it is not something that a newcomer would be able to do. But external dependencies are not something extraordinary - they are rather standard, this procedure I described here is not exactly a failure mode - it is anticipated.

Someone could argue that when I install Cache::Memcached::Managed I should know that it needs the memcached binary - but it is only obvious for those that know that module, and if that module is somewhere deep in the dependency tree you might not even realize that you are installing it.

Now imagine this:

zby@zby:~/progs/Enzyme$ cpanp i Some::Module
Computing prerequisites.
You have unmet external prerequisites:
memcached
xslt version 1.1.18
...
Please install the prerequisites first then run the installation again.

wouldn't that be helpful?

PS. How about

requires_external memcached => sub { -x "memcached" }

in Makefile.PL?

CPAN installation failures

The "Chance of all tests passing" at the CPAN dependencies and test results checker is not a scientific measure - but it is great in making it visible that installing from CPAN is still a game of chance with quite big odds of failure. The linked above Catalyst dependencies page quotes the 'probability' of installation failure as 22%, and that is for the most advertised Perl web framework. The important thing in that picture is how this relatively big failure chance comes from the individual small failure chances of the Catalyst prerequisites.

So what can we do? One thing that I would propose is to stop using optional features and dependencies because they really get into the way of automatic installs. We can also simplify the dependency graph of some more important modules, make better statistics about the failure rates and they origins, but probably the most effective way will be to be just more vigilant about the test failures in our modules and in the prerequisites. The important lesson here is that a 1% failure rate seems like nothing important - but it can make our module inadequate to become a prerequisite for some more complex library.

Let's write some deprecation policies

Sebastian writes about version numbers and backwards compatibility and asks:

Should a project have deprecation policies as soon as it hits CPAN with a development release?

I think the answer should be yes - there should always be a deprecation policy. But this deprecation policy does not need to be the full monty of always keeping the backcompat to at least one major version, using strict deprecation cycles etc - it just needs to be stated explicitely and not relying on the version numbering convention as they are not really universal. There is no negative side to putting a few sentences into the docs, only a bit work to formulate them. And here is my idea:

Let's write some deprecation policies

Let's have a list of well formulated policies covering as much as we can of possible circumstances (from experimental to mature, and from one-person projects to team work) that could be copied to the CPAN distribution docs with minimal effort. For the start - here are some ideas:

• Experimental - no policies at all, use at your own risk
  
• Experimental with mailing list announcements of new versions and promise to take into account problems encountered by people testing the new code
  
• Backcompat to one version, with a specific minimal time between versions
  
• Backcompat to one major version, with a specific minimal time between major releases
  

Update:
In related news mst gives an overview of techniques that can be used for maintaining back-compat - sometimes these are really simple things.

Dissecting the Rails Screencast

Most comments to my previous post on marketing for the Perl community focused on the visual identity of Perl web sites. It is very important, but marketing is not only that. Ruby on Rails would not be what it is now if not for the RoR screencast. It was:

Simple

Writing a blog engine - what can be simpler?

Unexpected

The first surprise is the medium - that was the one of the first popular screencasts to circulate the Internet. But of course the main load of unexpectedness was in the fact that a simplistic, but still functional, blog engine can be build in 15 minutes. Not months or years of a team of programmers, a designer and a HTML coder - but 15 minutes of one person - that is surprising.

Concrete

That was not an abstract plan, framework with holes to be filled in later - but a concrete implementation. And once again - blog engine - what can be more concrete for the web afficionados?

Credible

You can see it for yourself - it is not a description of how to do it - it is filmed the whole process of doing it. And later you can also try it on your own.

Emotional

Not really - but you don't need to cover all of the points.

Story

This is the most important thing - the screencast is a story. The viewer can easily imagine himself writing his own web application with the same efficiency - this is how stories work.

PS. Don't read this as an argument for (more) screencasts. Yes - they can be effective - but it all depends. What I am advocating here is using that checklist for all marketing projects.

Marketing for Perlers

Sources on the intertubes report that marketing is becoming a hot topic around the Perl community. It might be a good time to learn a bit about that subject and perhaps exchange some recommendations. I don't want to retract anyone from hiring a professional marketer, to the contrary I think this is the mature thing to do. We need to admit that our own knowledge in that area is limited and seek professional help. But learning a bit more will not hurt.


The book that I would like to recommend is Made to Stick. Why Some Ideas Survive and Others Die ... - it is about creating messages that will be remembered. It is build around a simple list:

• Simple
  
• Unexpected
  
• Concrete
  
• Credible
  
• Emotional
  
• Stories
  

these are the ingredients of an effective marketing message (and yeah the first letters do form an mnemotechnic acronym).

The message needs to be simple, so that the basics are easily understood and can be easily spread around. I think this will be well understood to programmers. It is not about dumbing it down - it is about finding the real core. It needs to be unexpected to catch peoples attention. To hold attention the book advices to use curiosity gaps. Next quality is concrete - this is about involving things the audience already knows well, tangible things and painting a mental image. Credibility comes from inside or from an outside authority, inside credibility comes from using vivid details, statistics but also can be gained by inviting the audience to test for themselves. Emotional is about speaking to emotions. Finally the best messages are stories - the power of story telling comes from the fact that that people reach the conclusions for themselves and they easily absorb them as their own.

Apparently this was tested by the ads business:

The final group was trained for two hours on how to use the six creative templates. Once again, the fifteen best ads were selected by the creative director and tested with consumers. Suddenly these novices sprouted creativity. Their ads were rated as 50 percent more creative and produced a 55 percent more positive attitude toward the products advertised. This is a stunning improvement for a two-hour investment in learning a few basic templates! It appears that there are indeed systematic ways to produce creative ideas.

from excerpts.

The Catalyst API

Now, after the Moose port is completed, it looks like a good time for reviewing the Catalyst API. It's not a secret that in my opinion some of the Catalyst practice is a cargo cult (for example using forward instead of a subroutine call), other parts could be greatly simplified. In How I Use Catalyst the interesting part is how Dave also does not use some important parts of the Catalyst API, and Marcus in his reply, even though he disagrees with Dave, still he does agree that there might be a more suitable API for some things.

It's time to review the API, check how Moose can improve it, remove the redundancies and make it slim.

One more piece of food for thought: the thing that always bothered me is the first parameter every action receives:



I've never seen it used. I even was never sure if that thing is a controller object or if it is the controller class (i.e. if the actions are class or object methods), is it ever explained in the docs? Now I tested it and the output was:



so at least that action was an object method. I've reviewed the whole Catalyst Tutorial and, even though this parameter is passed to every action example there it is used in only a couple places (as $self->action_for('list') and $self->roles). This does not seem like the best Huffman coding.

Dear TPF - don't send money, send thank you letters

I have already commented at the Proposed Payments for Clearing Perl 5 Bugs blog post from The Perl Foundation - but now I think I have an even better idea. I am now unemployed, maybe I could try this bug hunting as a way to get some income during these hard times, but the money, that is most probably at stake, could hardly solve any problem for me. Still there is one thing that could have much impact on my situation - I am sure you already know what I have on mind - a printed thank you letter from The Perl Foundation that I could take with me to the interviews. This would not cost much, even for something printed on a nice paper - but it could have a great impact on peoples careers.

Even if the money rewards are already something decided - and no change can be made - TPF could at least accompany them with such a thank you letter. And there should be a way to make it signed by Larry Wall himself - this is crucial for the impressiveness factor.

Fixing dependencies

Fixing other people code is not easy - but this is what we need to do if we want to clean our dependencies. In answer to John's invitation here is what I did.

I wanted to write some code using Catalyst::Authentication::Credential::OpenID unfortunately one of it's dependencies fails it's tests. Here I'll document how I analysed that and produced a patch, it's not yet approved by the author - but the tests pass.

Step first - checkout the ParanoidAged from it's repository (I had the repository address from an email to the Catalyst mailing list). If you don't have access to the repository of your failing dependency - then just download the source from CPAN unpack it and work from there. Then run the tests:

zby@zby:~/progs/pa$ perl -Ilib t/00-all.t
.
.
.
ok 27
5 second tarpit (tolerance 2)...
not ok 28
# Failed test at t/00-all.t line 180.
3 second tarpit (tolerance 4)...
ok 29
Killing child pid: 7669
# Looks like you failed 1 test of 29.

OK - the failing test is at line 180. Let's see what is there:

I changed that to:

ran the tests and save my debug output:
zby@zby:~/progs/pa$ perl t/00-all.t 2>debug.
What I got there is:



Now the info is there - but of course I did not spot it at once. I did a lot of more debugging and testing other hypothesis before I focused on: 'x-died' => 'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394.'. Yeah interesting - something died, but no error was reported. The line in question is:



To get some more evidence I changed that to:



Now the debug output changes to:



And again - it is there - but to spot it I had to play with the debugger first. I don't remember what exactly I did - but finally I sow it - there are two evals in this stack trace, the first one is in LWP::Protocol and the second in lib/LWPx/ParanoidAgent.pm line 313. I checked that second eval - and yeah - it tried to retrieve the error already catched by the first one. So when there was a timeout the agent was aborting the retrieval - but the error was being cleared and not reported. I wrote following patch:



and now all the tests pass. Finally I uploaded the patch to RT.

On dependencies

Ovid writes about Mojolicious::Lite:

The fact that it has no dependencies makes it even more compelling.It's the sort of thing one could more easily attract new developers with. Can you imagine rewriting my old CGI course with something like this? Or maybe a way to introduce new users to Perl? There are many excellent Perl frameworks out there which I would, nonetheless, hesitate to start new users on. Just trying to set up CPAN can be difficult for them; working through even one test failure is often too much. For people who want to jump in and learn "Web programming", though, this might work.

On the other hand Jay advices to stop worrying about dependencies The lie of independence - or - how I learned to stop worrying and love the dependency chain. There is truth in both statements. I have the impression that the problem of CPAN dependencies have improved much recently. The visibility of installation problems directly from the cpan search engine (cpan testers matrix, dependencies review) is a great step ahead. But still installing Catalyst and DBIC and TT is a big hurdle - and I can understand why Ovid is not enthusiastic about teaching Perl novices how to do that.

For me Mojo sounds like iPod - slick and beautiful and really useful - but you are not supposed to change the battery. It is a nicely integrated product - but somehow closed. The author apparently tries to make it perfect, tries to polish every detail - what perhaps would not be possible if he had to compromise with the visions of the authors of the other packages. I can feel his frustration here. But rejecting all dependencies and rewriting everything in his own way sounds a bit extreme and not welcoming.

The big question here is how much can we trust fellow CPAN authors to make the right decisions, can we rely on their work? Can someone, who put's a lot of attention for the details of his API, rely on modules with imperfect APIs, perhaps spoiling his own work and not become insane? The story of the phalanx project shows that there are no easy answers to these questions, but I believe we can do better than rewriting everything from scratch each time.

Google Wave

One, sometimes feverishly defended, rule of our technical irc channels is about not pasting code into the conversation. To show your code you are supposed to use some web based paste application and then copy the link into the channel. The need to use two separate tools, irc client and web browser, for one task, code review, is just a small annoyance - but eliminating those small annoyances is the only way towards really usable software. Many people have internalized the procedure so much, that they don't think about it any more and I am sure there will be some of them arguing that this is not annoying at all. But there are additional steps to do: choose the tool, paste the link, switch the windows and all this is a burden to the brain, even if it is processed in the background - it still consumes some brain resources.

And now imagine that you have it in one tool - not only those additional steps go away - but it also opens the way for new features like seeing a pointer to the part of the code that the other person is talking about, seeing his edits in real-time and commenting on them immediately etc.

This is why I believe in convergence of communication tools.

Nothing is forever

I have been writing about the scaffolding metaphor - but after some conversations with my colleagues I realized that I did not do a good job there. I did not conduct well the main idea: nothing is forever. No library and no program is forever and we should not shy off from writing libraries that are useful only at some phase of the application construction. It is still valuable if you can quickly get to the point of having a preview, a simplified prototype of the functionality you need even if eventually you'll remove all of that prototype code from the project. This is very 'agile' and it is also 'worse is better (if you can have it early)' and those libraries, those tools that you use only temporarily, I call the scaffolding.

Now one would ask why write a library that is useful only temporarily? Why not include at once all the parts needed in the final product? There can be several reasons:

• something that is only a part of the scaffolding in one project can stay in the final solution in another one
  
• the author of the library might not yet fully understand the problem, and it can take years to reach the right abstraction
  
• the distribution of possibly needed modification can be so uniform that there is no reason to concentrate on one of them more than on another one and it is virtually impossible to cover them all
  
• and finally: nothing is forever - you never know where a project will go in the future and what you'll need to change
  

A good example (of the third point) are code samples from manuals - you copy them to quickly get something working - but later you modify it and sometimes you change every character from the original. Another one can be the scaffolding code in Rails - you can debate if it fits the first or the second or other points - but it was useful to a lot of people.

Every library can be used as scaffolding - so why not accept this and help the developers in using it that way?

CPAN and code reuse

CPAN is wonderful thing - it is a great achievement of the whole Perl community and the thing that makes it apart from other languages. But somehow it works mostly for libraries. And what about the other modes of code reuse? How about CPAN for code examples? One can say we don't need CPAN for code examples - everyone is using code examples all the time and they do that without any special infrastructure. But what I see around is that some of CPAN modules are actually code examples in disguise. Some of them would be better without any code in the package and only documentation and tests validating the documented techniques. Hmm maybe we don't really need a separate CPAN for that - maybe we can keep them at the existing one - and just mark them in a special way? And maybe we need some social acceptance for packaging such 'empty' libraries?

And what with applications?

Code reuse styles

There is much buzz around PHP tools like Drupal and Joomla. They are ready-made solutions, but in fact there is also a lot of tweaking the templates and writing add-ons. It seems to work very well - the clients get something usable from the beginning - and later they can modify it to fit their purpose better. This is very agile. In contrast the reuse in Perl is mostly around libraries. This is much more flexible - you don't tweak a ready solution - you assemble it from the thousands of CPAN packages. But it is also less agile.

I wonder why it is like that. There are some technical differences that lead developers in one or the other direction - but there is also a cultural/psychological cause. Doing some self-analysis I must admit that I am not really enthusiastic about contributing to a Perl CMS (like for example Bricolage) - in my view all the already-build solutions use obsolete libraries. I'd rather contribute to building the perfect library - then code an extension to Bricloage. I have the feeling that I am not alone in that crazy perfectionism around Perl hackers. PHP clearly wins here with more pragmatic approach.

Deprecated code analyzer for HTML::FormHandler

So the other day I've read Deprecated code analyzer for perl and I thought - this is a GREAT idea! And I stole it for HTML::FormHandler. Now with new releases our users will have a tool to find all the deprecated API calls in their code.

Is using the Catalyst 'forward' method a cargo cult?

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:

$c->forward( 'some_action' )

and the standard Perl:

$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.

Functional versus Unit testing

Benefits of automated functional testing (was: Why unit testing is a waste of time) is a comprehensive list of arguments for functional and against unit testing. It is a bit long and maybe it is preaching to the choir as I have never seen unit testing in CPAN modules, but it might be handy if you needed to convince your co-workers.

Also discussed at Hacker News

InstantCRUD and FormHandler

The new version of Catalyst::Example::InstantCRUD is on CPAN. This version uses FormHandler, is a bit nicer visually (I hope) and has a proof of concept experimental REST interface.

Iron Man Blogging Challenge

First I need to confirm that I really like the idea. Perl needs more publicity. Some may say that quality is more important than quantity but there is no better way to improve your writing skills than practice. So yeah - it's a great idea. In general I think we need more competitions - but that is really for another post.

There is just one small nitpick - if the goal is: "to promote the language to the world outside the Perl echo-chamber" then the rule that the all the posts need to be about Perl is misguided. If we want to talk to "the world outside the Perl echo-chamber" then we need to talk about the subjects that they care about. I can understand why we need some focusing - and posts about what you had for dinner are not the best way to promote the language - but it will be much easier to engage the outside audience if we wrote about general programming subjects. Look at advertisings - they don't talk about the advertised brand all the time, the more subtle the link is, the more engaging they are.

Update: I should have added that this post was inspired by Gabor Szabo: http://szabgab.com/blog/2009/06/1245788806.html

CatalystX::Comments - RFC

In Packaging cross cutting Catalyst features I promised to start developing a generic Catalyst comment sub-system. Now I have incorporated the code samples into another experiment - ravlog(a blog engine by Gerda), and I would like to ask you what you think about it before releasing it to CPAN.

Here is the current API - any comments are welcome. First you need to declare the controller using the new Moose style:

package RavLog::Controller::View;

use Moose;
BEGIN {
extends 'Catalyst::Controller';
with 'CatalystX::Comments::ControllerFormRole';
}

has model_name => ( is => 'ro', isa => 'Str', default => 'DB' );


The important part for us is of course with 'CatalystX::Comments::ControllerFormRole', the model_name attribute should hold the name of the model CatalystX::Comments will use to store the comments.
After the declarations you can use CatalystX::Comments to put the comment form on the stash:

sub view : Chained('base') PathPart('view') Args(0)
{
my ( $self, $c, $id ) = @_;

...

$self->stash_comment_form( $c, $self->article->id );
}

That's the whole controller part - what it handles is generating the form for display and saving the submitted comments. In the view templates you need to add:

[% comment_form.render %]

For displaying the comments themselves ravlog already had some template code and I did not change it.

Next is the model part. The library assumes that the DBIC model contains a Comment Result class like that one in ravlog. Of course it does not need to have all of the columns, and can also have other columns - but these are the columns that will be filled by the CatalystX::Comments form. Ideally that model part should be a Result base class - so that it can be subclassed and adapted to local circumstances (for example the belongs_to relationship to article needs to be changed), but for now I don't know how to do that.

Now some more details. Thanks to html_prefix all parameters sent from this form are prefixed with the form name - so this form can be added to pages with other forms, it will recognize it's parameters. The comments are saved only when the HTTP method used is POST, and after a successful comment creation the user is redirected to the same page (this seems a bit constraining - but see next paragraph - this is only meant to be temporary solution - I try to keep the API simple).

Of course I understand that this can never cover the needs of a comment sub-system in a mature social web site. I am thinking about it as more of scaffolding - code that let's you quickly develop a feature, see how it integrates with the rest of the user experience, formulate a more complete requirements list - and replace it part by part.

On Scaffoldings

Scaffolding is a wonderful metaphor. Was it invented in Ruby on Rails? I am not sure about that - but nevertheless it is RoR that popularized it and undoubtedly it was one of the engines of it's popularity. It is very "Agile" - like a programming sprint - it implementa a feature in the most simplistic way that can possibly work - but instead of a programming team doing the work the code for that feature is generated.

Scaffolding is a wonderful metaphor - and the brilliance of it shows in how it transcends the original area of code generation where it was introduced in Rails. It can be applicable to any reusable software component meant to be temporary support and to be replaced later. An example of that are code samples in manuals - you copy them just to have something working, but then you modify it and nearly completely replace the original code. And since nothing is universal and no library is ideal - it can be used as a generic measure of how well a library supports this iterative development style. And this brings me to the question - what are the qualities of a good scaffolding? One is that it needs to be gradually replaceable.

Module Rank and Informed Choice

There is much talk recently about CPAN and how to improve it. I firmly believe that initiatives like CPAN Top 100 will eventually lead to some good, Page Rank like, measures of if not module quality then at least module utility. People vote with their legs - they use other modules in their own productions and we don't need any other voting mechanisms. After all Google does quite well with Page Rank, much better than the human edited Internet Directories (like the original Yahoo or Open Directory).

But it all still relies on people choosing the right modules for their own projects - I believe this is a good assumption, the popular "The Wisdom of Crowds" shows why - but still it can be improved by people doing more informed choice. This was my motivation behind my API design blog post. Design of APIs is tricky and subtle and there are things that you just don't spot until it is too late, or alternatively there are deficiencies that are invisible for the seasoned user because he is used to the quirks, he internalized them and he uses the API automatically without thinking.

Form rendering and CSS

I tried a few times to learn CSS, but not really seriously, I guess, since I still know only the most basic things. So I am a bit lost now when I need a nice and universal CSS for FormHandler forms in Catalyst::Example::InstantCRUD. Or perhaps I should say HTML::FormHandler::Render::Simple forms instead of just FormHandler forms as theoretically the HTML rendering part is separated from the main FormHandler code, so that if this Simple renderer turns out really too simplistic I can also write a HTML::FormHandler::Render::NotSoSimple to get the right HTML.

I started reviewing the options and it seems that there are three general approaches to having aligned forms:

• tables
  
• divs
  
• lists
  
• no additional markup
  

Plus some appriopriate CSS. I think the 'no additional markup' option could never accommodate for error messages, so I leave this one out. This leaves me with three options. I am sure that not everything can be done in each one - but I think we can add a 'style' parameter to the renderer to make it configurable.

Next thing is the recently popular CSS grids - anyone using them? They seem like a nice option for someone needing some universal defaults and the SenCSS example even contains a form styled in two ways (vertical and horizontal alignments) - but would it work with error messages? And the Blueprint one even shows a div with the error class - but how that is to be inserted into the form? I guess each one requires it's own way of HTML structuring.

Packaging cross cutting Catalyst features

A recurring subject on the Catalyst mailing list is about packaging functionality that is not entirely comprised in one of the Model View or Controller parts of the framework (and is not a full application on itself). This is hard and still there aren't many CPAN libraries that do that. One solution to that is writing "helpers" that generate the integration code into your application files. As with all code generation this solution creates many new problems - and ideally we would like to avoid it.

To try out what can be done about that I decided to write a generic comment system for Catalyst (+DBIC, TT and FormHandler). I've already posted the the first code samples to the Catalyzed wiki. The form code and the controller stuff seems closed - it does not need to refer to things from outside but in the DBIC declarations the belongs_to relation of the comments needs to relate to an external table and it's primary key (I assume that comments belong to an article/post/something). How that can be made configurable?

API design

In the past programming was more about writing from scratch - now code reuse take more and more prominent place in development. In the Perl sphere this is clearly visible in the growth of CPAN, but what is also visible is that this quantity is not everything.

There are many aspects of quality - here I would like to focus on the APIs. The problem with APIs is that their shortcomings are much more subtle than let's say bugs. Many of them are just inconveniences - things that you stop to see once you get accustomed to them, and all of them require some experience or theory just to understand that it can be done better. Arguments about APIs can become dangerously subjective. We are in dire need of API theories, objective measures to evaluate them - and here I try to compile a list of on line resources on that subject.

How To Design A Good API and Why it Matters (from Google Tech Talks)"

- see also an ACM paper with the same title and a nice abstract in the form of bullet points.

My take-away points (the most non-obvious or potentially useful in arguments):

• write examples from the very start and build the API from use cases
  
• good names are important - API is language
  
• "when in doubt leave it out"
  
• implementation somtimes leaks out to the API
  
• for inheritance document self-use pattern (how methods use one another)
  
• "don't make the client do what the module can do"
  
• you can't please everyone so aim to displease everyone equally.
  

The Little Manual of API Design

Characteristics of good design:

• Easy to learn and memorize
  
• Leads to readable code
  
• Hard to misuse
  
• Easy to extend
  
• Complete
  

Again stress on writing use cases and code using the API during the design process - and one surprising but seemingly valid point: "prefer specific names to general names".

API Design Matters

This one is mostly arguing why API design matters - but there are a few points worth noting:

• always design the API from the perspective of the caller (rather obvious)
  
• again start with use cases
  
• there used to be much more programming from scratch - now we use more libraries and designing API becomes more and more important, but this is not yet reflected in university curricula (this is exactly the intuition that led me to this blog post)
  

Designing and Evaluating Reusable Components

The needs of the programmer changes as her project proceeds and it is important to account for that fact when writing the API. In particular more advanced projects usually need more granular APIs while at the beginning they are only an obstacle.

API design guidelines

- my question on Reddit.

Update: Read Curing Python's Neglect - it's not really about Python in my opinion - and it is about how people get used to bad APIs and stop noticing the ugliness of them.

Constraints and Types (in Moose)

Moose::Manual::Type says that Moose types are mostly just constraints with a name. When designing new constraints API for Formhandler we decided to use Moose types - but with some playing around it also exposed the fact that not all constraints are types. I am not an expert in Type Theory - but I checked wikipedia and it seems to support my intuition that type is an attribute of data value - while when validating we often need to check not just the value - but also it's relation to the whole system.

If that is not yet clear - let's take the example of checking the age of a application user (from his provided birth date). The question if a user with a given birth date can legally participate in a web forum cannot be answered without knowing what is current date - it is not a property of the birth date alone - but of it's relation to the current date. I think that using the term 'type' for this kind of constraints can be confusing, even though technically in Moose this is possible to declare this.

Roles for FormHandler

This is a more concrete version of the question from my last post. This is a form defined in FormHandler:

package BookDB::Form::BookWithOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'Author' );

has_field 'title' => ( type => 'Text', required => 1 );
has_field 'publisher' => ( type => 'Text', required => 1 );
has_field 'owner' => ( type => 'BookDB::Form::BookOwner' );

This should be quite self-explanatory for anyone working with forms (and DBIC).
Now - I think Model::DBIC should be just a role here, but that is just a side track. The real question is about the fields here - ideally I would want that doing:

with 'HTML::FormHandler::Model::DBIC';

(this is after it is made a Role) - would add appriopriate methods to the fields - an example here could be validation of fields that need a unique value. I would like to not need to change the definitions above to:

has_field 'title' => ( type => 'Text::DBIC', required => 1 );

Fields like 'owner' above which are Compound fields and work mostly like a form itself and need to load the options for SELECT boxes from the model, set the values of fields from a database row etc:

package BookDB::Form::BookOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Field::Compound';
with 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'User' );

has_field 'user_name';
has_field 'fav_cat' => ( label => 'Favorite Book Category' );
has_field 'fav_book' => ( label => 'Favorite Book' );
has_field 'occupation';
has_field 'country' => ( type => 'Select' );


Ideally this definition should not need 'HTML::FormHandler::Model::DBIC' - applying this role to the root form should be enough.

Roles and data structures

Let's say you have a tree like data structure, something not exactly so uniform as a tree but with enough uniformity that it makes sense that large part of the code is recursive. My example is something representing an HTML Form for editing a complex data structure (with sub-forms etc.). The main form code is used for inflating external data represetntation (key, value pairs) into internal objects and validating it. But there are two additional 'aspects' of this form processing - this is generating the HTML of the form (with all the error indicators and messages) and saving the form data into the data store (by manipulating DBIx::Class schema for example). So you have a tree-like data structure and three separate functionality areas related to it. How do you structure the code?

You can start with defining the data structure and one of the aspects as the 'primary package' - and then add the other aspects as (Moose) roles. This is how it is started in FormHandler - the primary aspect here is the data processing and validation - then saving the the database is added as a role (so you can use it with different data store libraries DBIx::Class, Class::DBI and in the future all the other), the same with rendering - it is a role added to the form class. This is all nice untill you realize that in this way you add the additional roles only to the main form object - but not to the fields (i.e. nodes in the tree) - so you can walk the fields and do the stuff - but you regress here to mostly procedural code.

What we need is to somehow add this role globally to all the field and form classes used.

Hierarchy of code reuse

Hierarchy of code reuse by flexibility:

• Code generation and cut and paste: unconstrained modifications
  
• Inheritance and passing callbacks: modifications only at the method granularity
  
• Normal libraries: no modification of existing algorithms - only composing them into larger constructions
  

Reading "Design Principles and Design Patterns"

Design Patterns are an intriguing idea - there are some attempts to port it to Perl, but I would not call them too sucessfull in spreading it. Design Principles and Design Patterns by Robert C. Martin tries a higher level view on the design patterns by defining some general rules about dependencies and abstraction and then showing how the patterns follow from them. This perhaps should be easier to translate into dynamic languages.

I think the real takeaway is "Stable packages should be abstract packages" - where stable for the author means 'many other packages depend on that one'. The problem with translating that to dynamic languages is that the most abstract (in the OO sense) packages, the pure interfaces don't need to be written down in code at all. Duck typing gives us a great flexibility - but it also means that there is no requirement to write down the abstract interfaces - and if there is no additional pressure then they will not be defined by the programmers.

Another problem with translating the ideas into dynamic languages is the meaning of the word 'depend'. With static typing it can be clearly defined by the need to recompile the package if there is some change in the package it depends on. For dynamic typing this needs to be defined in some more subtle ways - basing it on the type of the coupling between the packages.

Open Source fragmentation and cognitive biases

I've just finished Predictably Irrational (it was nice, many of the of the theses I already knew from Internet articles - so it was not ground-breaking for me, but still I learned from it). In the seventh chapter of this book entitled "The High Price of Ownership" the author presents some surprising stories about how we "overvalue what we have". Apparently we all give much more value to things that we posses then to those that we could potentially have - this is a common cognitive bias.

This made me thinking. Wouldn't that explain the fragmentation of Open Source libraries (and CPAN in particular)? Would that not explain why one author values the miniature efficiency gains of his reinvented wheel over the robustness of time-tested code?

Maybe once we know how this mechanis works, we could design better Open Source library repositories, limitting the power of that fragmentation drive? Maybe by being more explicite about our choices we can become more rational?

Catalyst helpers - plugins for Module::Starter?

Code generation is hard. First of all it is hard to determine what really needs to be created - in (computing science) theory you can always uncurry the generator and use it as a library. So you need different theory to really determine if code generation makes sense at all. In practice many people use it - since it let's us start new projects faster (and more correctly). This is an interesting subject deserving many more blog posts - but for now I just wanted to toss the idea that joining forces with other Perl projects facing this problem could help us improve the mess that is Catalyst helpers now.

I've learned about Module::Starter plugins from Recursive development that leads nowhere.

Progress

Now that I use CGI::Application, Class::DBI and HTML::Template at work I can see the advantages of using Catalyst, DBIx::Class and Template Tookit. Personally, I find using HTML::Template as the most annoying of those three set backs - with TT I've got used that I don't need to worry about displaying the data I feed to the template, I need to make sure that it contains everything needed but that's all. I know that the view can take any data and display it. This makes a nice separation of work and of mind sets between the Controller and the View. Not so with HTML::Template.

One gotcha less!

Looks like one of the anomalies I described in my previous post is fixed now in the svn (see Revision 4773 of DBIC). Thumbs up!

DBIx::Class gotchas

DBIC is a complex beast - and it has it's rather obscure corners where it can burn the inexperienced. Here are two of those, comments with others are welcome.

Don't use $rs->update_or_create(...) (or $rs->find_or_create) on a table with autoincrement primary key

The ->find method requires that all the columns comprising the key used for the search are present in the query or in the resultsource conditions. Otherwise it will return some random record (and emit a warning). This is because currently there is no way to determine if all the needed columns are present. To do this you would have to parse the arbitrary SQL::Abstract query that can be in the resultsource condition and check how the columns are used there. Think for example about:
$new_rs = $cd_rs->search( { -and => [ id => [1, 2], id => { '>' => 1 } ] } )
and then $new_rs->update_or_create( { artist => 1 } ).

But for autoincrement primary keys the ->create method requires that the primary key is not present in the query. If you provide { pk => 'undef' } to ->create, this would generate a query like: INSERT INTO cd ( pk, ...) VALUES ( NULL, ... ) - some databases would accept that but others (PostgreSQL for example) not.

So in summary, for tables with autoincrement primary keys ->find requires that you provide the primary key in the query, and ->create requires that you don't - this means that you cannot pass the same query hash to both methods and this is what ->update_or_create does.

Update: Looks like the 4773 revision fixes the problem described below. That's why I moved this point to the end of the post. I did not check it too deeply though.

Don't call $row->some_relation when the $row has not loaded the columns required to resolve some_relation

Column values in DBIC are kept in a hash - this means that a column can have some defined value, have undefined value or can be completely non-existent - and DBIC interprets all of those states differently. When $row->some_relation is called and ->some_relation uses a foreign key column that is non-existent then the result is currently random (and in the future most probably an exception will be raised).

When this can happen? When you create a row and don't provide the value for the column (or if you retrieve the row from the database and explicitelly omit the column). For example if you do
$cd = $cd_rs->create( { name => "Cisza" } )
(or $cd = $cd_rs->new...). Now the 'artist' column in $cd is non-existent and when you try to call $cd->artist it blows up.
In a library where you don't know how a record was created you need to check first if the relevant columns are loaded before you call a relation on the record.

I thought that a good remedy would be also to always provide undef in the ->create and ->new calls for all the columns that we don't have values for, but allegedly this is not a good strategy (don't ask me why).

Installing Test::WWW::Mechanize::Catalyst

It seems that there is some incompatibility between the latest LWP version (5.812) and Test::WWW::Mechanize::Catalyst. The tests die with an:

HTTP::Message content must be bytes at lib/Test/WWW/Mechanize/Catalyst.pm line 88

error message. After downgrading LWP to 5.808 Test::WWW::Mechanize::Catalyst (version 41 - which looks better than 42 at CPAN Testers: Reports for Test-WWW-Mechanize-Catalyst) worked. Looks like this change is the culprit: "Don't allow HTTP::Message content to be set to Unicode strings".

confound - thanks for the hint.

Interesting articles from the Ruby community

Why coupling is always bad / Cohesion vs. coupling and Why Rails is total overkill and why I love Rack - looks like the Catalyst philosophy is quite close to that of the Rack project (but maybe less articulated).

As a side-note - I cannot comment there (even though I have javascript turned on) - do you get the same problem?

Rose::HTML::Form and DBIx::Class

Has anyone tried that combination?

Writing Facebook apps in Catalyst

A few weeks ago I've started a howto page for writing Facebook applications in Catalyst at the Catalyst wiki. Now I have accumulated there a few tips. Everyone using Catalyst for Facebook apps is invited to add their problems and solutions.

Refreshing a row from the DB - another note to myself

This is another thing that I keep forgetting - how to refresh a row with values from the database.
So here it is:

$row->discard_changes;

It comes from DBIx::Class::PK (and not from DBIx::Class::Row where I usually start the serach).

Update: The core devs say that the new version of documentation will mention discard_changes in DBIx::Class::Row POD. Thanks.

Automatic DBIC schema creation from a database

This is something that I keep forgetting and then I have to wade through the docs again. And there are traps in the docs - for example SQLT can do the schema creation - but it does not build the relationships.

So here is the magic invocation for DBIx::Class::Schema::Loader which do create the relationships (if you use the right switch):

perl -MDBIx::Class::Schema::Loader=make_schema_at,dump_to_dir:./lib -e make_schema_at("New::Schema::Name", { relationships => 1 }, [ "dbi:Pg:dbname=foo","postgres" ])’


A free implementation of Amazon SimpleDB

Here is a tip to Amazon - there should be a free implementation of SimpleDB. I am now considering using SimpleDB for a web application - but I would like to develop it on my desktop computer - without paying Amazon, without even connecting to Amazon. This might be a minor issue - the fee you would pay for developing the application on Amazon web services seem to be very small. But it would be so much more convenient: no fiddling with credit cards, no problems for developers with shaky internet connection and more compatibility with Free Software/Open Source development models where money is always a difficult subject.

For Amazon it would not mean much less income - since as I've said the fee for the development use of the services is rather small - but it would mean many more customers. Perhaps it would also mean some competition - but this would only validate the whole model and make it more suitable for big business which does not like so much dependence on one provider.

Update to 'Tags and search and DBIx::Class'

Advanced Search in web DBIx::Class based applications (with tags, full text search and searching by location) is an update to Tags and search and DBIx::Class.

I think it is a nice technique.

Have you ever forgotten where some subroutine comes from?

Or have you ever been learning a big stack of modules, like DBIx::Class or Catalyst and stumble from time to time over a new method name and you did not know where to look for it's documentation? I have once even wrote a script that generated an index of all methods documented in some code directory tree. It worked for me when learning DBIx::Class - but you needed to be very careful when choosing what you are indexing - you indexed too much and the list just got too long, you indexed too little and it's coverage was too limited.

Until discovering Pod::POM::Web I did not know that what I need is actually a full text search over all the Pod documents - you cut and paste the unknown method name into the search box press Enter and voila you have list of modules that document that method. And if you know that it comes from one of the DBIx::Class submodules you just add DBIx::Class to the search criteria - and don't get all the Class::DBI documentation pages. No problem with too long index lists. Someone should hack it to run against a full CPAN mirror and beat the other CPAN search websites.