Is using the Catalyst 'forward' method a cargo cult?

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:

$c->forward( 'some_action' )

and the standard Perl:

$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.

Functional versus Unit testing

Benefits of automated functional testing (was: Why unit testing is a waste of time) is a comprehensive list of arguments for functional and against unit testing. It is a bit long and maybe it is preaching to the choir as I have never seen unit testing in CPAN modules, but it might be handy if you needed to convince your co-workers.

Also discussed at Hacker News

InstantCRUD and FormHandler

The new version of Catalyst::Example::InstantCRUD is on CPAN. This version uses FormHandler, is a bit nicer visually (I hope) and has a proof of concept experimental REST interface.

Iron Man Blogging Challenge

First I need to confirm that I really like the idea. Perl needs more publicity. Some may say that quality is more important than quantity but there is no better way to improve your writing skills than practice. So yeah - it's a great idea. In general I think we need more competitions - but that is really for another post.

There is just one small nitpick - if the goal is: "to promote the language to the world outside the Perl echo-chamber" then the rule that the all the posts need to be about Perl is misguided. If we want to talk to "the world outside the Perl echo-chamber" then we need to talk about the subjects that they care about. I can understand why we need some focusing - and posts about what you had for dinner are not the best way to promote the language - but it will be much easier to engage the outside audience if we wrote about general programming subjects. Look at advertisings - they don't talk about the advertised brand all the time, the more subtle the link is, the more engaging they are.

Update: I should have added that this post was inspired by Gabor Szabo: http://szabgab.com/blog/2009/06/1245788806.html

CatalystX::Comments - RFC

In Packaging cross cutting Catalyst features I promised to start developing a generic Catalyst comment sub-system. Now I have incorporated the code samples into another experiment - ravlog(a blog engine by Gerda), and I would like to ask you what you think about it before releasing it to CPAN.

Here is the current API - any comments are welcome. First you need to declare the controller using the new Moose style:

package RavLog::Controller::View;

use Moose;
BEGIN {
extends 'Catalyst::Controller';
with 'CatalystX::Comments::ControllerFormRole';
}

has model_name => ( is => 'ro', isa => 'Str', default => 'DB' );


The important part for us is of course with 'CatalystX::Comments::ControllerFormRole', the model_name attribute should hold the name of the model CatalystX::Comments will use to store the comments.
After the declarations you can use CatalystX::Comments to put the comment form on the stash:

sub view : Chained('base') PathPart('view') Args(0)
{
my ( $self, $c, $id ) = @_;

...

$self->stash_comment_form( $c, $self->article->id );
}

That's the whole controller part - what it handles is generating the form for display and saving the submitted comments. In the view templates you need to add:

[% comment_form.render %]

For displaying the comments themselves ravlog already had some template code and I did not change it.

Next is the model part. The library assumes that the DBIC model contains a Comment Result class like that one in ravlog. Of course it does not need to have all of the columns, and can also have other columns - but these are the columns that will be filled by the CatalystX::Comments form. Ideally that model part should be a Result base class - so that it can be subclassed and adapted to local circumstances (for example the belongs_to relationship to article needs to be changed), but for now I don't know how to do that.

Now some more details. Thanks to html_prefix all parameters sent from this form are prefixed with the form name - so this form can be added to pages with other forms, it will recognize it's parameters. The comments are saved only when the HTTP method used is POST, and after a successful comment creation the user is redirected to the same page (this seems a bit constraining - but see next paragraph - this is only meant to be temporary solution - I try to keep the API simple).

Of course I understand that this can never cover the needs of a comment sub-system in a mature social web site. I am thinking about it as more of scaffolding - code that let's you quickly develop a feature, see how it integrates with the rest of the user experience, formulate a more complete requirements list - and replace it part by part.

On Scaffoldings

Scaffolding is a wonderful metaphor. Was it invented in Ruby on Rails? I am not sure about that - but nevertheless it is RoR that popularized it and undoubtedly it was one of the engines of it's popularity. It is very "Agile" - like a programming sprint - it implementa a feature in the most simplistic way that can possibly work - but instead of a programming team doing the work the code for that feature is generated.

Scaffolding is a wonderful metaphor - and the brilliance of it shows in how it transcends the original area of code generation where it was introduced in Rails. It can be applicable to any reusable software component meant to be temporary support and to be replaced later. An example of that are code samples in manuals - you copy them just to have something working, but then you modify it and nearly completely replace the original code. And since nothing is universal and no library is ideal - it can be used as a generic measure of how well a library supports this iterative development style. And this brings me to the question - what are the qualities of a good scaffolding? One is that it needs to be gradually replaceable.

Module Rank and Informed Choice

There is much talk recently about CPAN and how to improve it. I firmly believe that initiatives like CPAN Top 100 will eventually lead to some good, Page Rank like, measures of if not module quality then at least module utility. People vote with their legs - they use other modules in their own productions and we don't need any other voting mechanisms. After all Google does quite well with Page Rank, much better than the human edited Internet Directories (like the original Yahoo or Open Directory).

But it all still relies on people choosing the right modules for their own projects - I believe this is a good assumption, the popular "The Wisdom of Crowds" shows why - but still it can be improved by people doing more informed choice. This was my motivation behind my API design blog post. Design of APIs is tricky and subtle and there are things that you just don't spot until it is too late, or alternatively there are deficiencies that are invisible for the seasoned user because he is used to the quirks, he internalized them and he uses the API automatically without thinking.

Form rendering and CSS

I tried a few times to learn CSS, but not really seriously, I guess, since I still know only the most basic things. So I am a bit lost now when I need a nice and universal CSS for FormHandler forms in Catalyst::Example::InstantCRUD. Or perhaps I should say HTML::FormHandler::Render::Simple forms instead of just FormHandler forms as theoretically the HTML rendering part is separated from the main FormHandler code, so that if this Simple renderer turns out really too simplistic I can also write a HTML::FormHandler::Render::NotSoSimple to get the right HTML.

I started reviewing the options and it seems that there are three general approaches to having aligned forms:

• tables
  
• divs
  
• lists
  
• no additional markup
  

Plus some appriopriate CSS. I think the 'no additional markup' option could never accommodate for error messages, so I leave this one out. This leaves me with three options. I am sure that not everything can be done in each one - but I think we can add a 'style' parameter to the renderer to make it configurable.

Next thing is the recently popular CSS grids - anyone using them? They seem like a nice option for someone needing some universal defaults and the SenCSS example even contains a form styled in two ways (vertical and horizontal alignments) - but would it work with error messages? And the Blueprint one even shows a div with the error class - but how that is to be inserted into the form? I guess each one requires it's own way of HTML structuring.

Packaging cross cutting Catalyst features

A recurring subject on the Catalyst mailing list is about packaging functionality that is not entirely comprised in one of the Model View or Controller parts of the framework (and is not a full application on itself). This is hard and still there aren't many CPAN libraries that do that. One solution to that is writing "helpers" that generate the integration code into your application files. As with all code generation this solution creates many new problems - and ideally we would like to avoid it.

To try out what can be done about that I decided to write a generic comment system for Catalyst (+DBIC, TT and FormHandler). I've already posted the the first code samples to the Catalyzed wiki. The form code and the controller stuff seems closed - it does not need to refer to things from outside but in the DBIC declarations the belongs_to relation of the comments needs to relate to an external table and it's primary key (I assume that comments belong to an article/post/something). How that can be made configurable?

API design

In the past programming was more about writing from scratch - now code reuse take more and more prominent place in development. In the Perl sphere this is clearly visible in the growth of CPAN, but what is also visible is that this quantity is not everything.

There are many aspects of quality - here I would like to focus on the APIs. The problem with APIs is that their shortcomings are much more subtle than let's say bugs. Many of them are just inconveniences - things that you stop to see once you get accustomed to them, and all of them require some experience or theory just to understand that it can be done better. Arguments about APIs can become dangerously subjective. We are in dire need of API theories, objective measures to evaluate them - and here I try to compile a list of on line resources on that subject.

How To Design A Good API and Why it Matters (from Google Tech Talks)"

- see also an ACM paper with the same title and a nice abstract in the form of bullet points.

My take-away points (the most non-obvious or potentially useful in arguments):

• write examples from the very start and build the API from use cases
  
• good names are important - API is language
  
• "when in doubt leave it out"
  
• implementation somtimes leaks out to the API
  
• for inheritance document self-use pattern (how methods use one another)
  
• "don't make the client do what the module can do"
  
• you can't please everyone so aim to displease everyone equally.
  

The Little Manual of API Design

Characteristics of good design:

• Easy to learn and memorize
  
• Leads to readable code
  
• Hard to misuse
  
• Easy to extend
  
• Complete
  

Again stress on writing use cases and code using the API during the design process - and one surprising but seemingly valid point: "prefer specific names to general names".

API Design Matters

This one is mostly arguing why API design matters - but there are a few points worth noting:

• always design the API from the perspective of the caller (rather obvious)
  
• again start with use cases
  
• there used to be much more programming from scratch - now we use more libraries and designing API becomes more and more important, but this is not yet reflected in university curricula (this is exactly the intuition that led me to this blog post)
  

Designing and Evaluating Reusable Components

The needs of the programmer changes as her project proceeds and it is important to account for that fact when writing the API. In particular more advanced projects usually need more granular APIs while at the beginning they are only an obstacle.

API design guidelines

- my question on Reddit.

Update: Read Curing Python's Neglect - it's not really about Python in my opinion - and it is about how people get used to bad APIs and stop noticing the ugliness of them.

Constraints and Types (in Moose)

Moose::Manual::Type says that Moose types are mostly just constraints with a name. When designing new constraints API for Formhandler we decided to use Moose types - but with some playing around it also exposed the fact that not all constraints are types. I am not an expert in Type Theory - but I checked wikipedia and it seems to support my intuition that type is an attribute of data value - while when validating we often need to check not just the value - but also it's relation to the whole system.

If that is not yet clear - let's take the example of checking the age of a application user (from his provided birth date). The question if a user with a given birth date can legally participate in a web forum cannot be answered without knowing what is current date - it is not a property of the birth date alone - but of it's relation to the current date. I think that using the term 'type' for this kind of constraints can be confusing, even though technically in Moose this is possible to declare this.

Roles for FormHandler

This is a more concrete version of the question from my last post. This is a form defined in FormHandler:

package BookDB::Form::BookWithOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'Author' );

has_field 'title' => ( type => 'Text', required => 1 );
has_field 'publisher' => ( type => 'Text', required => 1 );
has_field 'owner' => ( type => 'BookDB::Form::BookOwner' );

This should be quite self-explanatory for anyone working with forms (and DBIC).
Now - I think Model::DBIC should be just a role here, but that is just a side track. The real question is about the fields here - ideally I would want that doing:

with 'HTML::FormHandler::Model::DBIC';

(this is after it is made a Role) - would add appriopriate methods to the fields - an example here could be validation of fields that need a unique value. I would like to not need to change the definitions above to:

has_field 'title' => ( type => 'Text::DBIC', required => 1 );

Fields like 'owner' above which are Compound fields and work mostly like a form itself and need to load the options for SELECT boxes from the model, set the values of fields from a database row etc:

package BookDB::Form::BookOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Field::Compound';
with 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'User' );

has_field 'user_name';
has_field 'fav_cat' => ( label => 'Favorite Book Category' );
has_field 'fav_book' => ( label => 'Favorite Book' );
has_field 'occupation';
has_field 'country' => ( type => 'Select' );


Ideally this definition should not need 'HTML::FormHandler::Model::DBIC' - applying this role to the root form should be enough.

Roles and data structures

Let's say you have a tree like data structure, something not exactly so uniform as a tree but with enough uniformity that it makes sense that large part of the code is recursive. My example is something representing an HTML Form for editing a complex data structure (with sub-forms etc.). The main form code is used for inflating external data represetntation (key, value pairs) into internal objects and validating it. But there are two additional 'aspects' of this form processing - this is generating the HTML of the form (with all the error indicators and messages) and saving the form data into the data store (by manipulating DBIx::Class schema for example). So you have a tree-like data structure and three separate functionality areas related to it. How do you structure the code?

You can start with defining the data structure and one of the aspects as the 'primary package' - and then add the other aspects as (Moose) roles. This is how it is started in FormHandler - the primary aspect here is the data processing and validation - then saving the the database is added as a role (so you can use it with different data store libraries DBIx::Class, Class::DBI and in the future all the other), the same with rendering - it is a role added to the form class. This is all nice untill you realize that in this way you add the additional roles only to the main form object - but not to the fields (i.e. nodes in the tree) - so you can walk the fields and do the stuff - but you regress here to mostly procedural code.

What we need is to somehow add this role globally to all the field and form classes used.

Hierarchy of code reuse

Hierarchy of code reuse by flexibility:

• Code generation and cut and paste: unconstrained modifications
  
• Inheritance and passing callbacks: modifications only at the method granularity
  
• Normal libraries: no modification of existing algorithms - only composing them into larger constructions
  

Reading "Design Principles and Design Patterns"

Design Patterns are an intriguing idea - there are some attempts to port it to Perl, but I would not call them too sucessfull in spreading it. Design Principles and Design Patterns by Robert C. Martin tries a higher level view on the design patterns by defining some general rules about dependencies and abstraction and then showing how the patterns follow from them. This perhaps should be easier to translate into dynamic languages.

I think the real takeaway is "Stable packages should be abstract packages" - where stable for the author means 'many other packages depend on that one'. The problem with translating that to dynamic languages is that the most abstract (in the OO sense) packages, the pure interfaces don't need to be written down in code at all. Duck typing gives us a great flexibility - but it also means that there is no requirement to write down the abstract interfaces - and if there is no additional pressure then they will not be defined by the programmers.

Another problem with translating the ideas into dynamic languages is the meaning of the word 'depend'. With static typing it can be clearly defined by the need to recompile the package if there is some change in the package it depends on. For dynamic typing this needs to be defined in some more subtle ways - basing it on the type of the coupling between the packages.

Open Source fragmentation and cognitive biases

I've just finished Predictably Irrational (it was nice, many of the of the theses I already knew from Internet articles - so it was not ground-breaking for me, but still I learned from it). In the seventh chapter of this book entitled "The High Price of Ownership" the author presents some surprising stories about how we "overvalue what we have". Apparently we all give much more value to things that we posses then to those that we could potentially have - this is a common cognitive bias.

This made me thinking. Wouldn't that explain the fragmentation of Open Source libraries (and CPAN in particular)? Would that not explain why one author values the miniature efficiency gains of his reinvented wheel over the robustness of time-tested code?

Maybe once we know how this mechanis works, we could design better Open Source library repositories, limitting the power of that fragmentation drive? Maybe by being more explicite about our choices we can become more rational?

Catalyst helpers - plugins for Module::Starter?

Code generation is hard. First of all it is hard to determine what really needs to be created - in (computing science) theory you can always uncurry the generator and use it as a library. So you need different theory to really determine if code generation makes sense at all. In practice many people use it - since it let's us start new projects faster (and more correctly). This is an interesting subject deserving many more blog posts - but for now I just wanted to toss the idea that joining forces with other Perl projects facing this problem could help us improve the mess that is Catalyst helpers now.

I've learned about Module::Starter plugins from Recursive development that leads nowhere.

Progress

Now that I use CGI::Application, Class::DBI and HTML::Template at work I can see the advantages of using Catalyst, DBIx::Class and Template Tookit. Personally, I find using HTML::Template as the most annoying of those three set backs - with TT I've got used that I don't need to worry about displaying the data I feed to the template, I need to make sure that it contains everything needed but that's all. I know that the view can take any data and display it. This makes a nice separation of work and of mind sets between the Controller and the View. Not so with HTML::Template.

One gotcha less!

Looks like one of the anomalies I described in my previous post is fixed now in the svn (see Revision 4773 of DBIC). Thumbs up!

DBIx::Class gotchas

DBIC is a complex beast - and it has it's rather obscure corners where it can burn the inexperienced. Here are two of those, comments with others are welcome.

Don't use $rs->update_or_create(...) (or $rs->find_or_create) on a table with autoincrement primary key

The ->find method requires that all the columns comprising the key used for the search are present in the query or in the resultsource conditions. Otherwise it will return some random record (and emit a warning). This is because currently there is no way to determine if all the needed columns are present. To do this you would have to parse the arbitrary SQL::Abstract query that can be in the resultsource condition and check how the columns are used there. Think for example about:
$new_rs = $cd_rs->search( { -and => [ id => [1, 2], id => { '>' => 1 } ] } )
and then $new_rs->update_or_create( { artist => 1 } ).

But for autoincrement primary keys the ->create method requires that the primary key is not present in the query. If you provide { pk => 'undef' } to ->create, this would generate a query like: INSERT INTO cd ( pk, ...) VALUES ( NULL, ... ) - some databases would accept that but others (PostgreSQL for example) not.

So in summary, for tables with autoincrement primary keys ->find requires that you provide the primary key in the query, and ->create requires that you don't - this means that you cannot pass the same query hash to both methods and this is what ->update_or_create does.

Update: Looks like the 4773 revision fixes the problem described below. That's why I moved this point to the end of the post. I did not check it too deeply though.

Don't call $row->some_relation when the $row has not loaded the columns required to resolve some_relation

Column values in DBIC are kept in a hash - this means that a column can have some defined value, have undefined value or can be completely non-existent - and DBIC interprets all of those states differently. When $row->some_relation is called and ->some_relation uses a foreign key column that is non-existent then the result is currently random (and in the future most probably an exception will be raised).

When this can happen? When you create a row and don't provide the value for the column (or if you retrieve the row from the database and explicitelly omit the column). For example if you do
$cd = $cd_rs->create( { name => "Cisza" } )
(or $cd = $cd_rs->new...). Now the 'artist' column in $cd is non-existent and when you try to call $cd->artist it blows up.
In a library where you don't know how a record was created you need to check first if the relevant columns are loaded before you call a relation on the record.

I thought that a good remedy would be also to always provide undef in the ->create and ->new calls for all the columns that we don't have values for, but allegedly this is not a good strategy (don't ask me why).