Nothing is forever

I have been writing about the scaffolding metaphor - but after some conversations with my colleagues I realized that I did not do a good job there. I did not conduct well the main idea: nothing is forever. No library and no program is forever and we should not shy off from writing libraries that are useful only at some phase of the application construction. It is still valuable if you can quickly get to the point of having a preview, a simplified prototype of the functionality you need even if eventually you'll remove all of that prototype code from the project. This is very 'agile' and it is also 'worse is better (if you can have it early)' and those libraries, those tools that you use only temporarily, I call the scaffolding.

Now one would ask why write a library that is useful only temporarily? Why not include at once all the parts needed in the final product? There can be several reasons:

• something that is only a part of the scaffolding in one project can stay in the final solution in another one
  
• the author of the library might not yet fully understand the problem, and it can take years to reach the right abstraction
  
• the distribution of possibly needed modification can be so uniform that there is no reason to concentrate on one of them more than on another one and it is virtually impossible to cover them all
  
• and finally: nothing is forever - you never know where a project will go in the future and what you'll need to change
  

A good example (of the third point) are code samples from manuals - you copy them to quickly get something working - but later you modify it and sometimes you change every character from the original. Another one can be the scaffolding code in Rails - you can debate if it fits the first or the second or other points - but it was useful to a lot of people.

Every library can be used as scaffolding - so why not accept this and help the developers in using it that way?

CPAN and code reuse

CPAN is wonderful thing - it is a great achievement of the whole Perl community and the thing that makes it apart from other languages. But somehow it works mostly for libraries. And what about the other modes of code reuse? How about CPAN for code examples? One can say we don't need CPAN for code examples - everyone is using code examples all the time and they do that without any special infrastructure. But what I see around is that some of CPAN modules are actually code examples in disguise. Some of them would be better without any code in the package and only documentation and tests validating the documented techniques. Hmm maybe we don't really need a separate CPAN for that - maybe we can keep them at the existing one - and just mark them in a special way? And maybe we need some social acceptance for packaging such 'empty' libraries?

And what with applications?

Code reuse styles

There is much buzz around PHP tools like Drupal and Joomla. They are ready-made solutions, but in fact there is also a lot of tweaking the templates and writing add-ons. It seems to work very well - the clients get something usable from the beginning - and later they can modify it to fit their purpose better. This is very agile. In contrast the reuse in Perl is mostly around libraries. This is much more flexible - you don't tweak a ready solution - you assemble it from the thousands of CPAN packages. But it is also less agile.

I wonder why it is like that. There are some technical differences that lead developers in one or the other direction - but there is also a cultural/psychological cause. Doing some self-analysis I must admit that I am not really enthusiastic about contributing to a Perl CMS (like for example Bricolage) - in my view all the already-build solutions use obsolete libraries. I'd rather contribute to building the perfect library - then code an extension to Bricloage. I have the feeling that I am not alone in that crazy perfectionism around Perl hackers. PHP clearly wins here with more pragmatic approach.

Deprecated code analyzer for HTML::FormHandler

So the other day I've read Deprecated code analyzer for perl and I thought - this is a GREAT idea! And I stole it for HTML::FormHandler. Now with new releases our users will have a tool to find all the deprecated API calls in their code.

Is using the Catalyst 'forward' method a cargo cult?

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:

$c->forward( 'some_action' )

and the standard Perl:

$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.

Functional versus Unit testing

Benefits of automated functional testing (was: Why unit testing is a waste of time) is a comprehensive list of arguments for functional and against unit testing. It is a bit long and maybe it is preaching to the choir as I have never seen unit testing in CPAN modules, but it might be handy if you needed to convince your co-workers.

Also discussed at Hacker News

InstantCRUD and FormHandler

The new version of Catalyst::Example::InstantCRUD is on CPAN. This version uses FormHandler, is a bit nicer visually (I hope) and has a proof of concept experimental REST interface.

Iron Man Blogging Challenge

First I need to confirm that I really like the idea. Perl needs more publicity. Some may say that quality is more important than quantity but there is no better way to improve your writing skills than practice. So yeah - it's a great idea. In general I think we need more competitions - but that is really for another post.

There is just one small nitpick - if the goal is: "to promote the language to the world outside the Perl echo-chamber" then the rule that the all the posts need to be about Perl is misguided. If we want to talk to "the world outside the Perl echo-chamber" then we need to talk about the subjects that they care about. I can understand why we need some focusing - and posts about what you had for dinner are not the best way to promote the language - but it will be much easier to engage the outside audience if we wrote about general programming subjects. Look at advertisings - they don't talk about the advertised brand all the time, the more subtle the link is, the more engaging they are.

Update: I should have added that this post was inspired by Gabor Szabo: http://szabgab.com/blog/2009/06/1245788806.html

CatalystX::Comments - RFC

In Packaging cross cutting Catalyst features I promised to start developing a generic Catalyst comment sub-system. Now I have incorporated the code samples into another experiment - ravlog(a blog engine by Gerda), and I would like to ask you what you think about it before releasing it to CPAN.

Here is the current API - any comments are welcome. First you need to declare the controller using the new Moose style:

package RavLog::Controller::View;

use Moose;
BEGIN {
extends 'Catalyst::Controller';
with 'CatalystX::Comments::ControllerFormRole';
}

has model_name => ( is => 'ro', isa => 'Str', default => 'DB' );


The important part for us is of course with 'CatalystX::Comments::ControllerFormRole', the model_name attribute should hold the name of the model CatalystX::Comments will use to store the comments.
After the declarations you can use CatalystX::Comments to put the comment form on the stash:

sub view : Chained('base') PathPart('view') Args(0)
{
my ( $self, $c, $id ) = @_;

...

$self->stash_comment_form( $c, $self->article->id );
}

That's the whole controller part - what it handles is generating the form for display and saving the submitted comments. In the view templates you need to add:

[% comment_form.render %]

For displaying the comments themselves ravlog already had some template code and I did not change it.

Next is the model part. The library assumes that the DBIC model contains a Comment Result class like that one in ravlog. Of course it does not need to have all of the columns, and can also have other columns - but these are the columns that will be filled by the CatalystX::Comments form. Ideally that model part should be a Result base class - so that it can be subclassed and adapted to local circumstances (for example the belongs_to relationship to article needs to be changed), but for now I don't know how to do that.

Now some more details. Thanks to html_prefix all parameters sent from this form are prefixed with the form name - so this form can be added to pages with other forms, it will recognize it's parameters. The comments are saved only when the HTTP method used is POST, and after a successful comment creation the user is redirected to the same page (this seems a bit constraining - but see next paragraph - this is only meant to be temporary solution - I try to keep the API simple).

Of course I understand that this can never cover the needs of a comment sub-system in a mature social web site. I am thinking about it as more of scaffolding - code that let's you quickly develop a feature, see how it integrates with the rest of the user experience, formulate a more complete requirements list - and replace it part by part.

On Scaffoldings

Scaffolding is a wonderful metaphor. Was it invented in Ruby on Rails? I am not sure about that - but nevertheless it is RoR that popularized it and undoubtedly it was one of the engines of it's popularity. It is very "Agile" - like a programming sprint - it implementa a feature in the most simplistic way that can possibly work - but instead of a programming team doing the work the code for that feature is generated.

Scaffolding is a wonderful metaphor - and the brilliance of it shows in how it transcends the original area of code generation where it was introduced in Rails. It can be applicable to any reusable software component meant to be temporary support and to be replaced later. An example of that are code samples in manuals - you copy them just to have something working, but then you modify it and nearly completely replace the original code. And since nothing is universal and no library is ideal - it can be used as a generic measure of how well a library supports this iterative development style. And this brings me to the question - what are the qualities of a good scaffolding? One is that it needs to be gradually replaceable.

Module Rank and Informed Choice

There is much talk recently about CPAN and how to improve it. I firmly believe that initiatives like CPAN Top 100 will eventually lead to some good, Page Rank like, measures of if not module quality then at least module utility. People vote with their legs - they use other modules in their own productions and we don't need any other voting mechanisms. After all Google does quite well with Page Rank, much better than the human edited Internet Directories (like the original Yahoo or Open Directory).

But it all still relies on people choosing the right modules for their own projects - I believe this is a good assumption, the popular "The Wisdom of Crowds" shows why - but still it can be improved by people doing more informed choice. This was my motivation behind my API design blog post. Design of APIs is tricky and subtle and there are things that you just don't spot until it is too late, or alternatively there are deficiencies that are invisible for the seasoned user because he is used to the quirks, he internalized them and he uses the API automatically without thinking.

Form rendering and CSS

I tried a few times to learn CSS, but not really seriously, I guess, since I still know only the most basic things. So I am a bit lost now when I need a nice and universal CSS for FormHandler forms in Catalyst::Example::InstantCRUD. Or perhaps I should say HTML::FormHandler::Render::Simple forms instead of just FormHandler forms as theoretically the HTML rendering part is separated from the main FormHandler code, so that if this Simple renderer turns out really too simplistic I can also write a HTML::FormHandler::Render::NotSoSimple to get the right HTML.

I started reviewing the options and it seems that there are three general approaches to having aligned forms:

• tables
  
• divs
  
• lists
  
• no additional markup
  

Plus some appriopriate CSS. I think the 'no additional markup' option could never accommodate for error messages, so I leave this one out. This leaves me with three options. I am sure that not everything can be done in each one - but I think we can add a 'style' parameter to the renderer to make it configurable.

Next thing is the recently popular CSS grids - anyone using them? They seem like a nice option for someone needing some universal defaults and the SenCSS example even contains a form styled in two ways (vertical and horizontal alignments) - but would it work with error messages? And the Blueprint one even shows a div with the error class - but how that is to be inserted into the form? I guess each one requires it's own way of HTML structuring.

Packaging cross cutting Catalyst features

A recurring subject on the Catalyst mailing list is about packaging functionality that is not entirely comprised in one of the Model View or Controller parts of the framework (and is not a full application on itself). This is hard and still there aren't many CPAN libraries that do that. One solution to that is writing "helpers" that generate the integration code into your application files. As with all code generation this solution creates many new problems - and ideally we would like to avoid it.

To try out what can be done about that I decided to write a generic comment system for Catalyst (+DBIC, TT and FormHandler). I've already posted the the first code samples to the Catalyzed wiki. The form code and the controller stuff seems closed - it does not need to refer to things from outside but in the DBIC declarations the belongs_to relation of the comments needs to relate to an external table and it's primary key (I assume that comments belong to an article/post/something). How that can be made configurable?

API design

In the past programming was more about writing from scratch - now code reuse take more and more prominent place in development. In the Perl sphere this is clearly visible in the growth of CPAN, but what is also visible is that this quantity is not everything.

There are many aspects of quality - here I would like to focus on the APIs. The problem with APIs is that their shortcomings are much more subtle than let's say bugs. Many of them are just inconveniences - things that you stop to see once you get accustomed to them, and all of them require some experience or theory just to understand that it can be done better. Arguments about APIs can become dangerously subjective. We are in dire need of API theories, objective measures to evaluate them - and here I try to compile a list of on line resources on that subject.

How To Design A Good API and Why it Matters (from Google Tech Talks)"

- see also an ACM paper with the same title and a nice abstract in the form of bullet points.

My take-away points (the most non-obvious or potentially useful in arguments):

• write examples from the very start and build the API from use cases
  
• good names are important - API is language
  
• "when in doubt leave it out"
  
• implementation somtimes leaks out to the API
  
• for inheritance document self-use pattern (how methods use one another)
  
• "don't make the client do what the module can do"
  
• you can't please everyone so aim to displease everyone equally.
  

The Little Manual of API Design

Characteristics of good design:

• Easy to learn and memorize
  
• Leads to readable code
  
• Hard to misuse
  
• Easy to extend
  
• Complete
  

Again stress on writing use cases and code using the API during the design process - and one surprising but seemingly valid point: "prefer specific names to general names".

API Design Matters

This one is mostly arguing why API design matters - but there are a few points worth noting:

• always design the API from the perspective of the caller (rather obvious)
  
• again start with use cases
  
• there used to be much more programming from scratch - now we use more libraries and designing API becomes more and more important, but this is not yet reflected in university curricula (this is exactly the intuition that led me to this blog post)
  

Designing and Evaluating Reusable Components

The needs of the programmer changes as her project proceeds and it is important to account for that fact when writing the API. In particular more advanced projects usually need more granular APIs while at the beginning they are only an obstacle.

API design guidelines

- my question on Reddit.

Update: Read Curing Python's Neglect - it's not really about Python in my opinion - and it is about how people get used to bad APIs and stop noticing the ugliness of them.

Constraints and Types (in Moose)

Moose::Manual::Type says that Moose types are mostly just constraints with a name. When designing new constraints API for Formhandler we decided to use Moose types - but with some playing around it also exposed the fact that not all constraints are types. I am not an expert in Type Theory - but I checked wikipedia and it seems to support my intuition that type is an attribute of data value - while when validating we often need to check not just the value - but also it's relation to the whole system.

If that is not yet clear - let's take the example of checking the age of a application user (from his provided birth date). The question if a user with a given birth date can legally participate in a web forum cannot be answered without knowing what is current date - it is not a property of the birth date alone - but of it's relation to the current date. I think that using the term 'type' for this kind of constraints can be confusing, even though technically in Moose this is possible to declare this.

Roles for FormHandler

This is a more concrete version of the question from my last post. This is a form defined in FormHandler:

package BookDB::Form::BookWithOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'Author' );

has_field 'title' => ( type => 'Text', required => 1 );
has_field 'publisher' => ( type => 'Text', required => 1 );
has_field 'owner' => ( type => 'BookDB::Form::BookOwner' );

This should be quite self-explanatory for anyone working with forms (and DBIC).
Now - I think Model::DBIC should be just a role here, but that is just a side track. The real question is about the fields here - ideally I would want that doing:

with 'HTML::FormHandler::Model::DBIC';

(this is after it is made a Role) - would add appriopriate methods to the fields - an example here could be validation of fields that need a unique value. I would like to not need to change the definitions above to:

has_field 'title' => ( type => 'Text::DBIC', required => 1 );

Fields like 'owner' above which are Compound fields and work mostly like a form itself and need to load the options for SELECT boxes from the model, set the values of fields from a database row etc:

package BookDB::Form::BookOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Field::Compound';
with 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'User' );

has_field 'user_name';
has_field 'fav_cat' => ( label => 'Favorite Book Category' );
has_field 'fav_book' => ( label => 'Favorite Book' );
has_field 'occupation';
has_field 'country' => ( type => 'Select' );


Ideally this definition should not need 'HTML::FormHandler::Model::DBIC' - applying this role to the root form should be enough.

Roles and data structures

Let's say you have a tree like data structure, something not exactly so uniform as a tree but with enough uniformity that it makes sense that large part of the code is recursive. My example is something representing an HTML Form for editing a complex data structure (with sub-forms etc.). The main form code is used for inflating external data represetntation (key, value pairs) into internal objects and validating it. But there are two additional 'aspects' of this form processing - this is generating the HTML of the form (with all the error indicators and messages) and saving the form data into the data store (by manipulating DBIx::Class schema for example). So you have a tree-like data structure and three separate functionality areas related to it. How do you structure the code?

You can start with defining the data structure and one of the aspects as the 'primary package' - and then add the other aspects as (Moose) roles. This is how it is started in FormHandler - the primary aspect here is the data processing and validation - then saving the the database is added as a role (so you can use it with different data store libraries DBIx::Class, Class::DBI and in the future all the other), the same with rendering - it is a role added to the form class. This is all nice untill you realize that in this way you add the additional roles only to the main form object - but not to the fields (i.e. nodes in the tree) - so you can walk the fields and do the stuff - but you regress here to mostly procedural code.

What we need is to somehow add this role globally to all the field and form classes used.

Hierarchy of code reuse

Hierarchy of code reuse by flexibility:

• Code generation and cut and paste: unconstrained modifications
  
• Inheritance and passing callbacks: modifications only at the method granularity
  
• Normal libraries: no modification of existing algorithms - only composing them into larger constructions
  

Reading "Design Principles and Design Patterns"

Design Patterns are an intriguing idea - there are some attempts to port it to Perl, but I would not call them too sucessfull in spreading it. Design Principles and Design Patterns by Robert C. Martin tries a higher level view on the design patterns by defining some general rules about dependencies and abstraction and then showing how the patterns follow from them. This perhaps should be easier to translate into dynamic languages.

I think the real takeaway is "Stable packages should be abstract packages" - where stable for the author means 'many other packages depend on that one'. The problem with translating that to dynamic languages is that the most abstract (in the OO sense) packages, the pure interfaces don't need to be written down in code at all. Duck typing gives us a great flexibility - but it also means that there is no requirement to write down the abstract interfaces - and if there is no additional pressure then they will not be defined by the programmers.

Another problem with translating the ideas into dynamic languages is the meaning of the word 'depend'. With static typing it can be clearly defined by the need to recompile the package if there is some change in the package it depends on. For dynamic typing this needs to be defined in some more subtle ways - basing it on the type of the coupling between the packages.

Open Source fragmentation and cognitive biases

I've just finished Predictably Irrational (it was nice, many of the of the theses I already knew from Internet articles - so it was not ground-breaking for me, but still I learned from it). In the seventh chapter of this book entitled "The High Price of Ownership" the author presents some surprising stories about how we "overvalue what we have". Apparently we all give much more value to things that we posses then to those that we could potentially have - this is a common cognitive bias.

This made me thinking. Wouldn't that explain the fragmentation of Open Source libraries (and CPAN in particular)? Would that not explain why one author values the miniature efficiency gains of his reinvented wheel over the robustness of time-tested code?

Maybe once we know how this mechanis works, we could design better Open Source library repositories, limitting the power of that fragmentation drive? Maybe by being more explicite about our choices we can become more rational?