Fixing dependencies

Fixing other people code is not easy - but this is what we need to do if we want to clean our dependencies. In answer to John's invitation here is what I did.

I wanted to write some code using Catalyst::Authentication::Credential::OpenID unfortunately one of it's dependencies fails it's tests. Here I'll document how I analysed that and produced a patch, it's not yet approved by the author - but the tests pass.

Step first - checkout the ParanoidAged from it's repository (I had the repository address from an email to the Catalyst mailing list). If you don't have access to the repository of your failing dependency - then just download the source from CPAN unpack it and work from there. Then run the tests:

zby@zby:~/progs/pa$ perl -Ilib t/00-all.t
.
.
.
ok 27
5 second tarpit (tolerance 2)...
not ok 28
# Failed test at t/00-all.t line 180.
3 second tarpit (tolerance 4)...
ok 29
Killing child pid: 7669
# Looks like you failed 1 test of 29.

OK - the failing test is at line 180. Let's see what is there:

I changed that to:

ran the tests and save my debug output:
zby@zby:~/progs/pa$ perl t/00-all.t 2>debug.
What I got there is:



Now the info is there - but of course I did not spot it at once. I did a lot of more debugging and testing other hypothesis before I focused on: 'x-died' => 'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394.'. Yeah interesting - something died, but no error was reported. The line in question is:



To get some more evidence I changed that to:



Now the debug output changes to:



And again - it is there - but to spot it I had to play with the debugger first. I don't remember what exactly I did - but finally I sow it - there are two evals in this stack trace, the first one is in LWP::Protocol and the second in lib/LWPx/ParanoidAgent.pm line 313. I checked that second eval - and yeah - it tried to retrieve the error already catched by the first one. So when there was a timeout the agent was aborting the retrieval - but the error was being cleared and not reported. I wrote following patch:



and now all the tests pass. Finally I uploaded the patch to RT.

On dependencies

Ovid writes about Mojolicious::Lite:

The fact that it has no dependencies makes it even more compelling.It's the sort of thing one could more easily attract new developers with. Can you imagine rewriting my old CGI course with something like this? Or maybe a way to introduce new users to Perl? There are many excellent Perl frameworks out there which I would, nonetheless, hesitate to start new users on. Just trying to set up CPAN can be difficult for them; working through even one test failure is often too much. For people who want to jump in and learn "Web programming", though, this might work.

On the other hand Jay advices to stop worrying about dependencies The lie of independence - or - how I learned to stop worrying and love the dependency chain. There is truth in both statements. I have the impression that the problem of CPAN dependencies have improved much recently. The visibility of installation problems directly from the cpan search engine (cpan testers matrix, dependencies review) is a great step ahead. But still installing Catalyst and DBIC and TT is a big hurdle - and I can understand why Ovid is not enthusiastic about teaching Perl novices how to do that.

For me Mojo sounds like iPod - slick and beautiful and really useful - but you are not supposed to change the battery. It is a nicely integrated product - but somehow closed. The author apparently tries to make it perfect, tries to polish every detail - what perhaps would not be possible if he had to compromise with the visions of the authors of the other packages. I can feel his frustration here. But rejecting all dependencies and rewriting everything in his own way sounds a bit extreme and not welcoming.

The big question here is how much can we trust fellow CPAN authors to make the right decisions, can we rely on their work? Can someone, who put's a lot of attention for the details of his API, rely on modules with imperfect APIs, perhaps spoiling his own work and not become insane? The story of the phalanx project shows that there are no easy answers to these questions, but I believe we can do better than rewriting everything from scratch each time.

Google Wave

One, sometimes feverishly defended, rule of our technical irc channels is about not pasting code into the conversation. To show your code you are supposed to use some web based paste application and then copy the link into the channel. The need to use two separate tools, irc client and web browser, for one task, code review, is just a small annoyance - but eliminating those small annoyances is the only way towards really usable software. Many people have internalized the procedure so much, that they don't think about it any more and I am sure there will be some of them arguing that this is not annoying at all. But there are additional steps to do: choose the tool, paste the link, switch the windows and all this is a burden to the brain, even if it is processed in the background - it still consumes some brain resources.

And now imagine that you have it in one tool - not only those additional steps go away - but it also opens the way for new features like seeing a pointer to the part of the code that the other person is talking about, seeing his edits in real-time and commenting on them immediately etc.

This is why I believe in convergence of communication tools.

Nothing is forever

I have been writing about the scaffolding metaphor - but after some conversations with my colleagues I realized that I did not do a good job there. I did not conduct well the main idea: nothing is forever. No library and no program is forever and we should not shy off from writing libraries that are useful only at some phase of the application construction. It is still valuable if you can quickly get to the point of having a preview, a simplified prototype of the functionality you need even if eventually you'll remove all of that prototype code from the project. This is very 'agile' and it is also 'worse is better (if you can have it early)' and those libraries, those tools that you use only temporarily, I call the scaffolding.

Now one would ask why write a library that is useful only temporarily? Why not include at once all the parts needed in the final product? There can be several reasons:

• something that is only a part of the scaffolding in one project can stay in the final solution in another one
  
• the author of the library might not yet fully understand the problem, and it can take years to reach the right abstraction
  
• the distribution of possibly needed modification can be so uniform that there is no reason to concentrate on one of them more than on another one and it is virtually impossible to cover them all
  
• and finally: nothing is forever - you never know where a project will go in the future and what you'll need to change
  

A good example (of the third point) are code samples from manuals - you copy them to quickly get something working - but later you modify it and sometimes you change every character from the original. Another one can be the scaffolding code in Rails - you can debate if it fits the first or the second or other points - but it was useful to a lot of people.

Every library can be used as scaffolding - so why not accept this and help the developers in using it that way?

CPAN and code reuse

CPAN is wonderful thing - it is a great achievement of the whole Perl community and the thing that makes it apart from other languages. But somehow it works mostly for libraries. And what about the other modes of code reuse? How about CPAN for code examples? One can say we don't need CPAN for code examples - everyone is using code examples all the time and they do that without any special infrastructure. But what I see around is that some of CPAN modules are actually code examples in disguise. Some of them would be better without any code in the package and only documentation and tests validating the documented techniques. Hmm maybe we don't really need a separate CPAN for that - maybe we can keep them at the existing one - and just mark them in a special way? And maybe we need some social acceptance for packaging such 'empty' libraries?

And what with applications?

Code reuse styles

There is much buzz around PHP tools like Drupal and Joomla. They are ready-made solutions, but in fact there is also a lot of tweaking the templates and writing add-ons. It seems to work very well - the clients get something usable from the beginning - and later they can modify it to fit their purpose better. This is very agile. In contrast the reuse in Perl is mostly around libraries. This is much more flexible - you don't tweak a ready solution - you assemble it from the thousands of CPAN packages. But it is also less agile.

I wonder why it is like that. There are some technical differences that lead developers in one or the other direction - but there is also a cultural/psychological cause. Doing some self-analysis I must admit that I am not really enthusiastic about contributing to a Perl CMS (like for example Bricolage) - in my view all the already-build solutions use obsolete libraries. I'd rather contribute to building the perfect library - then code an extension to Bricloage. I have the feeling that I am not alone in that crazy perfectionism around Perl hackers. PHP clearly wins here with more pragmatic approach.

Deprecated code analyzer for HTML::FormHandler

So the other day I've read Deprecated code analyzer for perl and I thought - this is a GREAT idea! And I stole it for HTML::FormHandler. Now with new releases our users will have a tool to find all the deprecated API calls in their code.

Is using the Catalyst 'forward' method a cargo cult?

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:

$c->forward( 'some_action' )

and the standard Perl:

$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.

Functional versus Unit testing

Benefits of automated functional testing (was: Why unit testing is a waste of time) is a comprehensive list of arguments for functional and against unit testing. It is a bit long and maybe it is preaching to the choir as I have never seen unit testing in CPAN modules, but it might be handy if you needed to convince your co-workers.

Also discussed at Hacker News

InstantCRUD and FormHandler

The new version of Catalyst::Example::InstantCRUD is on CPAN. This version uses FormHandler, is a bit nicer visually (I hope) and has a proof of concept experimental REST interface.

Iron Man Blogging Challenge

First I need to confirm that I really like the idea. Perl needs more publicity. Some may say that quality is more important than quantity but there is no better way to improve your writing skills than practice. So yeah - it's a great idea. In general I think we need more competitions - but that is really for another post.

There is just one small nitpick - if the goal is: "to promote the language to the world outside the Perl echo-chamber" then the rule that the all the posts need to be about Perl is misguided. If we want to talk to "the world outside the Perl echo-chamber" then we need to talk about the subjects that they care about. I can understand why we need some focusing - and posts about what you had for dinner are not the best way to promote the language - but it will be much easier to engage the outside audience if we wrote about general programming subjects. Look at advertisings - they don't talk about the advertised brand all the time, the more subtle the link is, the more engaging they are.

Update: I should have added that this post was inspired by Gabor Szabo: http://szabgab.com/blog/2009/06/1245788806.html

CatalystX::Comments - RFC

In Packaging cross cutting Catalyst features I promised to start developing a generic Catalyst comment sub-system. Now I have incorporated the code samples into another experiment - ravlog(a blog engine by Gerda), and I would like to ask you what you think about it before releasing it to CPAN.

Here is the current API - any comments are welcome. First you need to declare the controller using the new Moose style:

package RavLog::Controller::View;

use Moose;
BEGIN {
extends 'Catalyst::Controller';
with 'CatalystX::Comments::ControllerFormRole';
}

has model_name => ( is => 'ro', isa => 'Str', default => 'DB' );


The important part for us is of course with 'CatalystX::Comments::ControllerFormRole', the model_name attribute should hold the name of the model CatalystX::Comments will use to store the comments.
After the declarations you can use CatalystX::Comments to put the comment form on the stash:

sub view : Chained('base') PathPart('view') Args(0)
{
my ( $self, $c, $id ) = @_;

...

$self->stash_comment_form( $c, $self->article->id );
}

That's the whole controller part - what it handles is generating the form for display and saving the submitted comments. In the view templates you need to add:

[% comment_form.render %]

For displaying the comments themselves ravlog already had some template code and I did not change it.

Next is the model part. The library assumes that the DBIC model contains a Comment Result class like that one in ravlog. Of course it does not need to have all of the columns, and can also have other columns - but these are the columns that will be filled by the CatalystX::Comments form. Ideally that model part should be a Result base class - so that it can be subclassed and adapted to local circumstances (for example the belongs_to relationship to article needs to be changed), but for now I don't know how to do that.

Now some more details. Thanks to html_prefix all parameters sent from this form are prefixed with the form name - so this form can be added to pages with other forms, it will recognize it's parameters. The comments are saved only when the HTTP method used is POST, and after a successful comment creation the user is redirected to the same page (this seems a bit constraining - but see next paragraph - this is only meant to be temporary solution - I try to keep the API simple).

Of course I understand that this can never cover the needs of a comment sub-system in a mature social web site. I am thinking about it as more of scaffolding - code that let's you quickly develop a feature, see how it integrates with the rest of the user experience, formulate a more complete requirements list - and replace it part by part.

On Scaffoldings

Scaffolding is a wonderful metaphor. Was it invented in Ruby on Rails? I am not sure about that - but nevertheless it is RoR that popularized it and undoubtedly it was one of the engines of it's popularity. It is very "Agile" - like a programming sprint - it implementa a feature in the most simplistic way that can possibly work - but instead of a programming team doing the work the code for that feature is generated.

Scaffolding is a wonderful metaphor - and the brilliance of it shows in how it transcends the original area of code generation where it was introduced in Rails. It can be applicable to any reusable software component meant to be temporary support and to be replaced later. An example of that are code samples in manuals - you copy them just to have something working, but then you modify it and nearly completely replace the original code. And since nothing is universal and no library is ideal - it can be used as a generic measure of how well a library supports this iterative development style. And this brings me to the question - what are the qualities of a good scaffolding? One is that it needs to be gradually replaceable.

Module Rank and Informed Choice

There is much talk recently about CPAN and how to improve it. I firmly believe that initiatives like CPAN Top 100 will eventually lead to some good, Page Rank like, measures of if not module quality then at least module utility. People vote with their legs - they use other modules in their own productions and we don't need any other voting mechanisms. After all Google does quite well with Page Rank, much better than the human edited Internet Directories (like the original Yahoo or Open Directory).

But it all still relies on people choosing the right modules for their own projects - I believe this is a good assumption, the popular "The Wisdom of Crowds" shows why - but still it can be improved by people doing more informed choice. This was my motivation behind my API design blog post. Design of APIs is tricky and subtle and there are things that you just don't spot until it is too late, or alternatively there are deficiencies that are invisible for the seasoned user because he is used to the quirks, he internalized them and he uses the API automatically without thinking.

Form rendering and CSS

I tried a few times to learn CSS, but not really seriously, I guess, since I still know only the most basic things. So I am a bit lost now when I need a nice and universal CSS for FormHandler forms in Catalyst::Example::InstantCRUD. Or perhaps I should say HTML::FormHandler::Render::Simple forms instead of just FormHandler forms as theoretically the HTML rendering part is separated from the main FormHandler code, so that if this Simple renderer turns out really too simplistic I can also write a HTML::FormHandler::Render::NotSoSimple to get the right HTML.

I started reviewing the options and it seems that there are three general approaches to having aligned forms:

• tables
  
• divs
  
• lists
  
• no additional markup
  

Plus some appriopriate CSS. I think the 'no additional markup' option could never accommodate for error messages, so I leave this one out. This leaves me with three options. I am sure that not everything can be done in each one - but I think we can add a 'style' parameter to the renderer to make it configurable.

Next thing is the recently popular CSS grids - anyone using them? They seem like a nice option for someone needing some universal defaults and the SenCSS example even contains a form styled in two ways (vertical and horizontal alignments) - but would it work with error messages? And the Blueprint one even shows a div with the error class - but how that is to be inserted into the form? I guess each one requires it's own way of HTML structuring.

Packaging cross cutting Catalyst features

A recurring subject on the Catalyst mailing list is about packaging functionality that is not entirely comprised in one of the Model View or Controller parts of the framework (and is not a full application on itself). This is hard and still there aren't many CPAN libraries that do that. One solution to that is writing "helpers" that generate the integration code into your application files. As with all code generation this solution creates many new problems - and ideally we would like to avoid it.

To try out what can be done about that I decided to write a generic comment system for Catalyst (+DBIC, TT and FormHandler). I've already posted the the first code samples to the Catalyzed wiki. The form code and the controller stuff seems closed - it does not need to refer to things from outside but in the DBIC declarations the belongs_to relation of the comments needs to relate to an external table and it's primary key (I assume that comments belong to an article/post/something). How that can be made configurable?

API design

In the past programming was more about writing from scratch - now code reuse take more and more prominent place in development. In the Perl sphere this is clearly visible in the growth of CPAN, but what is also visible is that this quantity is not everything.

There are many aspects of quality - here I would like to focus on the APIs. The problem with APIs is that their shortcomings are much more subtle than let's say bugs. Many of them are just inconveniences - things that you stop to see once you get accustomed to them, and all of them require some experience or theory just to understand that it can be done better. Arguments about APIs can become dangerously subjective. We are in dire need of API theories, objective measures to evaluate them - and here I try to compile a list of on line resources on that subject.

How To Design A Good API and Why it Matters (from Google Tech Talks)"

- see also an ACM paper with the same title and a nice abstract in the form of bullet points.

My take-away points (the most non-obvious or potentially useful in arguments):

• write examples from the very start and build the API from use cases
  
• good names are important - API is language
  
• "when in doubt leave it out"
  
• implementation somtimes leaks out to the API
  
• for inheritance document self-use pattern (how methods use one another)
  
• "don't make the client do what the module can do"
  
• you can't please everyone so aim to displease everyone equally.
  

The Little Manual of API Design

Characteristics of good design:

• Easy to learn and memorize
  
• Leads to readable code
  
• Hard to misuse
  
• Easy to extend
  
• Complete
  

Again stress on writing use cases and code using the API during the design process - and one surprising but seemingly valid point: "prefer specific names to general names".

API Design Matters

This one is mostly arguing why API design matters - but there are a few points worth noting:

• always design the API from the perspective of the caller (rather obvious)
  
• again start with use cases
  
• there used to be much more programming from scratch - now we use more libraries and designing API becomes more and more important, but this is not yet reflected in university curricula (this is exactly the intuition that led me to this blog post)
  

Designing and Evaluating Reusable Components

The needs of the programmer changes as her project proceeds and it is important to account for that fact when writing the API. In particular more advanced projects usually need more granular APIs while at the beginning they are only an obstacle.

API design guidelines

- my question on Reddit.

Update: Read Curing Python's Neglect - it's not really about Python in my opinion - and it is about how people get used to bad APIs and stop noticing the ugliness of them.

Constraints and Types (in Moose)

Moose::Manual::Type says that Moose types are mostly just constraints with a name. When designing new constraints API for Formhandler we decided to use Moose types - but with some playing around it also exposed the fact that not all constraints are types. I am not an expert in Type Theory - but I checked wikipedia and it seems to support my intuition that type is an attribute of data value - while when validating we often need to check not just the value - but also it's relation to the whole system.

If that is not yet clear - let's take the example of checking the age of a application user (from his provided birth date). The question if a user with a given birth date can legally participate in a web forum cannot be answered without knowing what is current date - it is not a property of the birth date alone - but of it's relation to the current date. I think that using the term 'type' for this kind of constraints can be confusing, even though technically in Moose this is possible to declare this.

Roles for FormHandler

This is a more concrete version of the question from my last post. This is a form defined in FormHandler:

package BookDB::Form::BookWithOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'Author' );

has_field 'title' => ( type => 'Text', required => 1 );
has_field 'publisher' => ( type => 'Text', required => 1 );
has_field 'owner' => ( type => 'BookDB::Form::BookOwner' );

This should be quite self-explanatory for anyone working with forms (and DBIC).
Now - I think Model::DBIC should be just a role here, but that is just a side track. The real question is about the fields here - ideally I would want that doing:

with 'HTML::FormHandler::Model::DBIC';

(this is after it is made a Role) - would add appriopriate methods to the fields - an example here could be validation of fields that need a unique value. I would like to not need to change the definitions above to:

has_field 'title' => ( type => 'Text::DBIC', required => 1 );

Fields like 'owner' above which are Compound fields and work mostly like a form itself and need to load the options for SELECT boxes from the model, set the values of fields from a database row etc:

package BookDB::Form::BookOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Field::Compound';
with 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'User' );

has_field 'user_name';
has_field 'fav_cat' => ( label => 'Favorite Book Category' );
has_field 'fav_book' => ( label => 'Favorite Book' );
has_field 'occupation';
has_field 'country' => ( type => 'Select' );


Ideally this definition should not need 'HTML::FormHandler::Model::DBIC' - applying this role to the root form should be enough.

Roles and data structures

Let's say you have a tree like data structure, something not exactly so uniform as a tree but with enough uniformity that it makes sense that large part of the code is recursive. My example is something representing an HTML Form for editing a complex data structure (with sub-forms etc.). The main form code is used for inflating external data represetntation (key, value pairs) into internal objects and validating it. But there are two additional 'aspects' of this form processing - this is generating the HTML of the form (with all the error indicators and messages) and saving the form data into the data store (by manipulating DBIx::Class schema for example). So you have a tree-like data structure and three separate functionality areas related to it. How do you structure the code?

You can start with defining the data structure and one of the aspects as the 'primary package' - and then add the other aspects as (Moose) roles. This is how it is started in FormHandler - the primary aspect here is the data processing and validation - then saving the the database is added as a role (so you can use it with different data store libraries DBIx::Class, Class::DBI and in the future all the other), the same with rendering - it is a role added to the form class. This is all nice untill you realize that in this way you add the additional roles only to the main form object - but not to the fields (i.e. nodes in the tree) - so you can walk the fields and do the stuff - but you regress here to mostly procedural code.

What we need is to somehow add this role globally to all the field and form classes used.