Perl Alchemy - notes of a former Catalyst programmer

Schlep

2012-01-14T23:48:00.000-08:00

Schlep is tedious, unpleasant task. According to Paul Graham schlep is also what really defines a company - it is doing the tasks that are unpleasant and tedious for someone that they would pay you for.

Narrowing this down to my own Perl web development work - the schlep for me was always getting the basic web app running with user registration, login pages, password reset mechanisms, etc. - in every new project that was the most repeatable, boring work. I think everyone has the feeling that this does not need to be like that. I've started thinking about what could be a solution to this and here are my first experiments about fixing it: Plack-Middleware-Auth-Form, WebPrototypes::ResetPass, WebPrototypes::Registration (I might rename the first one to WebPrototypes as well). The point is to solve it across the multiple web frameworks, templating languages and storage layers - so that it can survive moving from project to project.

What is your schlep?

A kind of call by name

2011-12-10T13:20:00.001-08:00

I often write code like this:

$self->create_user( username => $username, email => $email, pass_token => $pass_token );

I wish I could get rid of the naming redundancy in this call:

$self->create_user( $username, $email, $pass_token );

(without changing 'create_user' of course).

Probably some new syntax would be needed.

'use strict' and cargo cult programming

2011-11-18T23:42:00.001-08:00

I've just read mjd's confession "Why I Hate strict", it is from 2003 so he might have changed his views now, but there is nothing that would indicate that on this web page. Anyway, his main argument that the usual advice to use strict is automatic and mindless and that it often does not really prevent the problems that people think it does. In other words it is a cargo cult programming to which he contrast programming with thinking and deep analysis of everything you do.

I used to program without use strict; use warnings but after exposure to the usual propaganda I switched and I found that the cost of mindlessly adding it is negligible, the cases where I need no strict are very rare, and there are many benefits of doing it, especially when working with old code. This cult is rather effective in luring the cargo planes to land in my atoll. On the other hand I am all for deep analysis and checking your assumptions from time to time. There are many valid points in Marc Lehman common::sense and I would like to see them discussed. While we are on the road to have use strict by default we might also try to make it better.

$ primes for money

2011-11-12T04:22:00.001-08:00

The thesis above sounds uncontroversial. It is also rather uncontroversial that '$' is relatively frequently used when programming in Perl. Now - what can be the consequences of that?

Money has been said to change people's motivation (mainly for the better) and their behavior toward others (mainly for the worse). The results of nine experiments suggest that money brings about a self-sufficient orientation in which people prefer to be free of dependency and dependents. Reminders of money, relative to nonmoney reminders, led to reduced requests for help and reduced helpfulness toward others. Relative to participants primed with neutral concepts, participants primed with money preferred to play alone, work alone, and put more physical distance between themselves and a new acquaintance.

from one of the first links in the query above. Pretty sad - can that apply to the Perl community? Another link from that list, an entertaining BBC video report suggests also some other effects: hunger and pain insensitivity.

Thesis: simple - antythesis: easy - synthesis: ...

2011-11-07T10:21:00.000-08:00

Rich Hickey's Simple Made Easy is a great talk, a must see, with lot's of insight, but together with that it also misrepresents what Agile is about. Hickey's main point is that we should try to write simple software, because this is the only way to have reliable software, and he is right of course. He notes that when you encounter a new bug and try to fix it - all the existing tests pass - so they will not help you in finding the cause of it. You need to do the bug analysis on your own and the complexity of your code is your enemy there. He is also right when he talks about how easy means familiar a not simple and that it is a trap because it drives us away from the other (in small increments I would add). He is insightful when he talks about things that are source of complexity. He is funny, but missing the point in his critique of Agile.

The development sprints he attacks are not about doing the bulk of the work - they are about building a prototype on which we can test our assumptions. Without the understanding that we get from these prototypes we could simplify as much as we want but it would not change the fact that our solution solves the wrong problem. Agile is not an enemy of simple, it puts a lot of weight to doing the easy - but not because this is the goal - rather it uses easy as a mean to get to the correct. Agile is the answer to the paradox that we don't know what we should make until we already have a prototype of that thing. I wish more developers cared about simple - but only after they know what is needed.

Notes on the Synthesis of Form

2011-11-01T10:23:00.000-07:00

According to Wikipedia the origin of Design Patterns lays in the Pattern Language ideas by the unorthodox architect and philosopher Christopher Alexander, but his earlier work also used to be widely read by computer scientists:

Alexander's Notes on the Synthesis of Form was required reading for researchers in computer science throughout the 1960s. It had an influence^[8] in the 1960s and 1970s on programming language design, modular programming, object-oriented programming, software engineering and other design methodologies. Alexander's mathematical concepts and orientation were similar to Edsger Dijkstra's influential A Discipline of Programming.

The solution to the design problem that he proposes there does not look too attractive now, but his models, his metaphors, his insight into the design process - it's all still relevant and spot on. I am surprised that the Agile movement does not quote "Notes" as one of their foundation texts.

Concentration and Flow or Yet Another Dependency Injection Note

2011-10-15T05:07:00.000-07:00

Imagine that you need to do some small home improvement or maintenance work and you have all the needed tools, in good quality, clean and well maintained with all cutting blades sharpened and no missing screwdriver heads. A nice feeling - isn't it? When you start work like this you can concentrate on the task at hand instead of thinking where you can borrow that drill tool.

Collaborators in an algorithm are like those tools, having them readily available lets you concentrate on the problem.

Object oriented versus functional interface

2011-10-14T02:41:00.000-07:00

I use DateTime::Format::W3CDTF for parsing my dates:

my $w3c = DateTime::Format::W3CDTF->new; my $dt = $w3c->parse_datetime( $date_string );

I wish it was:

my $dt = DateTime::Format::W3CDTF->parse_datetime( $date_string );

and that the library created the parser on the fly as needed. It's not only less typing - but also much simpler mental model. This simpler model is sometimes too simple - for example if you parse a lot of dates then sparing the parser creation each time can make a difference.

I think the optimal thing to do is provide two APIs - like JSON - a functional one:

$perl_hash_or_arrayref = decode_json $utf8_encoded_json_text;

and an object oriented one:

$json = JSON->new->allow_nonref; $perl_scalar = $json->decode( $json_text );

for those that need that extra control.

open expects filename as binary data encoded in the system characterset

2011-10-03T05:21:00.000-07:00

I guess this is not a surprise to anyone who thought about how this is supposed to work, but for the sake of being systematic, here is the code:
use strict; use warnings; use autodie; use HTML::Entities; use Encode; my $a = HTML::Entities::decode( 'ñ' ); open(my $fh, '>', $a ); print $fh "Without encoding\n"; close $fh; open(my $fh1, '>', encode( 'UTF-8', $a ) ); print $fh1 "With encoding\n"; close $fh1
And here is the result when run on an system with UTF8 locales:
zby@zby:~/myopera/tmp$ ls ? a.pl ñ zby@zby:~/myopera/tmp$ cat ñ With encoding
'a.pl' is the name of the script itself, the mark '?' hides the F1 hexadecimal code and that file contains 'Without encoding'.

Courriel::MMS

2011-09-30T00:20:00.000-07:00

I think it is time to announce Courriel::MMS - it is an extension for the new email handling library by Dave Rolsky for processing MMS messages forwarded as emails by the mobile operators. This is still just a github link, no CPAN package yet, but it works in our production servers since Wednesday - so I bet on this code :)

It is a bit heuristic - for example many operators send a subject like 'You have received an mms' - which is useless for us, so the library tries to find something else that would act as the subject we need. For dealing with the various mobile operators, that each send a slightly different format of these emails, I used the Factory design pattern together with Module::Pluggable - this is a novel design for me so I wait for comments.

URI->path expects binary data

2011-09-20T08:29:00.000-07:00

Update: changed new to path - with new it would be reasonable to require that the uri fed to the parser is already an ASCI string containing the already URI encoded url.
Consider this code:
use 5.010; use Encode 'encode'; use URI; my $uri = URI->new( 'http://example.com/' ); say $uri->path( encode("UTF-8", "can\x{00B4}t-make-it-work" ) ); say $uri->path( "can\x{00B4}t-make-it-work" );
The output (in perl 5.14.0) is:
http://example.com/can%C2%B4t-make-it-work http://example.com/can%B4t-make-it-work
If your page is encoded in UTF8 - then the first one is correct: %C2%B4 is the URI encoded UTF8 encoding of Unicode Character 'ACUTE ACCENT' (U+00B4). If your page encoding is Latin1 - then the second one would be correct - but this is only by accident - in that case you should still use encode("iso-8859-1", ...).

There are probably many other string manipulating libs that should document if their input should be binary encoded data or decoded character strings.

Names are special

2011-09-01T05:57:00.000-07:00

At perl.com Tom Christiansen talks about sorting names, among other things. It appears surprisingly difficult to do properly with many special rules for each language. By coincidence at programming.reddit.com just above a link to that article there is Personal names around the world - a link to a w3c article about even more complications with names. At PhilPapers we had to solve *somehow* a few of these - the result is Text::Names. It is still rather limited: "While it tries to accommodate non-Western names, this module definitely works better with Western names, especially English-style names" - but there is already lots of logic embedded there.

is_utf8 is useless - can we have is_character?

2011-08-28T15:46:00.001-07:00

Consider this code:



$data_structure = utf8::is_utf8($json)

   ? from_json($json)

   : decode_json($json);

taken, together with the is_character suggestion, from otherwise very informative post: Quick note on using module JSON. I have seen similar code in many places. The idea is to check if the string you have is character data or a string of bytes and treat it appropriately. Unfortunately is_utf8 does not do that check:



use strict;

use warnings;



use utf8;

use HTML::Entities;

use JSON;



my $a = HTML::Entities::decode( '&nbsp;' );

my $json = qq{{ "a": "$a" }};

print 'is_utf8: ' . ( utf8::is_utf8( $json ) ? 'yes' : 'no' ) . "\n";



my $data_structure = utf8::is_utf8( $json )

   ? from_json( $json )

   : decode_json( $json );

This fails (on my machine) with following output:



is_utf8: no

malformed UTF-8 character in JSON string, at character offset 8 (before "\x{8a0}" }") at a.pl line 12.

If that still is a mystery try this:



use strict;

use warnings;



use HTML::Entities;

use Devel::Peek;



Dump( HTML::Entities::decode( '&nbsp;' ) );

the output (on my machine) is:



SV = PV(0x24f2090) at 0x24f3de8

  REFCNT = 1

  FLAGS = (TEMP,POK,pPOK)

  PV = 0x2501620 "\240"\0

  CUR = 1

  LEN = 16

this string is internally encoded as "\240" i.e. "\x{0a}" which is Latin1 encoding of non-breaking space. It does not have the utf8 flag set - so the code above tries to treat it as UTF8 encoded stream of bytes and fails.

I don't know if we can have is_character easily - but the lack of introspection here is surely painful.

CPAN, decoupling and Dependency Injection

2011-08-24T06:26:00.000-07:00

Consider the code:



sub fetch

{

    my ($self, $uri) = @_;

    my $ua           = LWP::UserAgent->new;

    my $resp         = $ua->get( $uri );



    ...

}

Yes - this is taken from a post by chromatic.

Now imagine that this is code from a CPAN module you installed and that some security concerns require you to replace LWP::UserAgent with LWPx::ParanoidAgent there. Bad luck - you'll probably need to subclass it, override that whole fetch method and pray that it will not change too much with every new release of the original module.

This is really why I am drumming this Dependency Injection drum over and over again - code that uses it is more reusable, more universal:



use Moose;

has 'ua', is => 'ro', default => sub { LWP::UserAgent->new };



sub fetch

{

    my ($self, $uri) = @_;

    my $ua           = $self->ua;

    my $resp         = $ua->get( $uri );



    ...

}

Now you would not have any problem with providing a LWPx::ParanoidAgent object for the fetch method to use.

By the way, with classical DI you'd move that LWP::UserAgent->new completely out from the class, here it stays as a 'default' that can be overridden from outside if you need. The problem with classical DI is that you need to have a place where to move that initialization code - here it is sidestepped for the 'normal' usage and you need to worry about it only in the cases where you really need to. Java probably does not have this 'default' mechanism.

Dependency Injeciton - the cooking metaphor

2011-08-18T12:22:00.000-07:00

Let's take a typical recipe. It first describes the goal - and then it goes:

Ingredients:

2-1/4 cups sifted cake flour
2 teaspoons baking powder
1/2 teaspoon salt
1/2 pound (2 sticks) sweet butter, room temperature
...

Preparation:
Preheat the oven to 350 degrees Fahrenheit. Butter and line two 8 x 3-inch baking pans or one 12 x 3-inch pan with parchment.

It is not:

Preheat the oven to 350 degrees Fahrenheit. Find a cow and milk her, wait until ...

neither it is:

Preheat the oven to 350 degrees Fahrenheit. Take your credit card and go to the grocery around the corner ...

Dependency injection is about writing your programs in a very similar manner - you first declare the collaborators and then go on with using them.

So what is Dependency Injection again?

2011-08-11T06:21:00.000-07:00

The definition I like the most is that DI is simply about separating object creation from business logic. Object creation - wiring the application - is a special type of code, different from the rest and it is useful to keep it separated. This is similar to how we remove hardcoded magic constants from our code into config files, it can be thought as a Object Oriented extension of that practice. We remove magic constants because we need to change them more frequently than the rest of the code. We do DI because we need to change the object wiring much more frequently than we change business logic and in particular we need to change it in the tests - without that unit tests would not be possible. But it is also different - because the object wiring code is much more complex than configuration files.

But this is not all - DI is also about keeping all object collaborators in it's attributes instead of reaching out for global objects (or signletons which are globals in disguise or class attributes). It thus improves object's encapsulation, makes them more self-reliant and testable. Or maybe this part is not DI - but simply writing Object Oriented code?

On the other hand, the separated out object factories are hard to test because they depend on all the objects classes they create and you want to keep them as small and simple as possible. How many such factories you need? If we have something that has a http request object as attribute - then we cannot build it until the http request arrives from the user. If we keep all collaborators in objects attributes - then we cannot build them until we have all information needed to build all these collaborators first. We thus need one factory per scope.

Subclassing applications

2011-08-02T05:33:00.000-07:00

Subclassing is a great tool for making small changes to a piece of code to fit it to new requirements. It is as easy as copying code - but it still keeps the new constructs synchronized with later changes to the original. There are problems with inheritance hierarchies - but you need to have a 'hierarchy', not just two classes, to get there.

I imagine that it would be perfect for extended configuration of applications - including web applications. Wouldn't it be great if you could run a slightly changed version of you main web app by making it's code available from PERL5LIB and then subclassing it to change the colors used, add some minor new features and remove some pages for and affiliated site? Or if you could install a blog engine from CPAN, and then subclass it to add new and override old features? This could even make distribution of CPANized applications more popular.

This is one of the things I am experimenting with at Nblog (see also the screencast: Experiments with inheritance in WebNano based applications).

Dependency Injection - Introduction - my talk for YAPC::Europe 2011

2011-07-25T01:27:00.000-07:00

My talk was accepted and now I published the slides.

Callbacks versus the Template Method pattern

2011-06-27T07:26:00.000-07:00

Let's say you have an object oriented html form validation library and, beside all the default checks (like non-empty or is_number etc.), you want the users of your library to add their own checks coded in the native programming language of your library. You have two simple ways to do that - you can let the form object contain a callback to be used for the check or you can use the Template Method pattern and let programmers subclass your form (or maybe individual field) class and implement their own 'check' method. Both solutions have their own pros and cons but I've never seen a comprehensive comparison of them.

What do you think about them? It seems that some programmers groups prefer one solution over the other (it's hard not to notice how callbacks are popular in the Plack related stuff). Callbacks are simpler, you don't need additional classes there, they don't complicate the program structure, but they are also more constrained. Template Methods are more structural. You can make them call other template methods and make that whole thing fine grained - that is harder with callbacks. In FormHandler I wanted to be able to stack the many generic checks together into lists - but such lists are not very reusable, you cannot override parts of it like you could if they were named methods. The point is that this additional class is another indirection layer and as we know that all programming problems are solvable with in this way :)

The Bitcoin protocol - a highlevel explanation

2011-05-28T00:28:00.000-07:00

Bitcoin is a peer to peer financial protocol. There is no central authority that guarantees that the operations are valid, how can this work? How can you trust strangers with your money? Let's say you are selling something for bitcoins - you receive the amount quoted - how can you verify that the transaction was valid? You need to check three things:

That it was indeed the owner of the sender account that created the transaction.
That the sender received the BTCs he is now sending.
That he has not transfered these BTCs already somewhere else (double spending).

The first check is easy - all transactions are signed. For the second you need to know another transaction that transfers the money to the senders account. Of course now you'll need to verify the other transaction as well - and then, in a recursive fashion, all the others leading to the one that generated the bitcoins (more about generating later). The sender could send you this chain of signed transactions - but in fact he does not need to do that because all the transactions are public.

The third check is more tricky - for this you need to always know which transaction was first and which one is second and then possibly reject the second one if it is double spending. This would be easy with a central authority that would timestamp all transactions and log them somehow - but it is very hard to do in a p2p system. In bitcoin there is also a global log of all transactions - it is called block chain (for it's technical representation). The trick is that to add a valid record to that log you need to solve a hard computational problem. You practically have no chances to do that singlehandedly and you need the other nodes in the network to help you. The sender broadcast his will to transfer money to the whole network and then wait until someone from this network solves the problem and correctly saves the transaction to the global log. This is like going to the notary to make sure that all formalities are met. After the transaction is saved to the log it is very hard to undo that - because you'd need to solve this hard problem again - and without a conspiracy between a significant part of the peers this would be impossible.

To be more precise transactions are not logged individually - but rather in batches called blocks. Such a set of transactions is written to the log in one sweep. This is important for efficiency - but not only.

All of this happens in a distributed, asynchronous manner. This means that it is possible that two nodes will write two new blocks to the transaction log in parallel and we'll get two different and valid versions of the log. What would happen then is that when a next block is written down to one version of the log it will take over all of the transactions from the block in the alternative log version, so they are not lost. But it is possible that the alternative logs contain two conflicting transactions (double spending) - then only one of them will be eventually saved. This showcases one important thing - having the transaction written to the log does not yet guarantee that it will stay there, because it can happen that there exists an alternative, valid log version and that eventually this other one will be the one that is continued by the system. But if you wait until another block is written to the log after the one that you are checking - then you are much more safe. This is the most complex part of the protocol - but the general rule is that the more blocks are written to the log after your - the more safe you are, and the safety grows exponentially.

Now - why so many nodes in the network would spend their time trying to save a new transaction block to the log? This is because they are paid if they succeed. In each new block saved to the log there can be one transaction that generates a specific amount of BTCs (now it is 50 - but it will be less in the future). This is they way new bitcoins are generated.

Another question is how the bitcoin project can maintain that the transactions are anonymous when they are all public? The answer is that what is public is the addresses of the sender and the receiver but not who owns these addresses. It is easy to create new addresses. You can use a new one for every incoming transaction and you can also make transfers between your own accounts to hide the tracks. Maybe it is not total privacy - but it is still better then with current money transfers where the banks know everything.

This introduction leaves out lots of technical details - but I hope that it explains the protocol enough to convince readers that it can work :) For more details have a look at the original bitcoin paper and the bitcoin wiki (just remember that what I call transaction log here is called block chain there).

By the way - I've heard bitcoin makes internet tipping in fashion again - so 113uhu2LrJp8gXGDrDLLDqmFGxnC2e3suB is one of my addresses. The interesting twist to that is that the donations are public - you can view the current ballance at blockexplorer. I've already got some tips - thanks :) Consider this an experiment in micropayments for blog financing.

Update: A critical analysis of the decentralization of the bitcoin protocol: http://www.links.org/?p=1164

Synthetic attributes

2011-05-15T00:45:00.000-07:00

chromatic writes about synthetic attributes. I wonder why this is not more widely used - it's such a simple technique. I've been using it since I discovered Moose and all the time in the back of my head I had this thought: Why people don't use this more widely? It is so obvious solution to the testing problems. Maybe they know something that I don't? But maybe it is just the matter of some guru (like chromatic :) writing about this technique?

PS. Sorry I'll not explain what synthetic attributes mean - this post is only a comment. I cannot currently comment at chromatic's blog.

git push and other copy commands

2011-04-12T05:54:00.000-07:00

The usual pattern of a copy command:


copy source destination

The pattern of git push command


push destination source:destination

the first destination is the destination repository - the second is the branch (and 'source' is the source branch). I must have skimmed the linked manual dozens of times over many years. There was something that just did not make sense and I imagined that I would have to read some book or something to eventually understand the magic 'ref specs'. It never dawned on me that it was that simple.

The additional difficulty is that the destination repository is usually shortened to 'origin' (because it was the origin for the clone command).

Drupal like architecture

2011-03-26T06:06:00.000-07:00

The Drupal subject reappears in the Perl blogsphere regularly. In my opinion there are three key architectural choices for 'portable plugin apps'. First, obviously, the plugins needs to be highly independent. Second they need to be controlled by pages - that is the page decide what plugins to use. To meet the independence requirement, in the Plack world, the obvious choice for the plugis is that they are simply PSGI apps that in body return a page fragment. This will make it easy to plugin existing wikis and blogs - you just need to remove the header and footer templates and voila you have a plugin instead of a standalone app. You can also use all the Plack infrastructure, for example the Plack::Client if you ever care about scaling your application.

There is one thing that I forgot in the first version of this post - Javascript and CSS. Applications are rarely pure HTML these days - we'll need also some way to gather all the js and css files that the plugins use and link them in the main page. At the main page side this is nothing special - it is not unusual that applications concatenate many Javascript libraries into one file automatically - so this is already well explored. The complication arises from the requirement to pass something more structured then just a HTML fragment in the Plack response from the plugins.

More interesting is the other key choice - how the plugins are supposed to be plugged in. That is how page handlers will choose what plugin to use and what data pass there. As much as I hate the 'program in XML' approach to language syntax - the obvious established standard here is ESI. Maybe we could even steal the semantics and replace the syntax with something saner? Plack based ESI implementation was submitted as an example GSOC idea - I really hope some student will pick it up.

Update: Added js/css handling consideration (thanks for cezio comment below).

App initialization and web frameworks

2011-03-19T03:49:00.000-07:00

Building the application object and its components has nothing to do with serving web pages. The database model setup is exactly the same if you do it for a web application or for your cron utilities and it should also remain the same if you one day decide to make a desktop version of your app. There is no reason that this initialization is a part of web frameworks other then we don't currently have a good standalone application builders. Now every web framework reinvents the wheel and writes one from scratch. Fortunately (in the Perl universe) there are now multiple efforts to create the ultimate configuration engine. There is just one step from a config reader to an application configurator.

As I have already commented elsewhere, I think the ideal for big projects would be a simple config file using simplified syntax that could be changed by the admins plus a Dependency Injection container that would build all the application components from the primitive values provided by this config. This DI container would be a second level configuration - still simple code but touchable only by programmers. For small applications I was quite happy with MooseX::SimpleConfig and Moose type coercions.

Reblessing objects

2011-03-15T14:21:00.000-07:00

Yesterday I've read the example from Refactoring transcripted into Perl. Nice work (I am not sure about the legal status of this - but I am sure that it is a great advert for the original book). The last refactoring there starts with:

We have several types of movie that have different ways of answering the same question. This sounds like a job for subclasses. We can have three subclasses of movie, each of which can have its own version of charge (Figure 1.14).

This allows me to replace the switch statement by using polymorphism. Sadly it has one slight flaw - it doesn't work. A movie can change its classification during its lifetime. An object cannot change its class during its lifetime.

It then goes on to show how the State Pattern can solve the problem thus stated. I am not so sure about this argumentation - typically the movie data would be stored in a database and the object would be created just for the one transaction - I think it is reasonable to assume that the movie does not change it's classification during the transaction. But OK - all this is just to show how the State Pattern solves the problem even if in this case the problem itself would not be real - for sure there are cases where it would.

Then it got me thinking - well in Perl you can change the class of an object during it's lifetime. I've even seen it done, I don't quite remember where. In general it is considered to be nasty - but is it always? In real life it is normal that things change - a caterpillar becomes a butterfly, a child becomes an adult.

Spark::Form

2011-03-14T13:49:00.000-07:00

I remember when when James was starting his project - his basic ideas were exactly the same as our HTML::FormHandler design. Unfortunately he learned about FormHandler after he had started coding - no way to talk him to change his plans and join FormHandler. Reinventing the wheel - but what I can decipher from his latest presentation - is quite interesting. Model integration - which is a major source of the complexity he criticized in HFH - is still lacking but the validation method is intriguing. Maybe cross-pollination will be possible?

Plack::Middleware::Auth::Form on CPAN

2011-03-01T10:15:00.000-08:00

I've got a bit impatient in waiting for feedback and this week I uploaded Plack::Middleware::Auth::Form to CPAN. This does not mean it is entirely finished - but at least it should be easier to install.

Plack::Middleware::Auth::Form has similar functionality as CatalystX::SimpleLogin, but since it is at the Plack level it can be reused by any Plack based framework.

Plack::Middleware::Auth::Form - like CatalystX::SimpleLogin but at Plack level.

2011-02-19T09:56:00.000-08:00

I've just published a first version to github: zby / Plack-Middleware-Auth-Form. The authenticator parameter (and it's initialization) is copied from Plack::Middleware::Auth::Basic - this is a much simpler interface then Catalyst::Plugin::Authentication - but I think it can be enough.

The redirects from and to the login page are modeled after CatalystX::SimpleLogin.

t/app.psgi is the only example so far, but I plan to use it in Nblog.

Is Newables/Injectables a fundamental distinction?

2011-02-10T12:11:00.002-08:00

In To “new” or not to “new”… Misko Hevery advices us to divide object classes into Injectables and Newables. Newables are objects that cannot be injected with DI frameworks (that is by one type of DI frameworks), Injectables are those that can. The point is to build Injectables only from other Injectables and Newables only from other Newables. To complete this recursive definition the simple Newables are defined as objects that have attributes of only primitive types. It is not so clear what are the atomic Injectables. I guess technically this needs be classes with parameter-less constructors. The examples given suggest things that you get from the OS or external libraries like an audio device or socket, or database connection. Of course sockets are not really parameterless - you need to pass the port number as an argument to the external libraries that create one, neither are audio devices if you have more then one attached to your computer - so this will only work if we assume that the given program works with just one port number or only with '/dev/audio'. What if you wanted to test it on another port? And what if you wanted to have and object containing an array of available sockets - you could not pass the number of the sockets in its constructor, because this would spoil its Injectableness. Parameter-less constructors are only hiding globals.

This distinction between Newables and Injectables proves to be rather constraining - it would be much cleaner if every class (and indeed also primitive types) were Injectable. Fortunately it is only an artefact of the DI framework chosen - there are DI frameworks that can inject any value, because they find them by name not by type. In these frameworks all classes are Injectables - but it might still be useful to keep the name of Newables for objects that can be build internally without calling any external libraries.

Returning to current page after login

2011-01-30T08:24:00.000-08:00

This is one of the must have conveniences of a modern web-site: if a logged out user encounters a page that requires a login then - after logging in the user goes directly to that original page (instead of staying on the login page or going to some other predefined page). Here is a quick list of possible high-level solutions - anything missing? What is your choice?

1. No redirect

Either you have the login form on every page (a small one) - or you can display the login form instead of the page text if the login is required. You need more then a small box to fit in all the currently broadly used options like OpenId or logging in via Facebook - so for a universal solution you need a full page. Let's focus on displaying the login form on a page requiring authentication instead of that page content. This is a very simple solution - would it break any of the HTTP protocol rules? But then wouldn't displaying the error message saying 'Login required' break the same rule?

2. Passing the current url in the link to the login form

The minus of this solution is the ugly url of the login page. I also read somewhere that in some implementations there is a security problem - but I cannot find now any reference about it (it was something about crafting the uri param in a way that resulted in two http requests when the redirection was done).

3. Saving the current url in the session

Additionally to all the problems with sessions (and for example multiple tabs open to the site), there is a slight efficiency disadvantage if we do that for each link to the login page, because the session then needs to be saved every time. If the session is save to the database I imagine that this could have a slight impact.

4. Using the Referer HTTP header

This would also be quite simple - but can we rely on it? Additionally it would also require passing that url from that header back to the form for the case when the first login attempt fails. This can be done with hidden field - and since everyone uses POST for the login form method this would not spoil the login page url (or it can also be done with a session - but this time the session would be saved only in the case of a failed login).

This would only work if the user followed a link from the current page to the login page (presumably displayed together with the authentication required error message). But actually it is better to automate that step and:

5. Redirecting both to the login page and from it

This still requires saving the current page in the url (as in 2) or in the session (as in 3) - but without saving current url in the session at every page containing a link to the login page. It can also be combined with 4 if additionally to this mechanism we want to have have links to the login page that would lead the user back to the original page. From the user perspective this is as convenient as 1 but might be a bit more clean from the technology perspective (see Dave's comment below - although is 303 more correct here?). Anyway - this is the solution used by CatalystX::SimpleLogin (and Catalyst::ActionRole::NeedsLogin).

Updates: Added 5.

Is $ responsible for the 'ugliness' of Perl code?

2011-01-10T00:48:00.000-08:00

We cannot hide it that Perl code is littered with the $ characters and, compared to the original well rounded Latin letters, that went through the thousands of years of tweaking to please the human eye, compared to them, $ is just an ugly hack. Unfortunately this is what people see when they first look at Perl code - lots of $ signs (by the way @ looks a bit better - maybe we should use more arrays?) - and this is how unconsciously they form the first impression about our language. This is not fair, this is so superficial - but the way the human mind works dictates that this first impression colours every information about Perl that comes later. People don't like to change their minds, and they always will find something to support what they thought initially, even if, or maybe the more so when, that initial thought was just an unconscious impression.

What can we do then? Changing the typeface of $ in the most popular web and editor fonts does not seem like a practical solution, but maybe if we show the sheer shallowness of judging the language by the shape of the typeface of some used character - maybe this would have some impact?

Update: Bear in mind that I don't advocate to get rid of $ in Perl - I find it very useful. The point is that the type face ugliness of $ can project a shadow on the whole language completely independently from it's usefulness as language construct.

Dependency Injection - some updates

2011-01-06T22:18:00.000-08:00

I finally found a succinct and clear definition of Dependency Injection:

Dependency injection means giving an object its instance variables instead of letting it create them.

based on answers to What is dependency injection? on Stackoverflow. There are some variations on how you do that - you can pass the instance variables in the call to the object creator (the new method) or you can use a setter to pass it to the object after it is created. This is exactly what I was thinking. Inversion of Control here happens only if you use a framework that takes over the object creation - but this is not required for the practice of DI. The name does not fit too well to its definition - but it is probably too late to change it.

Reasons to do DI:

The most immediate reason is that it makes it easy to use mockups (or even undef's or Nulls) for the dependencies in tests. But this is is just a concrete example of the more general reasons that follow.
To change the coupling between the main class and the dependency from control coupling to data-structured coupling (Types of coupling in wikipedia). Intuitively this also means increasing the encapsulation of the main module.
To remove more frequently changing code from inside of less frequently changing code. There is an analogy between hardcoded literal constants and hardcoded objects - this is the main subject of my previous post.

It is worth noting that Moose by taking over the new method encourages DI, but of course people will always find ways to sidestep this by creating an init or setup method or using BUILD or BUILDARGS from the Moose API.

Dependency Injection or Removing Hardcoded Values?

2011-01-04T23:45:00.000-08:00

Ever since I've heard about Dependency Injection (DI) a few years ago, I read and reread all the available documents and I kept having this feeling that I still miss something. Yesterday after some googling - I found that even those whom I always expected to explain things, have similar problems with it (see also this interesting reply to that rant). I am not sure if I now really understand DI, maybe I am still missing something important, but here is my tentative theory about it.

Let us say that somewhere deep in your web app code you have these lines:

Every minimally experienced programmer will notice immediately that for any re-usability the database name, user and his password should be extracted into a place where they can be changed easily, like the first lines of the program or better a config file. This would come very natural - but the reasoning behind it does not come from any theory - it is our experience that tells us that these pieces of data will change more frequently then the surrounding code.

I don't know if anyone would call this Dependency Injection, I think he would rather say that he is removing hardcoded constants.

Now let's take this snippet:

This is better - but the name of the DBI class and the way you construct the $dbh object is still hardcoded here - the point about DI is that this is still wrong because these parts also will change more often then the rest of the MyClass. The most important case for this is tests. In Perl it is possible to change the definition of the whole DBI package (mock it up) for the tests - but a more elegant solution is to move the database connection creation out of the MyClass code:

then, to test the MyClass methods that don't use the dbh field, you could pass an undef there and not bother with setting the database at all. But it is not only tests that require changes to that code - if for example at some point you decide that you want to log all the DBI warnings and add { PrintWarn => 1 } to the DBI->connect call - this would also be easier with this design. Again, just like with the initial example, I haven't seen any theory explaining this, but from my programming practice I am assured that this kind of change happens much more often, and at different phases of development, then changes to the rest of the MyClass code. Maybe the distinction is about what i.e. values (literals or objects) and how i.e. methods and subroutines - what is more expected and easier to change - and to get the advantage of that ease you cannot mix it together with how.

The code removed in the last sample is a bit more complicated then in the initial example - but at the core it has the same nature - it is about computing some values that are later used across a program part (or the whole program). In the initial example these values are literals, here they are objects. Literals can be created independently, objects depend on other objects: we can have a MyUser that depends on MyClass that (as in our example) depends on the database connection. Fortunately solving the dependency graph and finding out what needs to be created to have a MyUser object can be automated - this is what the various DI frameworks (like Bread::Board) do. In more 'strictly' typed languages like Java - this dependency graph is mostly defined by the types used, in Perl the dependency needs to be defined by the programmer explicitly - but the graph solving works in the same way. This is why this object creation code is (or can be made) quite simple and for all objects in the same scope it can fit into one package where you could adjust the various cooperating objects just like you do with the values of the configuration variables.

This is my understanding of Dependency Injection - maybe I am still missing something - but I don't see where, in any non-convoluted way, there is the Injection happening. There are no new dependencies injected into the code. Maybe more appropriate would Dependency Passing - as the dependencies are passed as parameters - but most of the work done when performing this refactorisation is removing parts of code, parameter passing is the trivial part. Additionally Inversion of Control, which is used to explain DI in many places, is just a minor characteristic of it, not very visible one and very different from other more common Inversion of Control occurrences. Mark Fawler coined the term Dependency Injection specifically to differ it from Inversion of Control in general - but I think that even presenting it as an example of IoC is misleading, because it is not very important to understand what is going on and it is so different from the Hollywood Principle that everyone thinks about when reading Inversion of Control. Finally what is frequently blurring explanations of DI is the details of the DI frameworks used and dwelling over constructor injections or setter injections - i.e. going into the how before the why is well explained.

I believe that removing hardcoded values is what Dependency Injection is really about. Most programmers already have good intuitions on why this is needed.

Images inheritance in Plack based apps

2010-12-23T00:53:00.000-08:00

My plan for Nblog is to let users customize it by subclassing and overloading, in the screencast I showed how to do that with controllers and templates, in the latest revision of Nblog I added the possibility of overloading images. There are Plack components for all parts of this task - but I had a bit of struggle with getting the settings right. Maybe the following code snippet will spare similar time waste for someone?

Some explanations: psgi_callback is the method in WebNano that constructs the anonymous subroutine required by the PSGI standard and here I use the around Moose method modifier for it to add some additional processing there. static_roots returns a list of directories where to search for the static files - this is our search path - the first file found in these directories is served. In the base class this list contains only 'static' - if you want to overload the static files in a subclass you need to add your directories before this one, in my example subclass config I have:

static_root => [ 'static', '../Nblog/static' ]

(these are relative paths). See also Plack::App::Cascade. All of this would be a bit easier to assemble if the Plack components logged some debug info to STDOUT, like what you can find in the standard Apache logs about files not found, when run in development environment. I am volunteering to write a patch if there is a green light from the core devs.

Unit Testing

2010-12-20T04:47:00.000-08:00

With one functional/system test you can test lot's of low level parts that in Unit Testing would require a separate test each. It is also easier to write - because you only need to test how the module/library is used and using it should be easy (and not require mocking etc) - otherwise the library would not be a good library to start with. And what are the reasons to do unit testing? Personally I was never convinced by the arguments until I started reading Misko Hevery - here is for example is the single best explanation of why to do unit testing I've ever seen:

Lets say you would like to test a car, which you are in the process of designing, would you test is by driving it around and making modifications to it, or would you prove your design by testing each component separately? I think that testing all of the corner cases by driving the car around is very difficult, yes if the car drives you know that a lot of things must work (engine, transmission, electronics, etc), but if it does not work you have no idea where to look. However, there are some things which you will have very hard time reproducing in this end-to-end test. For example, it will be very hard for you to see if the car will be able to start in the extreme cold of the north pole, or if the engine will not overheat going full throttle up a sand dune in Sahara. I propose we take the engine out and simulate the load on it in a laboratory.

from It is not about writing tests, its about writing stories

In the rest of his writings you'll also find how to structure your code in a way that makes unit testing easy and does not require crazy mockings. I am also convinced that code structured in that way is well decoupled and easy to understand, modify and talk about.

Experiments in Inheritance - a screencast

2010-12-17T14:16:00.001-08:00

This is a short screencast I made to replace the video made at my London Perl Workshop presentation. It did not make the list eventually - but I still think it might be interesting: http://warszawa.pm.org/WebNanoInheritance.mpeg. It is my first screencast ever made - please comment.

The main idea presented there is about solving the "give mi an application like this one but ..." problem, illustrated by making a blog engine that inherits from Nblog, but extends it and overrides some of it's elements.

Namespace Matching

2010-12-13T23:17:00.000-08:00

The other day I was reading the Dancer Advent Calendar and I finally found the two words description of what WebNano does. WebNano is the minimal addition to PSGI that provides Namespace Matching. Beside easy deployment, which is currently covered by Plack, Namespace Matching is probably the main feature of Catalyst and now with WebNano you can have that without the heavy baggage of the whole of the Catalyst framework. I think it is important because it seems to produce just the right granularity of the controller classes and provides a path for the application growth (without overloading one file with too much stuff).

WebNano - some incompatible API changes ahead

2010-12-05T02:51:00.001-08:00

I tried carefully not to document some things that are likely to change in the closest WebNano releases, but the example code I put in the docs needs to be exact and that means it contains some details I have not yet decided are stable. One of these examples is about overriding the local_dispatch. It used to take the path as a string parameter - now it takes an array - i.e. the path split on the '/' character. The past way was more universal as it let the user programmer to specify how to parse the path - but I think splitting on '/' is such a common usage that it justifies the change. And it is not destructive - so if someone really want's to parse the path in a different way then he can join it back.

This change will be in the next WebNano release.

In the longer perspective I am thinking about being more compliant with the Law of Demeter and building the controllers with the needed model parts in their own attributes, instead of accessing them through app. So for example some code in Nblog would change from

$self->app->schema->resultset( 'Article' )->search
to
$self->article_rs->search

Now this can be done by overriding the handle class method - but maybe it needs something more elegant.

WebNano as Catalyst::Tiny

2010-12-01T04:52:00.000-08:00

If you have read The Philosophy of WebNano you might think that WebNano is radically different from Catalyst, the more so if you'd compare the size of these projects (sloccount reports over 5000 lines in lib for Catalyst, versus less then 250 for WebNano). But if you compare the code structure between Nblog and the original RavLog project you'll find it very similar. The DBIC schema and form classes were just copied around, you'll see mostly the same controllers with mostly the same methods.

Look for example at Nblog::Controller::Ajax and RavLog::Controller::Ajax - the changes are minimal, mostly just changing sub check_articles : Local to sub check_articles_action and accessing the model from $c->model('DB::Tag')->search to $self->app->schema->resultset( 'Tag')->search - a bit longer perhaps. Sure there are other controllers like: RavLog::Controller::View that I renamed to Nblog::Controller::Article. This renaming is not important - I just did not like a controller called View but there is also some difference in their methods. The RavLog controller uses the Chained Catalyst dispatcher - while in the Nblog one I overrode local_dispatch, and it uses stash to communicate with the template while in Nblog I passe the data directly as a parameter. Still some similarity remains.

WebNano uses some dependencies - so it does not fit into the original Adam's definition of tiny modules (by the way I cannot fin this definition now - maybe this should go to some semi-official place like the p5p wiki?). But the prerequisites are really minimal - and mostly tiny themselves. Maybe in the subject space of web frameworks this should be allowed?

Why WebNano - the article

2010-11-25T01:16:00.000-08:00

My article on the motivation for WebNano was published at perl.com.

Old code and survivor bias

2010-11-15T09:04:00.000-08:00

If code is well designed, decoupled and has good test coverage there are chances that it will be regularly updated to new technologies and new programming techniques. The other code will probably scare programmers and have much chance to stay in it's sad state until decommission.

Templates

2010-11-06T12:17:00.000-07:00

I've heard that Perl is ugly because of all those pesky '$'. Purely esthetically this might be true, depending on the font used, but I have the impression that people use the word 'ugly' in a more metaphorical way, like 'ugly subroutine'. Somehow they believe that using an otherwise unused character for marking variables is inelegant - but I think they are confused, I think their opposition arises from the kneejerk reaction to an unfamiliar character. For me the '$' signs make parsing and understanding Perl code easier.

I've heard that TT2 code is ugly because it uses '[%' instead of the more familiar '<', and that it 'FOR' is a heavy burden for the eyes of the programmer because it is "shouting" at him, but I think that the features that make TT2 code standout in the text being templated make it easier to parse and understand and they help my eyes in their work.

Beside that purely ergonomical argument I don't see any purpose of the mini language and I think I am not that thrilled by the future direction of TT.

WebNano

2010-10-20T11:54:00.000-07:00

In the weekend I published WebNano to CPAN. There is some documentation there - so it is ready for experimenting. I am thinking about a "Why WebNano" article. For now let me just say this: WebNano has just 232 lines of code in 'lib' (as reported by sloccount) and beside Plack it uses only minimal dependencies, but when porting a non-trivial blog engine from Catalyst to WebNano I did not miss any Catalyst features. To be honest in that conversion I did use two WebNano extensions (both also already published), Template Toolkit renderer with dynamic paths and a CRUD controller, but even including them the line count would not excede 500.

Extending the 'Template Method' design pattern

2010-10-18T13:06:00.000-07:00

There is this notion that you should not put semantics into names, or maybe that what your program do should not change in alpha-conversion. But of course everyone is breaking that rule if she is using the Template Method design pattern (if we agree that the library they use is somehow outside of the program and is not alpha-converted together with it). There the names of methods have a lot of semantics.

In Advanced Search in web DBIx::Class based applications I proposed an extension to DBIx::Class that would treat all methods with names search_for_* as predicate builders. This way programmers could easily extend the query language used by DBIx::Class with new predicates by adding methods to the ResultSet class. Now in WebNano all *_action controller methods also have a special meaning - these are the methods that can be called from outside via automatic dispatching from an url. Taken literally sub my_method_action is not that different from sub my_method : Action which is the way to specify that in Catalyst. The difference is only ' : A' versus '_a' (and I could use *_Action if I wanted to golf it even more) - but yeah - I am cramming the semantics into the name while Catalyst reserves it for the code attributes. Catalyst way is cleaner - but my goal was a 'minimal' framework. The 'Extended Template Method' design works in a few lines while MooseX-MethodAttributes is 13 packages with who knows how many lines of very subtle code.

What surprises me is that I have never seen a description of this 'Extended Template Method' design pattern. It looks rather obvious, maybe it is called by some other name in the literature.

Inviting non-Perlers to speak at Perl conferences

2010-10-12T13:14:00.000-07:00

There is an organized effort, by the Perl volunteer marketers, to attend non-Perl events and communicate about the all new things happening in the otherwise insulated Perl word. I wish this effort all the best, and I hope that The Perl Foundation will find a way to support it. To enforce the effect I would like to see also the symmetric and complementary action of inviting non-Perlers to our conferences. I am sure that is already happening to some extent - but personally I have not yet seen it - so I conclude that this extent is not enough.

Whom, from non-Perl speakers, would you like to hear at your next Perl conference?

Value Objects - some notes

2010-09-30T00:11:00.000-07:00

There is a controversy - or maybe a fuzzy consensus about what you should be able to do in templates. Some template engines, like Template Tookit, let you call subroutines and methods on stuff passed to the template. This is very useful - you don't need to care about turning the data into a specially formed hash that can be iterated by the template, you just pass the data there and can be sure that it can display it. But it can also be abused. I think everyone would agree that template should not for example change the data in the database, but where is exactly the limit is not clear. Value Objects seem to be a good answer, templates should work only with Value Objects.

The question is how sharply you can divide your objects into Value Objects and Service Objects. In ActiveRecord is hard to test Hevery proposes a design where the record in Active Record is a Value Object (newable in his terminology is a synonym) with no link to the repository object which would do all the database manipulation. This would work for normal attributes - you'd assign the value there on object creation - but what about relations? $user->books is very convenient, especially in templates, and it can be pure - if you know that you need the books on the user you could put them there when creating the user object. But then you'd have two kinds of objects - those with books, needed in some places - and those without, when you want to spare the additional database calls. Maybe we need subclasses of User? But then we'd need subclasses for all possible combinations of relations (users with books, users with friends, users with books and friends, ...), not to mention the mess with relations on the related objects.

Another argument for immutable objects

2010-09-26T22:35:00.000-07:00

Class is like a little program, it has data, it has code, it can be instantiated - like a program can be run. From the perspective of the object it's attributes are global. There can also be other variables - like parameters passed to the methods or other block scoped variables - those are local, but attributes are global. If they are immutable - they are like programs constants, but when they are changeable - they are like global variables. And we know that global variables are bad.

Managing Object Lifetime

2010-09-25T06:29:00.000-07:00

This is the title of a blog post by Misko Hevery. There are also lot's of other materials authored by him freely available on the net. I have just started exploring them, and there is a lot of repetition if you just google around like me - but I already want to recommend it to anyone working on improving the design of their programs.

WebNano does not (yet?) meet all the design criteria he is talking about - but he is putting into words many of the foggy intuitions that made me not satisfied with all of the existing Perl web framework and write yet another one.

Inheritance and rapid prototyping

2010-09-23T06:12:00.000-07:00

There is much talk about the dangers of too much inheritance and how composition is better then inheritance - and probably they are right if we are talking about the final product. What they don't account for are the dynamics of the development - where you grab the first thing that does something similar to what you need and you start testing it, changing it, evaluating your ideas. Inheritance allows you to do exactly that - making changes to something that was already finished and working somewhere else. With procedural code this is not possible, the most similar thing you can get there are examples.

In other words Inheritance might lead to tightly coupled code, with intricate execution paths, but at least it is an easy way to get something working quickly. You can refactor it later.

Installing Dist::Zilla plugins

2010-09-16T12:29:00.000-07:00

Dear lazyweb - when I try out a distribution converted to DistZilla I often discover that I don't have all the plugins used there. Starting the work then consists of a string of 'dzil build' commands, searching the monstrous error stacks for the name of the missing plugin and then running cpan with it. Sure that should be automated!

And, by the way, it could also install the distribution's prerequisites.

Update: Repeating one of the comments - apparently the latest Dist::Zilla supports following solution dzil authordeps | cpanm.

Object::Tiny:RW and MooseX::NonMoose

2010-08-14T02:36:00.000-07:00

There are conflicting requirements for WebNano. On one hand I would like to build big heavy web applications based on it, and of course I would like to use Moose as the object framework for them. On the other hand I would like to give it a chance to work in the restricted environments of most shared web hosting serving CGI scripts - so heavy dependencies and XS and big startup times are out. And I would also like to make it suitable for serving huge amounts of simple Ajax requests - so it needs to be fast.

First I tried using Any::Moose and let people decide if they used Moose or Mouse. By using Mouse people could solve the startup problem and also there is some way to use Mouse without XS.

But Mouse was still huge dependency - so next I asked myself it it would be possible to subclass in Moose a non Moose base classes. That proved to be rather simple with the aptly named MooseX::NonMoose. Using it in WebNano I can build my objects in what ever way I choose and as long as I stick to the standard hash based objects I can extend the WebNano classes in my Moose based applications with no problems. Or at least it seems to be with no problems for me now - if someone knows more about disadvantages of this solution - please leave a comment.

After discovering MooseX::NonMoose I switched to use Class::XSAccessor. It small and very fast, but unfortunately it was XS based. I was thinking about the possibilities of bundling the software as a PAR package or so and using it in the restricted shared hosting environments and relying on a C compiler seemed to complicate things a lot.

Next I tried Object::Tiny, it was tiny, fast and did not use XS, there was just one but - it did not produce the attribute setters. I could live with that and it did not take much time for me to port WebNano to use Object::Tiny, but still it seemed like an extreme choice. Fortunately just browsing CPAN I discovered a fork of O::T - Object::Tiny::RW - this one is small, fast, does not use XS and does produce attribute setters. I am still a bit worry about why Adam would not add that feature to his module - given that it is such a trivial change (compare the sources of Object::Tiny::RW and Object::Tiny - its just one line changed), he must have had a good reason for that?

RavLog - a perl blog engine

2010-08-07T05:56:00.000-07:00

For testing HTML::FormHandler Gerda Shank wrote RavLog. Recently I've made a fork of it, switched to SQLite as the main db engine and made it easy to install. It still does not pass all formatting tests (HTML::Tidy seems to be uninstallable) - but it is usable. My goal is to make another fork out of it and port at least part of it to WebNano, but I think that what is now there can also be interesting for someone looking for a Catalyst based weblog engine.

Technically right, socially not so

2010-07-09T03:48:00.001-07:00

Imagine that you let anyone use your hard work by publishing your code as Open Source and then instead of thank you notes you receive angry emails about purported bugs in it.

Imagine that you noticed a problem about some Open Source code and as a good citizen you analyze it and report your findings and instead of a thank you note you receive a reply explaining how much confused you are for thinking that that could be a bug.

Imagine that this happens in a public space like an mailing list, open bug tracker, etc. This is like the setting of a Greek tragedy - both sides have their own rights and values, both sides can be technically right - and yet the conflict spreads and escalates. People copy the emotions of each other, even perceived emotions and especially anger, when displayed publicly leads to more anger. This is a vicious circle of positive feedback loop. When it starts everyone gets irritated and everyone wants to know who started it, who is responsible for that.

But go back to the start - it's not anyone's fault - you cannot point the finger on one person and say 'he is responsible'. Looking for the main culprit is really looking for a scapegoat - a surrogate victim that would heal everyone's consciousness. This is not especially geek thing overall - it is a universal trait - but sadly inventing fodder for the scapegoat search is a prominent part of geek culture.

One of the many things that I admire about Larry Wall is how he can neutralize the poisons of angry reactions. People scorn Perl for being a 'scripting' language - he talks about Ada Lovelace:

Suppose you went back to Ada Lovelace and asked her the difference between a script and a program. She'd probably look at you funny, then say something like: Well, a script is what you give the actors, but a program is what you give the audience. That Ada was one sharp lady...

Installing dependencies - continued

2010-07-07T23:58:00.000-07:00

I don't know what I am doing wrong - but after installing Perl 5.12.1 under perlbrew "cpan -m ." stopped to install the dependencies. It seems that for making and testing it adds the libs in '.cpan/build' to the PERL5LIB search path so both can go on without really installing the dependencies:


Running make for /home/zby/progs/ravlog/.
Prepending blib/arch and blib/lib of 108 build dirs to PERL5LIB; for 'get'

"cpan -t . " works in the same way and if the tests fail then "cpan ." stops there - so I cannot tell if it would install the dependencies in the case when the tests pass.

I don't really know how to investigate it further - but I am noting that as it is very frustrating. Eventually I had to resort to cpanminus ("cpanm --installdeps ." is really handy as it addresses the problem directly).

Implementing file upload progress bar

2010-06-24T04:20:00.000-07:00

At work I was assigned the task of replacing the previous hodgepodge of tools that provide the progress bar functionality for forms with file uploads. At first glance this seems like a trivial thing to do - you periodically observe how much of the file you've got and update the progress bar, but there are details that make it hard to do with the current tools. I report here how I solved that problem - not because I think this is the optimal way - but rather to open the discussion and because I could not find any description of solving it when I googled for it.

First of all the common programming tools (like CGI.pm or Mason that we use here) assume that the page handler receives the whole request as input - and that whole request is not available until after the file is uploaded. So for example 'my $q = CGI.pm->new' will not finish until it is too late to measure the upload progress. The solution to that is to use another page to report the upload progress and call that page via Ajax from Javascript code updating the progress bar. This would work great - but the file is normally uploaded to a temporary file with a random name and the other script would not have any chance to guess it. We need to generate a new random file name in the form page and then pass that name to the form handler script so that it would save the data to that file, and in parallel to the Ajax scripts that would check the size of that file.

To save the data into a specified filename I used the CGI.pm callback feature:


my $q = CGI->new( \&hook, $fh, undef );
...
sub hook {
    my ($filename, $buffer, $bytes_read, $fh) = @_;
    print $fh substr($buffer, 0, $bytes_read);
    $fh->flush();
}

It is described in the subsection called "Progress bars for file uploads and avoiding temp files" of the CGI.pm documentaion, but actually it is a great leap of thought to say that it supports progress bar implementation, you still cannot use it directly to get the progress bar from the CGI object on the form landing page, you still need the separate scripts measuring the progress. For my solution all I needed was to pass the target file name to the code saving the data, this could be easier than writing this callback above. And the callback is still not everything - I yet need a way to pass the generated filename from the form page to that script - and not via form parameters, remember they are not available at that stage. So how can that be done? Simple - as PATH_INFO - which is available in the %ENV hash even before the params are parsed by CGI.pm.

This is the skeleton of the solution - there are a few more details in the actual implementation - but the code will be published soon as Open Source - so I hope everyone will be able to look them up there.

When those micro-seconds matter

2010-06-19T09:21:00.000-07:00

Catalyst is the only web framework that has a mailing list I subscribe to - but I am sure that it happens at others too. In a recurring pattern someone someone posts a benchmark showing that Catalyst for some trivial operation is many times (or many hundred times) slower than some other web framework or for that matter PHP. That does not fail to generate a heated debate - but eventually the seasoned framework developers gain the upper hand with the argument that for all the, often big, web sites they worked on, those few micro-seconds lost in the Catalyst dispatcher never mattered much because the application spent hundred times more in other code fragments and mostly in business logic parts, so shaving off some part of the few micro-seconds would not improve the overall speed more than 1%. This is a great argument, perfectly reasonable and rational but it is biased towards the status quo. It might be true that everywhere where Catalyst is currently used it works great but it is also not hard to imagine an application with very simple business logic that needs to serve millions of users, Twitter anyone? Or an app that does many simple Ajax callbacks. Sure you can always code the speed requiring, simple parts in PHP and keep Catalyst only for the other more heavy-weight tasks but having a universal solution would be so much more convenient.

Installing dependencies for your packaged for CPAN library

2010-06-13T05:46:00.000-07:00

Update: See the comments, apparently the warning against auto_install is outdated information. I don't have any opinion of myself here.

It always puzzled me what is the 'canonical' way to install dependencies for an unpacked distribution, be it downloaded from CPAN or your own in-house product packaged the CPAN way for convenience. Module::Install provides an 'auto_install' option that you add to your Makefile.PL and then perl Makefile.PL; make (or perl Makefile.PL; sudo make) should do that trick. This seemed like a good solution that could be a standard, but now I see that the FAQ of that module says:

Do I really have to avoid auto_install()?

auto_install() has a long history of breaking CPAN toolchains. Lots of people had a bad feeling on it, and have said it should be strongly avoided. In fact it was deprecated and removed once.

Although most of the known problems have been fixed and you can use it more safely than ever, the use of auto_install() is still discouraged, especially if what you are writing is a module to be uploaded on the CPAN. auto_install() does lots of things itself, thus does not always do the same things as other toolchains do (including extra attribute handling, etc; which can be fixed somehow but that's not too DRY). It only supports CPAN and CPANPLUS as its backends. If you use other tools to install, it may still cause a trouble.

Besides, now you can do what auto_install() does with other means. If your CPAN module is new enough, you can pass a dot to the cpan command it provides, and it will install all the required distributions from the CPAN:

$ cpan .

So apparently this is the way to go - using the cpan shell. There is a minor problem with this when used for testing - it would automatically install the package from the directory you are in. Sure you can suppress that you need by adding another command switch (-m), but it would be better if the default action was the safe one.

Templates and inheritance

2010-05-22T04:07:00.000-07:00

In every web site project I have been working on there inevitably came that moment that someone wanted to have a copy of the site with some small changes, or that someone wanted to use a piece the website in something else. In both cases it was all about (initially) some small changes, what they call as skin or a branded site. With programming part we know what to do then - we design base classes for the core undifferentiated functionality and inherit from them in the targeted classes. Most templating engines do have some notion of search path - so this should be doable with templates - but it always proves to be much more messy than it initially seems to be. I'll try to catalogue here all the use cases and how they compare to the standard inheritance in object oriented programming languages and describe them in terms of operations on the search path.

Initial setup

In 'normal' programming languages the atomic piece that can be inherited and overloaded and that is searched on the inheritance path is a method - in case of templates the atomic part is a whole template. The granularity is defined differently, but we can do analogues things with both. Here I will concentrate on the dispatching model familiar for users of most of the MVC frameworks with controllers and their methods as the endpoints (called actions). In this setting all the methods could have a matching template file and inheriting and overriding a template could work on the same level as inheriting and overriding an action.

Basic case

Lets say I have a controller 'MyApp::Controller::Book' with a few actions like 'view', 'list', 'edit' (a CRUD controller) and matching templates in 'templates/Book'. I would like that when calling the render from that controller I did not need to specify the full path of the templates - but just call them by name 'view'. Also when they have a common subcomponents in the same directory they should be able to [% INCLUDE date_formatting %] them by name. This is just like calling methods in a class and it can be accomplished by adding the 'templates/Book' directory at the start of the search path.

Library component

Sometimes we would like to use ready made components from some published library, maybe a CRUD controller for our Books. We inherit from the library class in our 'MyApp::Controller::Book' and we would like its templates to be inherited in a similar way. That means adding '/usr/lib/something/something/CRUD/templates' to our search path somewhere after 'templates/Book'.

Subclassing our own controllers

If our 'MyApp::Controller::Book' is a subclass of our own ''MyApp::Controller::Item' class - this is a very similar case to the above. Here as well we want to insert the search path of the parent class somewhere after 'templates/Book. But in this case what we insert is 'template/Item' - a path relative to our application root.

Application wide components

This is solved simply by adding an application wide element in the search path. It would be analogues to adding stuff to a common base class in standard programming - much more rare there.

Subclassing a whole site

This is the case of 'branded' sites where we would like to subclass everything at once. It is maybe the most common case - but also the most difficult because there are no good analogies/solutions in standard object oriented programming. If 'MyNewApp' inherits from 'MyApp' that does not automatically mean that 'MyNewApp::Controller::Book' springs to existence and can serve book information in the new app (but I've heard there are existing Object Oriented languages that work like that and some people are thinking about implementing this in Perl). So I think we need to first think over how this is handled in the main framework and then we can deduce something about how to treat templates from that. For the limited case of application wide components this can be solved by adding the new template root to the beginning of the search path - but it gets more complicated for all the other cases described above.

Conclusions

I mostly wrote that to have it all in one place - but I think it does show the need to make the template search path dependent on the controller used.

A subject for mst's talk

2010-05-19T14:43:00.001-07:00

I did not really intended to vote in that silly poll - but I think I have found a subject for the talk that could be highly beneficial for the Perl community: How using the insights from cognitive science (and what else is needed) I am going to make HTML::Zoom actually usable for mere mortals.

PSGI and Object Oriented Programming

2010-05-18T13:10:00.001-07:00

There is one thing that seems pretty simple - but it took me a bit of mulling over to fully understand

A PSGI application is a Perl code reference.

It is a reference to a sub that expects a hash ref representing the HTTP request passed to it as the one and only parameter and which returns an array ref representing a HTTP response. Hmm - yeah that's what web apps do - take the request and return the response.

But then you think - OK, but I like Object Oriented code and I want my application to be an object with attributes and stuff - so what can I do? Well - that is simple. Have your app as object as you like, and to the PSGI layer pass a closure referring to that object. For example if you application object normally handles the requests with a ->handle method then

would return a suitable callback. Which you can then use in your app.psgi file:

And this is exactly what I do in WebNano.

Web Applications and Dispatching

2010-05-12T11:15:00.000-07:00

Historically the operation of finding the code to handle the web request was trivial - the server just matched the path part of the web request into the filesystem to find the CGI program producing the requested page. But then it got more complicated when we moved from the CGI 'one program equals one page' way into the realm of web applications that serve many pages.

The tree (or DAG) data structure is the underlying concept of both the path part of URL and many programming language library (or class) structures - so there is a natural way of mapping between them. To have code handling requests to related pages in the same file we can extend it so that the last part of the path is interpreted as a subroutine name.

That idea gets a bit more complicated when we realize that we don't want to expose all of our often internal methods (and libraries/classes) to be callable from outside - we need a way to mark them as 'external' and expose only these.

Of course sometimes there are reasons that we don't want to use such a literal mapping, it can be some security (by obscurity) need to hide internal code structures, need for more elegant URLs or other requirements. So sometimes we need to extend it or completely replace it with something else.

The Perl web frameworks (that I know the best) in respect to usage of that mapping can be roughly divided into following styles (many use more than one style so the boundaries are fuzzy):

The 'old' CGI (or PHP) paradigm of one address one program
Using the mapping of paths to libraries, selecting 'externally callable' methods by placing them on a list ( run_modes in CGI::Aplication ).
Using the mapping of paths to libraries, but dispatching to methods configured via a hash (also run_modes in CGI::Application).
Using the mapping of paths to libraries (but not to methods) and calling the 'get' or 'post' method in the landing library (like in Tatsumaki).
Configuring the dispatching by code attributes of the methods (like in Catalyst).
Not using methods but anonymous subroutines. This way it is easy to assign the dispatching configuration to the code by using DSLs (like in Dancer).
Dispatching configuration external to the classes that are dispatched to.

Having the dispatching configuration close to the subroutine code let's you avoid switching between two places in the source code when adding a new action or changing an existing one. I think one of the greatest conveniences of Catalyst was that that configuration was not only in the same file but directly in the subroutine definition.

WebNano - what I did not say at the London.pm tech meeting

2010-04-17T01:07:00.000-07:00

About a month ago I had a presentation at the London.pm tech meeting about the design ideas I plan to experiment in WebNano. There are a few things that I did not have time to tell - and I plan to blog about them. First of all I think I should have admitted that I stole the both the name and the idea from Simon Cozen's MicroMaypole (I hope he will treat that in the terms of the well known quote about imitation and not intellectual property trespassing:).

At the presentation I mentioned that there is not much code in WebNano currently - but maybe this was not the best way to put it. I actually don't plan to ever have much code in it and the example of MicroMaypole shows that perhaps such a minimal web framework can be a practical thing, but I do plan to build some more examples using that minimal code so that I can be sure that even if minimal it does have all the desired features. What are those features will be the subject of other blog posts soon.

Frameworks are framing, libraries are liberating

2010-02-01T06:59:00.001-08:00

It seems that the season is open and everyone is writing his new web framework again (all authors of the dozens of other recently created Perl web framework - please accept my apologies - I could not possibly link to all of your babies). If you ask me why is that? My answer is: it is easy now. We have libraries that do nearly everything that the traditional web frameworks provide. You don't need to write much - you just saw together the available patches and voila: your own web framework. You can be proud.

For some time I was thinking: OK, so now we have that cambrian explosion of new frameworks, that's OK, it will all settle down, people will select the best of them and the rest will die off. All we need to do is waiting. We could also speed it up a bit and make it a bit more smooth and rational by learning how to evaluate the frameworks. To do that we need to talk about them, study their functions and compare designs, project goals, project management styles and all of that. This will not be easy, but once we get past the awsome depth of common analyses of the 'this sucks' style and start to think about 'this is good for that and that other thing is better if you need something else', the rest will be incremental.

And I still believe that we need to talk more about design solutions and expand our vocabulary and use less of 'this is stupid' authoritative evaluations. This is something beneficial in any circumstances. But now I also think that since we have the libraries that do the all the work that previously was done by frameworks - then why should we build any more frameworks? Frameworks are coupling, they tie together the libraries that do the actual work and send that loose, uncohesive package to the programmer, who he has no choice, but has to use all of it and in the way prescribed by the framework. But every educated programmer knows that coupling is bad and cohesion is good. So why don't we just use the available nicely decoupled and internally cohesive libraries instead? Yeah, the libraries don't yet do all the 100% of stuff traditionally provided by frameworks, they can do 80% or even 90% of it but not yet the whole 100%, it is still ahead of us to extract the other 10% of functionality from the existing frameworks and turn it into a nicely decoupled and internally cohesive libraries.

Historically the trend is clear - the first frameworks did everything, the MVC frameworks included all Model, View and Controller components, then came Catalyst which is mostly just the Controller part. But that was not the end of decoupling - the Controller part still can be broken down further. And it was, with libraries like HTML::Engine budding off it, splitting and heading for an independent CPAN life. Now it is time to have a look at Catalyst and other frameworks and think what are the exclusive features that it provides that are not yet available as independent CPAN libraries. We have Plack, that provides all the abstraction needed for easy deployment and testing, we have packages like Bread::Board that provide a generic way of initialising application components, and I am sure there are also other interesting libraries at CPAN that provide other parts of the framework functionality. So what is left?

Maybe I am missing something, but the only important functionality of Catalyst that I cannot find any generic libs for is dispatching - selecting the subroutine (or method) that will handle a given HTTP request. I don't see any reasons why this should be hard to do. In fact I have already started working on that. OK - to be frank, what I started to work on was another web framework. I decided that I would like to have a mini framework and started coding it. Only now, after having the first CRUD example written in it, I can see that all I wrote is just a dispatch system, the other parts are freely available at CPAN. I can use Plack for the web engine abstraction and Plack::Test for testing, I can use Bread::Board for application components creation and Moose as the object framework - no need to reinvent the wheel.

MST explains the ideas lying behind his 'being a bastard'

2010-01-13T02:51:00.000-08:00

It always seemed a bit weird when I saw mst kick himself out of an IRC channel - now we have it all nicely explained. Next time we'll have a self tickling machine, I am sure.

Bread::Board

2009-12-30T00:57:00.000-08:00

Lots of packages in the Catalyst namespace are simple adaptors of external libraries. In many cases what you need from that adaptor layer is only passing initialisation values from the config file to the appropriate call to the object creator. By using the Dependency Injection pattern you can move that initialisation code out of that "deep in the framework guts" place into something that can be called another config file - containing values that are complex objects instead of simple constants. This way your view object can be constructed there and then injected into the application which does not need to know how to construct it. You still might prefer to have a different config file with simple literal configuration values using the standard YAML or Config::General syntax.

If that was too vague for you - don't worry - Stevan promised some more docs for Bread::Board soon. The real take away from all of this is that Bread::Board and future sub-classes of it spares you from the need to writing adaptor classes for all the view or model classes of your yet another web framework. Maybe you'll not need a framework at all - and assemble your application using Bread::Board out of a web dispatcher and other ready made parts.

Frameworks are not yet well refactored libraries.

2009-12-23T23:06:00.000-08:00

At first we had Catalyst. After some refactoring we got HTTP::Engine and now we have PSGI/Plack and the tools around that abstraction that provide a lot of functionality that previously required using Catalyst. Now all of that is in libraries that can be used independently from each other, there is no coupling between the components other than the well defined and pretty minimal PSGI interface.

The title of this blog post might not be universally true, it is possible that there are some kernel parts in some frameworks or even most of the frameworks that could never be in any practical way reduced into sets of libraries. But what I observe is that frameworks tend to grow and only in very exceptional cases spawn libraries that are refactored out of their main body. It is just so much easier to just write new extensions, plugins or things that use the whole framework than find the right abstraction that would let to split some functionality out of the main code base. Certainly I would never guess that the simple PSGI abstraction would let people refactor so much into separate libraries, I bow to it's author. But we should always try:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Antoine de Saint-Exupery

Catalyst app/context split

2009-12-18T09:37:00.000-08:00

This week I've received a bounty from the Enlightened Perl Organisation for my work on the app/context split. As I understand they have doubled the private bounty from groditi who was too busy to keep his promise to do that job and wanted someone to free him from the shame of broken promises. Anyway I am very happy with the outcome. I am not sure if this can be a stable way of founding work on Open Source projects - but it this time it worked :)

Scope and Immutability

2009-12-10T15:06:00.000-08:00

I am sold on the immutability thing. I started to think about various designs and about what immutability of objects would mean to them and I think the results are promising - they tend to have much more clear structure - and just thinking about immutability gives you a chance to see an additional structure in the code. One quite obvious rule of such design is that you cannot keep data of narrower scope in objects with wider scope - and this forces you to think about the scope of various data pieces in your programs. For example in web oriented programming you have (usually) three scopes:

Application - i.e. all data that is persistent to the whole web server process (like configuration)
Session - i.e. data saved into the session variables (there is an opinion, and I mostly subscribe to it, that you should keep that minimal).
Request - i.e. the data you receive from the web client and that you send back to it

An example of framework that does not respect this division is known to everyone - it's Catalyst - fortunately the app/context split is mostly done.

Acknowledgements: Stevan thanks for the web apps scope break down :)

External dependencies once again

2009-12-05T02:29:00.000-08:00

My proposal to add a 'lib_requires' keyword to the META file - with identical semantics as Devel::CheckLib did not make at this round of META spec changes. Now I think that maybe a better strategy would be to start with adding this to the documentation. I am sure many packages already do that - but I have encountered others that don't. What I have in mind is a standard POD section that would be generated by all the module scaffolding generators (like Module::Starter) and which would contain all the information needed to install the external lib in common environments. After this becomes a common thing we would be able to make more informed decisions about including it into the META file.

Warszawa.pm - Biblioteczka (The Library)

2009-11-28T02:08:00.000-08:00

W czwartek odbyło się spotkanie Warszawa.pm. Warta zanotowania jest inauguracja naszej biblioteczki - pożyczyłem od Piotrka Czynnik Ludzki. Dzięki!

Last Thursday there was a Warszawa.pm meeting. Worth noting is the inauguration of our library - Piotrek lended Peopleware: Productive Projects and Teams to me. Thanks!

What are the features that Perl shares with Lisp?

2009-11-20T11:03:00.000-08:00

The author of the famous Higher-Order Perl book (it's available for free download - so no excuse for those that have not yet read it) writes:

Perl is much more like Lisp than it is like C. If you pick up a good
book about Lisp, there will be a section that describes Lisp’s good features. For example, the book Paradigms of Artificial Intelligence Programming, by Peter Norvig, includes a section titled What Makes Lisp Different? that describes seven features of Lisp. Perl shares six of these features; C shares none of them. These are big, important features, features like first-class functions, dynamic access to the symbol table, and automatic storage management.

Dear lazy web: what are the other three features?

On naming things

2009-11-14T13:06:00.000-08:00

One benefit that I have hope will be the result of the IronMan blogging challenge is that Perl people will get more expressive in describing their design. What I can see now is that often there is a strong belief that some code has a good or bad design but any inquiry into why said design is good or bad ends in a 'because you are stupid and you don't understand' argument. While I agree that coding can be sometimes more productive than discussing, I think this rule should not be treated absolutely. All design is trading one thing for another one, having this trade-off stated explicitly gives us the power to weight it intelligently and appropriately to circumstances. In contrast an unnamed intuition can be only an article of our faith in the designers skills.

There is power in naming things as the popularity of Design Patterns shows.

Stuff

2009-11-11T14:16:00.000-08:00

Only remotely related to Perl.

Socio-Technical Systems

I like the definition from The Social Requirements of Technical Systems, now I have what CPAN realy is. I have not yet read the whole Handbook of Research on Socio-Technical Design and Social Networking Systems but the rest seems rather dissapointing.

von Hippel

I recently discovered that most of his work is available on-line: Collected Papers, Downloadable Books. I recommend it to people trying to understand Open Source. He is much more 'technical' than Yochai Benkler. I've just finished reading Open Source Software and the 'Private-Collective' Innovation Model.

Rene Girard

I am working on a paper about on-line conflict in the light of mimetic theory.

Role that adds Traits to multiple classes

2009-11-03T08:47:00.000-08:00

The DBIC model stuff naturally spans the whole FormHandler class tree - there are things that need to be added to the main form class, but also to the Field classes. An obvious example of the latter is the lookup_options method which loads data from the db table (ResultSet object more precisely) appropriate for the field into the field's options array. It does not really touch any global form stuff and:


$field->form->lookup_options( $field, ... )

is like reaching with your left hand to your right pocket. I mean really awkward. So what we need is a form role that would add other roles to the fields classes. I imagined that this could be possible, in theory a role could be a hierarchy that would be applied to a class hierarchy - but I don't think you can do that with current Moose, so instead I used a hook in the method that builds fields and added there code to inject an appropriate Trait.

The code is in new branches in the main HTML::FormHandler repo and in the HTML::FormHandler::Model::DBIC repo. I am waiting for commetns. I would think that this is not something unusual - so I guess others must have already solved similar problems before - I would like to hear what they did.

We have a similar problem with rendering. It is a role now so that it can be easily replaced, but it is naturally recursive, fields should know how to render themselves so we need a way to apply a whole set of roles in one sweep to the whole form class structure. It looks like there really is some abstract super-role in there.
How much is it monkey patching, how much is it action at a distance?

Controller immutability

2009-10-27T22:57:00.000-07:00

On IRC I've got an answer from t0m and others about why they would not use controller attributes for storing data related to the current request - they want to keep their controller object immutable for the request phase. The idea is that if you have an object that is being reused then it would be nice if after each use of that object it is still in the same clean state. That is it is nice to always have the changes contained in a minimal area.

I'll leave it to the experts to list all the advantages of immutable data structures, but I've seen why we needed a similar compartmentalisation in HTML::FormHandler. The FormHandler form object is reusable, you can use it to validate many parameter hashes. We started coding it as a mutable object and storing all the changing values in the form object itself, the first problem was that we kept forgetting to reset them at the start of processing of new data. This lead to many subtle bugs, sometimes difficult to track so we started to just recreate the form object for each processing run. This worked OK, because the form object itself was not that heavy to create until someone had a form object that contained other heavy object and recreating it would require recreating them. So eventually we separated all the state that changes during the processing into a separate Result object, now to clean the form state we need to recreate only that separate object.

It needs to be stated clearly that this separation of immutability is a trade-off (as always) - it is an additional requirement so it increases the complexity of the code and means for example that the $self object in controller actions is underused - because actions should not change it's state (including storing something in it's attributes). Certainly it is not feasible to make such change now and this also would be a trade-off but for the sake of exercising imagination: maybe actions could be methods on the changing part (and not on the controller object) and only use the controller object in some way - this way all the basic code (like that from the Manual) would use $self object in the way common to object oriented code.

Catalyst::Component::InstancePerContext

2009-10-24T02:14:00.000-07:00

How often do you use the controller object? It is the first, most important parameter passed to all the controller methods - but it is nearly never used in the body of these methods. In the code samples in the whole Catalyst::Manual distribution $self is assigned to 117 times - but is used only 38 times (and not all code samples are controllers there). This sounds like a strange case for object oriented code which bases it's definition on the usage of the $self object.

There is much discussion about the stash and how to replace it. The main problem with stash is that it is global - there is no use in replacing it with another global thing. Just like global variables it needs to be replaced by many specialized local things like parameter passing and attributes, the controller object attributes, and for that to be entirely clean we need the a new controller object per response.

OpenID support in LoginSimple

2009-10-21T02:06:00.000-07:00

This week I've added some OpenID support to LoginSimple. Please clone the git repository and try it out! The hard part of the whole LoginSimple idea is to make it universal enough. We need people to try to apply it in many different scenarios and give us some feedback.

The test application is in t/lib/TestAppOpenID.pm, it has some tests in t/07-openid-live.t, but you can also run it with perl -Ilib t/lib/script/testappopenid_server.pl and then go to the http://localhost:3000/login page and test it manually.

HTML::FormHandler - a few notes

2009-10-13T04:35:00.000-07:00

This is not an introduction to HTML::FormHandler - the docs there are very good, but I wanted to write down the main ideas I had when contributing to the design of it.

Let's look at the functions of a form processor:

processing the input into internal representation
checking the values
saving the internal representation into the persistence layer and loading values from it
rendering the form (with input, error messages, etc)

The 'best practice' of rendering the HTML view of an application page is to do it in the templates. Unfortunately there is just too much logic needed for the correct rendering of forms (mostly with the display of errors) to leave that for the templates mini language (like that of TT) - that's why this needs to be done by the form processor. But because of the diversity of the rendering requirements it also needs to be easily replaceable, ideally part by part. Then the application could start with a crude but working HTML rendering, introduced with minimal effort from the programmer since it is built into the processor and later gradually refined and replaced.

CGI.pm and after it Catalyst and other libraries parse the url-encoded and body parameters into a hash of scalar values but internally we would like to manipulate objects like DateTime instead of lists of values for year, month and day. In HFH this conversion is done in two phases - first the input is turn into a deeper structure built of hashes and arrays - then then the processor goes recursively from the leaves and at each node applies a conversion if it is defined for that node.

There is always a question when we should do the checks - sometimes it is easier do do them on the input data sometimes on the internal values. A good example is DateTime - we can check if the day field is a number between 1 and 31, but we also need to check if the whole date is correct and the business rules could add another requirement that the day of the week is Monday or some other operation that is most convenient on the DateTime object. In FormHandler all of this is possible:

How this works? As explained above the processor starts with the leave nodes - and checks the day, month and year fields (here they don't define any transformations only checks, but they could). Then it goes to the parent node date_time and there it has the hash of values from the leave nodes ready for the next transformation and checks - it is passed to:

sub { DateTime->new( shift ) }

If that subroutine triggers an exception, for example when someone passes

{ year => 2009, month => 2, day => 30 }

to it - then the error is recorded with the message This is not a correct date. If it works - the value returned from it, a DateTime object, is then passed to the next check, and later presumably saved to the database. The checks and transformations are taken from a single list - so you can interweave them as you need.
Instead of transform and check in the apply attribute we can also use the Moose types and coercions and also we can mix these two styles (for example coerce some value using Moose types and then check it again). If someone needs even more power - he can also write his own field class with it's own validation method - and do both validation and transformations in there, that method would receive the same hash of child values.

To save the full data structure of values that can contain many inter-related records HTML::FormHandler::Model::DBIC uses
DBIx::Class::ResultSet::RecursiveUpdate. I separated it so that HFH is not related to the hacks I made there :) Or maybe I hoped that perhaps other form processors could reuse my module. Using it you can update a company record together with all it's related addresses in one form.

Barriers to entry

2009-10-07T10:27:00.000-07:00

Chromatic calls us to Remove the Little Pessimizations, to sand off the tiny, rough corners of our software that individually are small, ignorable distractions. I've read somewhere that each barrier to entry reduces the number of users by some percent - and it is a good model to think about it. Each one reduces the number of possible uses and users and the effect is cumulative - hundreds of small barriers make the tool useful in only a narrow niche, but even just one small barrier can make a tool not usable in a particular setting. The question is when fixing these little pessimizations becomes needlesly pedantic.

I believe that fixing CPAN installation issues, each one usually a small issue occurring in some special cases is needed (at least for the most popular modules) if we don't want to confine CPAN users, and since CPAN is the killer application of Perl, then it follows that also Perl users, to a narrow niche.

Another thing is how off-putting are the pessimizations that seem gratuitous, you immediately feel neglected and you imagine how the arrogant developers just don't care about your case. Then there are also pessimizations that look small if you have a big project - but are impenetrable barriers for small projects - this is how Perl lost the place of the most used language for WWW programming to PHP.

CPANHQ

2009-10-06T07:11:00.000-07:00

There is a project to build a Catalyst based CPAN search engine, with the idea to make it more hackable and ultimately incorporate some more 'social' features into it. The original effort was started by Brian Cassidy, he created a cpanhq github repository and wrote some initial code, later there were some features added by Shlomi Fish (in his repo) and now I am working on it. In my repo you'll find a version that finally has a search page and it even works. It uses the SQLite FTS3 full text search engine.

The biggest problem for me is that I had to modify the database quite extensively and now the loading scripts don't work any longer. To show something I decided to load the database into the repo - so now it can take awful long time to download the software, but in return right after the download you should be able to try out the searching. It is not at end user quality - you need to know some internals to use if effectively (like using 'me.name desc' in the order field to sort the packages by name), but it should be reasonably fast. Now I am thinking what are the most useful search strategies.

warszawa.pm

2009-09-29T00:03:00.000-07:00

Witam,

Co prawda zwykle piszę tutaj w języku angielskim - ale reklamówkę naszej grupy Perl Mongers postanowiłem napisać po polsku, żeby było widać, że u nas też się coś dzieje. Mamy już swoją wiki: http://warszawa.pm.org/. Jeszcze nie wszystko działa, ale miejmy nadzieję, że administrator sie postara i naprawi. Żeby dostać konto na wiki trzeba poprosić na liście dyskusyjnej, niestety ze względu na spam nie mogliśmy wiki zrobić zupełnie otwartej. Zapraszamy na wiki, listę i spotkania.

Optional Dependencies Are Going Out of Fashion

2009-09-22T01:00:00.000-07:00

I was reading the Changes document for the latest DBIC release and I spotted:

- Remove the recommends from Makefile.PL, DBIx::Class is not
supposed to have optional dependencies. ever.

I am noting this so that it will not missed by the general public that optional dependencies are going out of fashion.

The problem here is that if an optional dependency enables some useful feature (and otherwise why would you add it?) - then you should expect that someone writes a library using this feature and uploads that library to CPAN. What that other guy should add to his dependencies? How is he supposed to know which of your optional prerequisites are needed for his module? And what if this is not about just two modules in dependency relation but rather a whole chain of them? Then it becomes a mess.

How they do it in Python (core libraries).

2009-09-14T10:27:00.000-07:00

I don't read much from the Python blogosphere - so maybe it is not representative at all - but it seems that Python 'batteries included' approach to core libraries is also approaching its limits. I am sure that debate will happen in any popular Open Source language after reaching some maturity level. It is interesting what kind of compromise is proposed by the author of that piece:

I would first prune the standard library down to a more core focus of libraries that require cross-platform expertise, using widely accepted standards, and stuff needed to simply get a script to run. In terms of cross-platform stuff, that would mean os, multiprocess, etc. These are modules that a typical developer would have a hell of a time getting working on all the OSs that CPython runs on.

How to install Catalyst::Authentication::Credential::OpenID

2009-09-12T06:20:00.000-07:00

This post is meant to be both a quick instruction and another data point for the CPAN installation problems thread of my blog posts.

For those impatient the winning sequence of commands (for Debian/Ubuntu) is:


sudo apt-get install libgmp3-dev
cpanp -i Math::BigInt::GMP

and now you can:


cpanp -i --force Catalyst::Authentication::Credential::OpenID

Now the story - at my pristine installation of Perl 5.10.1 on my Ubuntu box I discovered two problems with C:A:C:O installation. First was a trivial POD error, detected with the new pod tests - this is why you need to do that --force (it is fixed in the dev version). The second is more interesting - installing one of the prerequisites - Crypt::DH seemed to take forever. Fortunately by chance I found a good explanation of the problem in Crypt::DH::GMP docs - apparently for any reasonable performance Crypt::DH needs Math::BigInt::GMP which is not in the prerequisites. Without it it still works - but it can take serveral hours to pass the tests (reported first 3 years ago). So next thing is install Math::BigInt::GMP - unfortunately this is not automatic either - you need to install gmp.h header file first (on Debian/Ubuntu it is the libgmp3-dev package). This is why you need to execute the first two commands.

I hope that will help someone.

Change Management and FOSS authors obligations

2009-09-07T01:52:00.000-07:00

The two recent controversies making rounds in the Perl blogging community made me thinking about what obligations FOSS authors have to those using their code. The traditional answer is none, but on the other hand there are some customs to the contrary - like maintaining API compatibility to at least one version back. Some people have expectation that software not marked upfront as 'experimental' will follow that custom. This is a good custom, because complete freedom for the authors can mean too much chaos for those using their code, but it is both not general enough and too radical in rejecting the traditional no obligations rule. It is not enough - because not all API changes can be made with a backcompat layer, and as shows the example of Dave porting Array::Compare to Moose - there are also non API changes that can make downstream software unusable. And it is too radical because of being an implicit, 'by default' rule. And it is also too ambiguous in it's formulation because it does not state when the new releases can be published. What we need is more general and explicite policies.

Unexpectedly changing a library's API can be like pulling a rug from under the feet of authors of software using that library. In the past that was a rare event - and usually there was time to warn all the parties involved and let them fork the project if needed. Now in CPAN we have such huge dependency tries that even if it is still rare for the individual leaf nodes - it becomes quite frequent for the root nodes where all the changes cascade from the leaves. These deep dependency hierarchies also makes forking a lot more problematic as it can mean forking not just one library - but a whole path of them leading from the leaf up to the root. CPAN might be still rather unique in it's size among distributed library projects - but with the overall exponential growth of FOSS software we can expect that to happen in other projects soon. That's why if we want to continue the process of lego-like assembling software components out of more basic libraries we need those libraries to adhere to some change management policies.

I am inviting FOSS authors to formulate their change management policies (and in particular API deprecation policies) and voluntarily add them to the manuals. I want to stress it - it is not about a top down rule forcing authors to do something - this is an invitation to voluntarily give up a bit of the authors freedom in leading the project in order to make it more useful. The bulk of FOSS can still be published as a gift to the humanity with no strings attached. I just want to show that additional option and also create some pre-canned policies that could be cut and pasted into the documentation just like the FOSS licences are.

What really fails in installation?

2009-09-02T10:49:00.000-07:00

I've finally installed Perl 5.10.1 and I had to reinstall most of the modules I use. I set PERL_MM_USE_DEFAULT=1 and did make for a module that requires half of the modules on my computer. Everything was installed automatically - with no additional intervention from me. The morale of this is that if you are using Ubuntu and if you have installed all the external prerequisites - then the installation is really rather automatic. The problem with systems other than a special flavour of Linux is both hard to solve and rather unsophisticated - we just need more people working on those systems and reporting and solving bugs. Or at least that is the solution I can imagine. But the problem with external prerequisites is where we could do better with just a bit of programming.

I remove memcached from my system and try


zby@zby:~/progs/Enzyme$ cpanp i Cache::Memcached::Managed
Installing Cache::Memcached::Managed (0.20)
Running [/home/zby/local/bin/perl /usr/local/bin/cpanp-run-perl /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/Makefile.PL]...
No executable memcached found: No such file or directory
Running [/usr/bin/make test]...
PERL_DL_NONLAZY=1 /home/zby/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
# Looks like you planned 176 tests but ran 184.
t/001basic.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 176 subtests passed
        (less 177 skipped subtests: -1 okay)
t/002inactive.t .. ok
t/003multi.t ..... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
t/010fork.t ...... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server(s)
t/020grab.t ...... ok
 
Test Summary Report
-------------------
t/001basic.t (Wstat: 65280 Tests: 184 Failed: 8)
  Failed tests: 177-184
  Non-zero exit status: 255
  Parse errors: Bad plan. You planned 176 tests but ran 184.
Files=5, Tests=36330, 18 wallclock secs ( 6.12 usr 0.03 sys + 3.86 cusr 0.20 csys = 10.21 CPU)
Result: FAIL
Failed 1/5 test programs. 8/36330 subtests failed.
make: *** [test_dynamic] Error 255
[ERROR] MAKE TEST failed: Bad file descriptor PERL_DL_NONLAZY=1 /home/zby/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
# Looks like you planned 176 tests but ran 184.
t/001basic.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 176 subtests passed
        (less 177 skipped subtests: -1 okay)
t/002inactive.t .. ok
t/003multi.t ..... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
t/010fork.t ...... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server(s)
t/020grab.t ...... ok
 
Test Summary Report
-------------------
t/001basic.t (Wstat: 65280 Tests: 184 Failed: 8)
  Failed tests: 177-184
  Non-zero exit status: 255
  Parse errors: Bad plan. You planned 176 tests but ran 184.
Files=5, Tests=36330, 18 wallclock secs ( 6.12 usr 0.03 sys + 3.86 cusr 0.20 csys = 10.21 CPU)
Result: FAIL
Failed 1/5 test programs. 8/36330 subtests failed.
make: *** [test_dynamic] Error 255
 
 
 
The tests for 'Cache::Memcached::Managed' failed. Would you like me to proceed anyway or should we abort?
 
Proceed anyway? [y/N]:
[ERROR] Unable to create a new distribution object for 'Cache::Memcached::Managed' -- cannot continue
 
*** Install log written to:
  /home/zby/.cpanplus/install-logs/Cache-Memcached-Managed-0.20-1251915361.log
 
Error installing 'Cache::Memcached::Managed'
Problem installing one or more modules
 
zby@zby:~/progs/Enzyme$

Yeah - the information is there - but now imagine that this was a deep dependency tree with lot's of similar external prerequisites. You try to install the module - it fails, you view the log, locate the problem, install the external dependency, run the installation again and it fails again. Everytime you need to parse the really long installation log and try to understand what happened. It is not that complicated (although it can become complicated when there aren't so explicite error messages, or when there are other installation failures mixed together) - but it is intimidating and really time consuming. Definitively it is not something that a newcomer would be able to do. But external dependencies are not something extraordinary - they are rather standard, this procedure I described here is not exactly a failure mode - it is anticipated.

Someone could argue that when I install Cache::Memcached::Managed I should know that it needs the memcached binary - but it is only obvious for those that know that module, and if that module is somewhere deep in the dependency tree you might not even realize that you are installing it.

Now imagine this:


zby@zby:~/progs/Enzyme$ cpanp i Some::Module
Computing prerequisites.
You have unmet external prerequisites:
memcached
xslt version 1.1.18
...
Please install the prerequisites first then run the installation again.

wouldn't that be helpful?

PS. How about


requires_external memcached => sub { -x "memcached" }

in Makefile.PL?

CPAN installation failures

2009-08-30T07:53:00.001-07:00

The "Chance of all tests passing" at the CPAN dependencies and test results checker is not a scientific measure - but it is great in making it visible that installing from CPAN is still a game of chance with quite big odds of failure. The linked above Catalyst dependencies page quotes the 'probability' of installation failure as 22%, and that is for the most advertised Perl web framework. The important thing in that picture is how this relatively big failure chance comes from the individual small failure chances of the Catalyst prerequisites.

So what can we do? One thing that I would propose is to stop using optional features and dependencies because they really get into the way of automatic installs. We can also simplify the dependency graph of some more important modules, make better statistics about the failure rates and they origins, but probably the most effective way will be to be just more vigilant about the test failures in our modules and in the prerequisites. The important lesson here is that a 1% failure rate seems like nothing important - but it can make our module inadequate to become a prerequisite for some more complex library.

Let's write some deprecation policies

2009-08-23T12:39:00.000-07:00

Sebastian writes about version numbers and backwards compatibility and asks:

Should a project have deprecation policies as soon as it hits CPAN with a development release?

I think the answer should be yes - there should always be a deprecation policy. But this deprecation policy does not need to be the full monty of always keeping the backcompat to at least one major version, using strict deprecation cycles etc - it just needs to be stated explicitely and not relying on the version numbering convention as they are not really universal. There is no negative side to putting a few sentences into the docs, only a bit work to formulate them. And here is my idea:

Let's write some deprecation policies

Let's have a list of well formulated policies covering as much as we can of possible circumstances (from experimental to mature, and from one-person projects to team work) that could be copied to the CPAN distribution docs with minimal effort. For the start - here are some ideas:

Experimental - no policies at all, use at your own risk
Experimental with mailing list announcements of new versions and promise to take into account problems encountered by people testing the new code
Backcompat to one version, with a specific minimal time between versions
Backcompat to one major version, with a specific minimal time between major releases

Update:
In related news mst gives an overview of techniques that can be used for maintaining back-compat - sometimes these are really simple things.

Dissecting the Rails Screencast

2009-08-14T01:07:00.000-07:00

Most comments to my previous post on marketing for the Perl community focused on the visual identity of Perl web sites. It is very important, but marketing is not only that. Ruby on Rails would not be what it is now if not for the RoR screencast. It was:

Simple: Writing a blog engine - what can be simpler?
Unexpected: The first surprise is the medium - that was the one of the first popular screencasts to circulate the Internet. But of course the main load of unexpectedness was in the fact that a simplistic, but still functional, blog engine can be build in 15 minutes. Not months or years of a team of programmers, a designer and a HTML coder - but 15 minutes of one person - that is surprising.
Concrete: That was not an abstract plan, framework with holes to be filled in later - but a concrete implementation. And once again - blog engine - what can be more concrete for the web afficionados?
Credible: You can see it for yourself - it is not a description of how to do it - it is filmed the whole process of doing it. And later you can also try it on your own.
Emotional: Not really - but you don't need to cover all of the points.
Story: This is the most important thing - the screencast is a story. The viewer can easily imagine himself writing his own web application with the same efficiency - this is how stories work.

PS. Don't read this as an argument for (more) screencasts. Yes - they can be effective - but it all depends. What I am advocating here is using that checklist for all marketing projects.

Marketing for Perlers

2009-08-11T04:56:00.000-07:00

Sources on the intertubes report that marketing is becoming a hot topic around the Perl community. It might be a good time to learn a bit about that subject and perhaps exchange some recommendations. I don't want to retract anyone from hiring a professional marketer, to the contrary I think this is the mature thing to do. We need to admit that our own knowledge in that area is limited and seek professional help. But learning a bit more will not hurt.

The book that I would like to recommend is Made to Stick. Why Some Ideas Survive and Others Die ... - it is about creating messages that will be remembered. It is build around a simple list:

Simple
Unexpected
Concrete
Credible
Emotional
Stories

these are the ingredients of an effective marketing message (and yeah the first letters do form an mnemotechnic acronym).

The message needs to be simple, so that the basics are easily understood and can be easily spread around. I think this will be well understood to programmers. It is not about dumbing it down - it is about finding the real core. It needs to be unexpected to catch peoples attention. To hold attention the book advices to use curiosity gaps. Next quality is concrete - this is about involving things the audience already knows well, tangible things and painting a mental image. Credibility comes from inside or from an outside authority, inside credibility comes from using vivid details, statistics but also can be gained by inviting the audience to test for themselves. Emotional is about speaking to emotions. Finally the best messages are stories - the power of story telling comes from the fact that that people reach the conclusions for themselves and they easily absorb them as their own.

Apparently this was tested by the ads business:

The final group was trained for two hours on how to use the six creative templates. Once again, the fifteen best ads were selected by the creative director and tested with consumers. Suddenly these novices sprouted creativity. Their ads were rated as 50 percent more creative and produced a 55 percent more positive attitude toward the products advertised. This is a stunning improvement for a two-hour investment in learning a few basic templates! It appears that there are indeed systematic ways to produce creative ideas.

from excerpts.

The Catalyst API

2009-08-07T00:08:00.000-07:00

Now, after the Moose port is completed, it looks like a good time for reviewing the Catalyst API. It's not a secret that in my opinion some of the Catalyst practice is a cargo cult (for example using forward instead of a subroutine call), other parts could be greatly simplified. In How I Use Catalyst the interesting part is how Dave also does not use some important parts of the Catalyst API, and Marcus in his reply, even though he disagrees with Dave, still he does agree that there might be a more suitable API for some things.

It's time to review the API, check how Moose can improve it, remove the redundancies and make it slim.

One more piece of food for thought: the thing that always bothered me is the first parameter every action receives:

I've never seen it used. I even was never sure if that thing is a controller object or if it is the controller class (i.e. if the actions are class or object methods), is it ever explained in the docs? Now I tested it and the output was:

so at least that action was an object method. I've reviewed the whole Catalyst Tutorial and, even though this parameter is passed to every action example there it is used in only a couple places (as $self->action_for('list') and $self->roles). This does not seem like the best Huffman coding.

Dear TPF - don't send money, send thank you letters

2009-08-02T23:33:00.000-07:00

I have already commented at the Proposed Payments for Clearing Perl 5 Bugs blog post from The Perl Foundation - but now I think I have an even better idea. I am now unemployed, maybe I could try this bug hunting as a way to get some income during these hard times, but the money, that is most probably at stake, could hardly solve any problem for me. Still there is one thing that could have much impact on my situation - I am sure you already know what I have on mind - a printed thank you letter from The Perl Foundation that I could take with me to the interviews. This would not cost much, even for something printed on a nice paper - but it could have a great impact on peoples careers.

Even if the money rewards are already something decided - and no change can be made - TPF could at least accompany them with such a thank you letter. And there should be a way to make it signed by Larry Wall himself - this is crucial for the impressiveness factor.

Fixing dependencies

2009-07-31T01:04:00.000-07:00

Fixing other people code is not easy - but this is what we need to do if we want to clean our dependencies. In answer to John's invitation here is what I did.

I wanted to write some code using Catalyst::Authentication::Credential::OpenID unfortunately one of it's dependencies fails it's tests. Here I'll document how I analysed that and produced a patch, it's not yet approved by the author - but the tests pass.

Step first - checkout the ParanoidAged from it's repository (I had the repository address from an email to the Catalyst mailing list). If you don't have access to the repository of your failing dependency - then just download the source from CPAN unpack it and work from there. Then run the tests:


zby@zby:~/progs/pa$ perl -Ilib t/00-all.t
.
.
.
ok 27
5 second tarpit (tolerance 2)...
not ok 28
#   Failed test at t/00-all.t line 180.
3 second tarpit (tolerance 4)...
ok 29
Killing child pid: 7669
# Looks like you failed 1 test of 29.

OK - the failing test is at line 180. Let's see what is there:

I changed that to:

ran the tests and save my debug output:
zby@zby:~/progs/pa$ perl t/00-all.t 2>debug.
What I got there is:

Now the info is there - but of course I did not spot it at once. I did a lot of more debugging and testing other hypothesis before I focused on: 'x-died' => 'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394.'. Yeah interesting - something died, but no error was reported. The line in question is:

To get some more evidence I changed that to:

Now the debug output changes to:

And again - it is there - but to spot it I had to play with the debugger first. I don't remember what exactly I did - but finally I sow it - there are two evals in this stack trace, the first one is in LWP::Protocol and the second in lib/LWPx/ParanoidAgent.pm line 313. I checked that second eval - and yeah - it tried to retrieve the error already catched by the first one. So when there was a timeout the agent was aborting the retrieval - but the error was being cleared and not reported. I wrote following patch:

and now all the tests pass. Finally I uploaded the patch to RT.

On dependencies

2009-07-29T05:22:00.000-07:00

Ovid writes about Mojolicious::Lite:

The fact that it has no dependencies makes it even more compelling.It's the sort of thing one could more easily attract new developers with. Can you imagine rewriting my old CGI course with something like this? Or maybe a way to introduce new users to Perl? There are many excellent Perl frameworks out there which I would, nonetheless, hesitate to start new users on. Just trying to set up CPAN can be difficult for them; working through even one test failure is often too much. For people who want to jump in and learn "Web programming", though, this might work.

On the other hand Jay advices to stop worrying about dependencies The lie of independence - or - how I learned to stop worrying and love the dependency chain. There is truth in both statements. I have the impression that the problem of CPAN dependencies have improved much recently. The visibility of installation problems directly from the cpan search engine (cpan testers matrix, dependencies review) is a great step ahead. But still installing Catalyst and DBIC and TT is a big hurdle - and I can understand why Ovid is not enthusiastic about teaching Perl novices how to do that.

For me Mojo sounds like iPod - slick and beautiful and really useful - but you are not supposed to change the battery. It is a nicely integrated product - but somehow closed. The author apparently tries to make it perfect, tries to polish every detail - what perhaps would not be possible if he had to compromise with the visions of the authors of the other packages. I can feel his frustration here. But rejecting all dependencies and rewriting everything in his own way sounds a bit extreme and not welcoming.

The big question here is how much can we trust fellow CPAN authors to make the right decisions, can we rely on their work? Can someone, who put's a lot of attention for the details of his API, rely on modules with imperfect APIs, perhaps spoiling his own work and not become insane? The story of the phalanx project shows that there are no easy answers to these questions, but I believe we can do better than rewriting everything from scratch each time.

Google Wave

2009-07-26T03:59:00.000-07:00

One, sometimes feverishly defended, rule of our technical irc channels is about not pasting code into the conversation. To show your code you are supposed to use some web based paste application and then copy the link into the channel. The need to use two separate tools, irc client and web browser, for one task, code review, is just a small annoyance - but eliminating those small annoyances is the only way towards really usable software. Many people have internalized the procedure so much, that they don't think about it any more and I am sure there will be some of them arguing that this is not annoying at all. But there are additional steps to do: choose the tool, paste the link, switch the windows and all this is a burden to the brain, even if it is processed in the background - it still consumes some brain resources.

And now imagine that you have it in one tool - not only those additional steps go away - but it also opens the way for new features like seeing a pointer to the part of the code that the other person is talking about, seeing his edits in real-time and commenting on them immediately etc.

This is why I believe in convergence of communication tools.

Nothing is forever

2009-07-17T23:44:00.000-07:00

I have been writing about the scaffolding metaphor - but after some conversations with my colleagues I realized that I did not do a good job there. I did not conduct well the main idea: nothing is forever. No library and no program is forever and we should not shy off from writing libraries that are useful only at some phase of the application construction. It is still valuable if you can quickly get to the point of having a preview, a simplified prototype of the functionality you need even if eventually you'll remove all of that prototype code from the project. This is very 'agile' and it is also 'worse is better (if you can have it early)' and those libraries, those tools that you use only temporarily, I call the scaffolding.

Now one would ask why write a library that is useful only temporarily? Why not include at once all the parts needed in the final product? There can be several reasons:

something that is only a part of the scaffolding in one project can stay in the final solution in another one
the author of the library might not yet fully understand the problem, and it can take years to reach the right abstraction
the distribution of possibly needed modification can be so uniform that there is no reason to concentrate on one of them more than on another one and it is virtually impossible to cover them all
and finally: nothing is forever - you never know where a project will go in the future and what you'll need to change

A good example (of the third point) are code samples from manuals - you copy them to quickly get something working - but later you modify it and sometimes you change every character from the original. Another one can be the scaffolding code in Rails - you can debate if it fits the first or the second or other points - but it was useful to a lot of people.

Every library can be used as scaffolding - so why not accept this and help the developers in using it that way?

CPAN and code reuse

2009-07-13T14:23:00.000-07:00

CPAN is wonderful thing - it is a great achievement of the whole Perl community and the thing that makes it apart from other languages. But somehow it works mostly for libraries. And what about the other modes of code reuse? How about CPAN for code examples? One can say we don't need CPAN for code examples - everyone is using code examples all the time and they do that without any special infrastructure. But what I see around is that some of CPAN modules are actually code examples in disguise. Some of them would be better without any code in the package and only documentation and tests validating the documented techniques. Hmm maybe we don't really need a separate CPAN for that - maybe we can keep them at the existing one - and just mark them in a special way? And maybe we need some social acceptance for packaging such 'empty' libraries?

And what with applications?

Code reuse styles

2009-07-09T00:57:00.000-07:00

There is much buzz around PHP tools like Drupal and Joomla. They are ready-made solutions, but in fact there is also a lot of tweaking the templates and writing add-ons. It seems to work very well - the clients get something usable from the beginning - and later they can modify it to fit their purpose better. This is very agile. In contrast the reuse in Perl is mostly around libraries. This is much more flexible - you don't tweak a ready solution - you assemble it from the thousands of CPAN packages. But it is also less agile.

I wonder why it is like that. There are some technical differences that lead developers in one or the other direction - but there is also a cultural/psychological cause. Doing some self-analysis I must admit that I am not really enthusiastic about contributing to a Perl CMS (like for example Bricolage) - in my view all the already-build solutions use obsolete libraries. I'd rather contribute to building the perfect library - then code an extension to Bricloage. I have the feeling that I am not alone in that crazy perfectionism around Perl hackers. PHP clearly wins here with more pragmatic approach.

Deprecated code analyzer for HTML::FormHandler

2009-07-06T10:56:00.000-07:00

So the other day I've read Deprecated code analyzer for perl and I thought - this is a GREAT idea! And I stole it for HTML::FormHandler. Now with new releases our users will have a tool to find all the deprecated API calls in their code.

Is using the Catalyst 'forward' method a cargo cult?

2009-07-04T03:01:00.000-07:00

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:


$c->forward( 'some_action' )

and the standard Perl:


$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.