Wednesday, July 07, 2010

Installing dependencies - continued

I don't know what I am doing wrong - but after installing Perl 5.12.1 under perlbrew "cpan -m ." stopped to install the dependencies. It seems that for making and testing it adds the libs in '.cpan/build' to the PERL5LIB search path so both can go on without really installing the dependencies:

Running make for /home/zby/progs/ravlog/.
Prepending blib/arch and blib/lib of 108 build dirs to PERL5LIB; for 'get'

"cpan -t . " works in the same way and if the tests fail then "cpan ." stops there - so I cannot tell if it would install the dependencies in the case when the tests pass.

I don't really know how to investigate it further - but I am noting that as it is very frustrating. Eventually I had to resort to cpanminus ("cpanm --installdeps ." is really handy as it addresses the problem directly).

Thursday, June 24, 2010

Implementing file upload progress bar

At work I was assigned the task of replacing the previous hodgepodge of tools that provide the progress bar functionality for forms with file uploads. At first glance this seems like a trivial thing to do - you periodically observe how much of the file you've got and update the progress bar, but there are details that make it hard to do with the current tools. I report here how I solved that problem - not because I think this is the optimal way - but rather to open the discussion and because I could not find any description of solving it when I googled for it.

First of all the common programming tools (like CGI.pm or Mason that we use here) assume that the page handler receives the whole request as input - and that whole request is not available until after the file is uploaded. So for example 'my $q = CGI.pm->new' will not finish until it is too late to measure the upload progress. The solution to that is to use another page to report the upload progress and call that page via Ajax from Javascript code updating the progress bar. This would work great - but the file is normally uploaded to a temporary file with a random name and the other script would not have any chance to guess it. We need to generate a new random file name in the form page and then pass that name to the form handler script so that it would save the data to that file, and in parallel to the Ajax scripts that would check the size of that file.


To save the data into a specified filename I used the CGI.pm callback feature:

my $q = CGI->new( \&hook, $fh, undef );
...
sub hook {
my ($filename, $buffer, $bytes_read, $fh) = @_;
print $fh substr($buffer, 0, $bytes_read);
$fh->flush();
}

It is described in the subsection called "Progress bars for file uploads and avoiding temp files" of the CGI.pm documentaion, but actually it is a great leap of thought to say that it supports progress bar implementation, you still cannot use it directly to get the progress bar from the CGI object on the form landing page, you still need the separate scripts measuring the progress. For my solution all I needed was to pass the target file name to the code saving the data, this could be easier than writing this callback above. And the callback is still not everything - I yet need a way to pass the generated filename from the form page to that script - and not via form parameters, remember they are not available at that stage. So how can that be done? Simple - as PATH_INFO - which is available in the %ENV hash even before the params are parsed by CGI.pm.

This is the skeleton of the solution - there are a few more details in the actual implementation - but the code will be published soon as Open Source - so I hope everyone will be able to look them up there.

Saturday, June 19, 2010

When those micro-seconds matter

Catalyst is the only web framework that has a mailing list I subscribe to - but I am sure that it happens at others too. In a recurring pattern someone someone posts a benchmark showing that Catalyst for some trivial operation is many times (or many hundred times) slower than some other web framework or for that matter PHP. That does not fail to generate a heated debate - but eventually the seasoned framework developers gain the upper hand with the argument that for all the, often big, web sites they worked on, those few micro-seconds lost in the Catalyst dispatcher never mattered much because the application spent hundred times more in other code fragments and mostly in business logic parts, so shaving off some part of the few micro-seconds would not improve the overall speed more than 1%. This is a great argument, perfectly reasonable and rational but it is biased towards the status quo. It might be true that everywhere where Catalyst is currently used it works great but it is also not hard to imagine an application with very simple business logic that needs to serve millions of users, Twitter anyone? Or an app that does many simple Ajax callbacks. Sure you can always code the speed requiring, simple parts in PHP and keep Catalyst only for the other more heavy-weight tasks but having a universal solution would be so much more convenient.

Sunday, June 13, 2010

Installing dependencies for your packaged for CPAN library

Update: See the comments, apparently the warning against auto_install is outdated information. I don't have any opinion of myself here.

It always puzzled me what is the 'canonical' way to install dependencies for an unpacked distribution, be it downloaded from CPAN or your own in-house product packaged the CPAN way for convenience. Module::Install provides an 'auto_install' option that you add to your Makefile.PL and then perl Makefile.PL; make (or perl Makefile.PL; sudo make) should do that trick. This seemed like a good solution that could be a standard, but now I see that the FAQ of that module says:


Do I really have to avoid auto_install()?

auto_install() has a long history of breaking CPAN toolchains. Lots of people had a bad feeling on it, and have said it should be strongly avoided. In fact it was deprecated and removed once.

Although most of the known problems have been fixed and you can use it more safely than ever, the use of auto_install() is still discouraged, especially if what you are writing is a module to be uploaded on the CPAN. auto_install() does lots of things itself, thus does not always do the same things as other toolchains do (including extra attribute handling, etc; which can be fixed somehow but that's not too DRY). It only supports CPAN and CPANPLUS as its backends. If you use other tools to install, it may still cause a trouble.

Besides, now you can do what auto_install() does with other means. If your CPAN module is new enough, you can pass a dot to the cpan command it provides, and it will install all the required distributions from the CPAN:

$ cpan .

So apparently this is the way to go - using the cpan shell. There is a minor problem with this when used for testing - it would automatically install the package from the directory you are in. Sure you can suppress that you need by adding another command switch (-m), but it would be better if the default action was the safe one.

Saturday, May 22, 2010

Templates and inheritance

In every web site project I have been working on there inevitably came that moment that someone wanted to have a copy of the site with some small changes, or that someone wanted to use a piece the website in something else. In both cases it was all about (initially) some small changes, what they call as skin or a branded site. With programming part we know what to do then - we design base classes for the core undifferentiated functionality and inherit from them in the targeted classes. Most templating engines do have some notion of search path - so this should be doable with templates - but it always proves to be much more messy than it initially seems to be. I'll try to catalogue here all the use cases and how they compare to the standard inheritance in object oriented programming languages and describe them in terms of operations on the search path.


Initial setup


In 'normal' programming languages the atomic piece that can be inherited and overloaded and that is searched on the inheritance path is a method - in case of templates the atomic part is a whole template. The granularity is defined differently, but we can do analogues things with both. Here I will concentrate on the dispatching model familiar for users of most of the MVC frameworks with controllers and their methods as the endpoints (called actions). In this setting all the methods could have a matching template file and inheriting and overriding a template could work on the same level as inheriting and overriding an action.

Basic case


Lets say I have a controller 'MyApp::Controller::Book' with a few actions like 'view', 'list', 'edit' (a CRUD controller) and matching templates in 'templates/Book'. I would like that when calling the render from that controller I did not need to specify the full path of the templates - but just call them by name 'view'. Also when they have a common subcomponents in the same directory they should be able to [% INCLUDE date_formatting %] them by name. This is just like calling methods in a class and it can be accomplished by adding the 'templates/Book' directory at the start of the search path.

Library component


Sometimes we would like to use ready made components from some published library, maybe a CRUD controller for our Books. We inherit from the library class in our 'MyApp::Controller::Book' and we would like its templates to be inherited in a similar way. That means adding '/usr/lib/something/something/CRUD/templates' to our search path somewhere after 'templates/Book'.

Subclassing our own controllers


If our 'MyApp::Controller::Book' is a subclass of our own ''MyApp::Controller::Item' class - this is a very similar case to the above. Here as well we want to insert the search path of the parent class somewhere after 'templates/Book. But in this case what we insert is 'template/Item' - a path relative to our application root.

Application wide components


This is solved simply by adding an application wide element in the search path. It would be analogues to adding stuff to a common base class in standard programming - much more rare there.

Subclassing a whole site


This is the case of 'branded' sites where we would like to subclass everything at once. It is maybe the most common case - but also the most difficult because there are no good analogies/solutions in standard object oriented programming. If 'MyNewApp' inherits from 'MyApp' that does not automatically mean that 'MyNewApp::Controller::Book' springs to existence and can serve book information in the new app (but I've heard there are existing Object Oriented languages that work like that and some people are thinking about implementing this in Perl). So I think we need to first think over how this is handled in the main framework and then we can deduce something about how to treat templates from that. For the limited case of application wide components this can be solved by adding the new template root to the beginning of the search path - but it gets more complicated for all the other cases described above.

Conclusions


I mostly wrote that to have it all in one place - but I think it does show the need to make the template search path dependent on the controller used.

Wednesday, May 19, 2010

A subject for mst's talk

I did not really intended to vote in that silly poll - but I think I have found a subject for the talk that could be highly beneficial for the Perl community: How using the insights from cognitive science (and what else is needed) I am going to make HTML::Zoom actually usable for mere mortals.

Tuesday, May 18, 2010

PSGI and Object Oriented Programming

There is one thing that seems pretty simple - but it took me a bit of mulling over to fully understand


It is a reference to a sub that expects a hash ref representing the HTTP request passed to it as the one and only parameter and which returns an array ref representing a HTTP response. Hmm - yeah that's what web apps do - take the request and return the response.

But then you think - OK, but I like Object Oriented code and I want my application to be an object with attributes and stuff - so what can I do? Well - that is simple. Have your app as object as you like, and to the PSGI layer pass a closure referring to that object. For example if you application object normally handles the requests with a ->handle method then


would return a suitable callback. Which you can then use in your app.psgi file:


And this is exactly what I do in WebNano.

Wednesday, May 12, 2010

Web Applications and Dispatching

Historically the operation of finding the code to handle the web request was trivial - the server just matched the path part of the web request into the filesystem to find the CGI program producing the requested page. But then it got more complicated when we moved from the CGI 'one program equals one page' way into the realm of web applications that serve many pages.

The tree (or DAG) data structure is the underlying concept of both the path part of URL and many programming language library (or class) structures - so there is a natural way of mapping between them. To have code handling requests to related pages in the same file we can extend it so that the last part of the path is interpreted as a subroutine name.

That idea gets a bit more complicated when we realize that we don't want to expose all of our often internal methods (and libraries/classes) to be callable from outside - we need a way to mark them as 'external' and expose only these.

Of course sometimes there are reasons that we don't want to use such a literal mapping, it can be some security (by obscurity) need to hide internal code structures, need for more elegant URLs or other requirements. So sometimes we need to extend it or completely replace it with something else.

The Perl web frameworks (that I know the best) in respect to usage of that mapping can be roughly divided into following styles (many use more than one style so the boundaries are fuzzy):


  1. The 'old' CGI (or PHP) paradigm of one address one program

  2. Using the mapping of paths to libraries, selecting 'externally callable' methods by placing them on a list ( run_modes in CGI::Aplication ).

  3. Using the mapping of paths to libraries, but dispatching to methods configured via a hash (also run_modes in CGI::Application).

  4. Using the mapping of paths to libraries (but not to methods) and calling the 'get' or 'post' method in the landing library (like in Tatsumaki).

  5. Configuring the dispatching by code attributes of the methods (like in Catalyst).

  6. Not using methods but anonymous subroutines. This way it is easy to assign the dispatching configuration to the code by using DSLs (like in Dancer).

  7. Dispatching configuration external to the classes that are dispatched to.


Having the dispatching configuration close to the subroutine code let's you avoid switching between two places in the source code when adding a new action or changing an existing one. I think one of the greatest conveniences of Catalyst was that that configuration was not only in the same file but directly in the subroutine definition.

Saturday, April 17, 2010

WebNano - what I did not say at the London.pm tech meeting

About a month ago I had a presentation at the London.pm tech meeting about the design ideas I plan to experiment in WebNano. There are a few things that I did not have time to tell - and I plan to blog about them. First of all I think I should have admitted that I stole the both the name and the idea from Simon Cozen's MicroMaypole (I hope he will treat that in the terms of the well known quote about imitation and not intellectual property trespassing:).

At the presentation I mentioned that there is not much code in WebNano currently - but maybe this was not the best way to put it. I actually don't plan to ever have much code in it and the example of MicroMaypole shows that perhaps such a minimal web framework can be a practical thing, but I do plan to build some more examples using that minimal code so that I can be sure that even if minimal it does have all the desired features. What are those features will be the subject of other blog posts soon.

Monday, February 01, 2010

Frameworks are framing, libraries are liberating

It seems that the season is open and everyone is writing his new web framework again (all authors of the dozens of other recently created Perl web framework - please accept my apologies - I could not possibly link to all of your babies). If you ask me why is that? My answer is: it is easy now. We have libraries that do nearly everything that the traditional web frameworks provide. You don't need to write much - you just saw together the available patches and voila: your own web framework. You can be proud.

For some time I was thinking: OK, so now we have that cambrian explosion of new frameworks, that's OK, it will all settle down, people will select the best of them and the rest will die off. All we need to do is waiting. We could also speed it up a bit and make it a bit more smooth and rational by learning how to evaluate the frameworks. To do that we need to talk about them, study their functions and compare designs, project goals, project management styles and all of that. This will not be easy, but once we get past the awsome depth of common analyses of the 'this sucks' style and start to think about 'this is good for that and that other thing is better if you need something else', the rest will be incremental.

And I still believe that we need to talk more about design solutions and expand our vocabulary and use less of 'this is stupid' authoritative evaluations. This is something beneficial in any circumstances. But now I also think that since we have the libraries that do the all the work that previously was done by frameworks - then why should we build any more frameworks? Frameworks are coupling, they tie together the libraries that do the actual work and send that loose, uncohesive package to the programmer, who he has no choice, but has to use all of it and in the way prescribed by the framework. But every educated programmer knows that coupling is bad and cohesion is good. So why don't we just use the available nicely decoupled and internally cohesive libraries instead? Yeah, the libraries don't yet do all the 100% of stuff traditionally provided by frameworks, they can do 80% or even 90% of it but not yet the whole 100%, it is still ahead of us to extract the other 10% of functionality from the existing frameworks and turn it into a nicely decoupled and internally cohesive libraries.

Historically the trend is clear - the first frameworks did everything, the MVC frameworks included all Model, View and Controller components, then came Catalyst which is mostly just the Controller part. But that was not the end of decoupling - the Controller part still can be broken down further. And it was, with libraries like HTML::Engine budding off it, splitting and heading for an independent CPAN life. Now it is time to have a look at Catalyst and other frameworks and think what are the exclusive features that it provides that are not yet available as independent CPAN libraries. We have Plack, that provides all the abstraction needed for easy deployment and testing, we have packages like Bread::Board that provide a generic way of initialising application components, and I am sure there are also other interesting libraries at CPAN that provide other parts of the framework functionality. So what is left?

Maybe I am missing something, but the only important functionality of Catalyst that I cannot find any generic libs for is dispatching - selecting the subroutine (or method) that will handle a given HTTP request. I don't see any reasons why this should be hard to do. In fact I have already started working on that. OK - to be frank, what I started to work on was another web framework. I decided that I would like to have a mini framework and started coding it. Only now, after having the first CRUD example written in it, I can see that all I wrote is just a dispatch system, the other parts are freely available at CPAN. I can use Plack for the web engine abstraction and Plack::Test for testing, I can use Bread::Board for application components creation and Moose as the object framework - no need to reinvent the wheel.

Wednesday, January 13, 2010

MST explains the ideas lying behind his 'being a bastard'

It always seemed a bit weird when I saw mst kick himself out of an IRC channel - now we have it all nicely explained. Next time we'll have a self tickling machine, I am sure.

Wednesday, December 30, 2009

Bread::Board

Lots of packages in the Catalyst namespace are simple adaptors of external libraries. In many cases what you need from that adaptor layer is only passing initialisation values from the config file to the appropriate call to the object creator. By using the Dependency Injection pattern you can move that initialisation code out of that "deep in the framework guts" place into something that can be called another config file - containing values that are complex objects instead of simple constants. This way your view object can be constructed there and then injected into the application which does not need to know how to construct it. You still might prefer to have a different config file with simple literal configuration values using the standard YAML or Config::General syntax.

If that was too vague for you - don't worry - Stevan promised some more docs for Bread::Board soon. The real take away from all of this is that Bread::Board and future sub-classes of it spares you from the need to writing adaptor classes for all the view or model classes of your yet another web framework. Maybe you'll not need a framework at all - and assemble your application using Bread::Board out of a web dispatcher and other ready made parts.

Wednesday, December 23, 2009

Frameworks are not yet well refactored libraries.

At first we had Catalyst. After some refactoring we got HTTP::Engine and now we have PSGI/Plack and the tools around that abstraction that provide a lot of functionality that previously required using Catalyst. Now all of that is in libraries that can be used independently from each other, there is no coupling between the components other than the well defined and pretty minimal PSGI interface.

The title of this blog post might not be universally true, it is possible that there are some kernel parts in some frameworks or even most of the frameworks that could never be in any practical way reduced into sets of libraries. But what I observe is that frameworks tend to grow and only in very exceptional cases spawn libraries that are refactored out of their main body. It is just so much easier to just write new extensions, plugins or things that use the whole framework than find the right abstraction that would let to split some functionality out of the main code base. Certainly I would never guess that the simple PSGI abstraction would let people refactor so much into separate libraries, I bow to it's author. But we should always try:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Antoine de Saint-Exupery

Friday, December 18, 2009

Catalyst app/context split

This week I've received a bounty from the Enlightened Perl Organisation for my work on the app/context split. As I understand they have doubled the private bounty from groditi who was too busy to keep his promise to do that job and wanted someone to free him from the shame of broken promises. Anyway I am very happy with the outcome. I am not sure if this can be a stable way of founding work on Open Source projects - but it this time it worked :)

Thursday, December 10, 2009

Scope and Immutability

I am sold on the immutability thing. I started to think about various designs and about what immutability of objects would mean to them and I think the results are promising - they tend to have much more clear structure - and just thinking about immutability gives you a chance to see an additional structure in the code. One quite obvious rule of such design is that you cannot keep data of narrower scope in objects with wider scope - and this forces you to think about the scope of various data pieces in your programs. For example in web oriented programming you have (usually) three scopes:

  • Application - i.e. all data that is persistent to the whole web server process (like configuration)
  • Session - i.e. data saved into the session variables (there is an opinion, and I mostly subscribe to it, that you should keep that minimal).
  • Request - i.e. the data you receive from the web client and that you send back to it

An example of framework that does not respect this division is known to everyone - it's Catalyst - fortunately the app/context split is mostly done.

Acknowledgements: Stevan thanks for the web apps scope break down :)

Saturday, December 05, 2009

External dependencies once again

My proposal to add a 'lib_requires' keyword to the META file - with identical semantics as Devel::CheckLib did not make at this round of META spec changes. Now I think that maybe a better strategy would be to start with adding this to the documentation. I am sure many packages already do that - but I have encountered others that don't. What I have in mind is a standard POD section that would be generated by all the module scaffolding generators (like Module::Starter) and which would contain all the information needed to install the external lib in common environments. After this becomes a common thing we would be able to make more informed decisions about including it into the META file.

Saturday, November 28, 2009

Warszawa.pm - Biblioteczka (The Library)

W czwartek odbyło się spotkanie Warszawa.pm. Warta zanotowania jest inauguracja naszej biblioteczki - pożyczyłem od Piotrka Czynnik Ludzki. Dzięki!


Last Thursday there was a Warszawa.pm meeting. Worth noting is the inauguration of our library - Piotrek lended Peopleware: Productive Projects and Teams to me. Thanks!

Friday, November 20, 2009

What are the features that Perl shares with Lisp?

The author of the famous Higher-Order Perl book (it's available for free download - so no excuse for those that have not yet read it) writes:

Perl is much more like Lisp than it is like C. If you pick up a good
book about Lisp, there will be a section that describes Lisp’s good features. For example, the book Paradigms of Artificial Intelligence Programming, by Peter Norvig, includes a section titled What Makes Lisp Different? that describes seven features of Lisp. Perl shares six of these features; C shares none of them. These are big, important features, features like first-class functions, dynamic access to the symbol table, and automatic storage management.

Dear lazy web: what are the other three features?

Saturday, November 14, 2009

On naming things

One benefit that I have hope will be the result of the IronMan blogging challenge is that Perl people will get more expressive in describing their design. What I can see now is that often there is a strong belief that some code has a good or bad design but any inquiry into why said design is good or bad ends in a 'because you are stupid and you don't understand' argument. While I agree that coding can be sometimes more productive than discussing, I think this rule should not be treated absolutely. All design is trading one thing for another one, having this trade-off stated explicitly gives us the power to weight it intelligently and appropriately to circumstances. In contrast an unnamed intuition can be only an article of our faith in the designers skills.

There is power in naming things as the popularity of Design Patterns shows.

Wednesday, November 11, 2009

Stuff

Only remotely related to Perl.

Socio-Technical Systems


I like the definition from The Social Requirements of Technical Systems, now I have what CPAN realy is. I have not yet read the whole Handbook of Research on Socio-Technical Design and Social Networking Systems but the rest seems rather dissapointing.

von Hippel


I recently discovered that most of his work is available on-line: Collected Papers, Downloadable Books. I recommend it to people trying to understand Open Source. He is much more 'technical' than Yochai Benkler. I've just finished reading Open Source Software and the 'Private-Collective' Innovation Model.

Rene Girard


I am working on a paper about on-line conflict in the light of mimetic theory.

Tuesday, November 03, 2009

Role that adds Traits to multiple classes

The DBIC model stuff naturally spans the whole FormHandler class tree - there are things that need to be added to the main form class, but also to the Field classes. An obvious example of the latter is the lookup_options method which loads data from the db table (ResultSet object more precisely) appropriate for the field into the field's options array. It does not really touch any global form stuff and:

$field->form->lookup_options( $field, ... )

is like reaching with your left hand to your right pocket. I mean really awkward. So what we need is a form role that would add other roles to the fields classes. I imagined that this could be possible, in theory a role could be a hierarchy that would be applied to a class hierarchy - but I don't think you can do that with current Moose, so instead I used a hook in the method that builds fields and added there code to inject an appropriate Trait.

The code is in new branches in the main HTML::FormHandler repo and in the HTML::FormHandler::Model::DBIC repo. I am waiting for commetns. I would think that this is not something unusual - so I guess others must have already solved similar problems before - I would like to hear what they did.

We have a similar problem with rendering. It is a role now so that it can be easily replaced, but it is naturally recursive, fields should know how to render themselves so we need a way to apply a whole set of roles in one sweep to the whole form class structure. It looks like there really is some abstract super-role in there.
How much is it monkey patching, how much is it action at a distance?

Tuesday, October 27, 2009

Controller immutability

On IRC I've got an answer from t0m and others about why they would not use controller attributes for storing data related to the current request - they want to keep their controller object immutable for the request phase. The idea is that if you have an object that is being reused then it would be nice if after each use of that object it is still in the same clean state. That is it is nice to always have the changes contained in a minimal area.

I'll leave it to the experts to list all the advantages of immutable data structures, but I've seen why we needed a similar compartmentalisation in HTML::FormHandler. The FormHandler form object is reusable, you can use it to validate many parameter hashes. We started coding it as a mutable object and storing all the changing values in the form object itself, the first problem was that we kept forgetting to reset them at the start of processing of new data. This lead to many subtle bugs, sometimes difficult to track so we started to just recreate the form object for each processing run. This worked OK, because the form object itself was not that heavy to create until someone had a form object that contained other heavy object and recreating it would require recreating them. So eventually we separated all the state that changes during the processing into a separate Result object, now to clean the form state we need to recreate only that separate object.

It needs to be stated clearly that this separation of immutability is a trade-off (as always) - it is an additional requirement so it increases the complexity of the code and means for example that the $self object in controller actions is underused - because actions should not change it's state (including storing something in it's attributes). Certainly it is not feasible to make such change now and this also would be a trade-off but for the sake of exercising imagination: maybe actions could be methods on the changing part (and not on the controller object) and only use the controller object in some way - this way all the basic code (like that from the Manual) would use $self object in the way common to object oriented code.

Saturday, October 24, 2009

Catalyst::Component::InstancePerContext

How often do you use the controller object? It is the first, most important parameter passed to all the controller methods - but it is nearly never used in the body of these methods. In the code samples in the whole Catalyst::Manual distribution $self is assigned to 117 times - but is used only 38 times (and not all code samples are controllers there). This sounds like a strange case for object oriented code which bases it's definition on the usage of the $self object.

There is much discussion about the stash and how to replace it. The main problem with stash is that it is global - there is no use in replacing it with another global thing. Just like global variables it needs to be replaced by many specialized local things like parameter passing and attributes, the controller object attributes, and for that to be entirely clean we need the a new controller object per response.

Wednesday, October 21, 2009

OpenID support in LoginSimple

This week I've added some OpenID support to LoginSimple. Please clone the git repository and try it out! The hard part of the whole LoginSimple idea is to make it universal enough. We need people to try to apply it in many different scenarios and give us some feedback.

The test application is in t/lib/TestAppOpenID.pm, it has some tests in t/07-openid-live.t, but you can also run it with perl -Ilib t/lib/script/testappopenid_server.pl and then go to the http://localhost:3000/login page and test it manually.

Tuesday, October 13, 2009

HTML::FormHandler - a few notes

This is not an introduction to HTML::FormHandler - the docs there are very good, but I wanted to write down the main ideas I had when contributing to the design of it.

Let's look at the functions of a form processor:

  1. processing the input into internal representation
  2. checking the values
  3. saving the internal representation into the persistence layer and loading values from it
  4. rendering the form (with input, error messages, etc)

The 'best practice' of rendering the HTML view of an application page is to do it in the templates. Unfortunately there is just too much logic needed for the correct rendering of forms (mostly with the display of errors) to leave that for the templates mini language (like that of TT) - that's why this needs to be done by the form processor. But because of the diversity of the rendering requirements it also needs to be easily replaceable, ideally part by part. Then the application could start with a crude but working HTML rendering, introduced with minimal effort from the programmer since it is built into the processor and later gradually refined and replaced.

CGI.pm and after it Catalyst and other libraries parse the url-encoded and body parameters into a hash of scalar values but internally we would like to manipulate objects like DateTime instead of lists of values for year, month and day. In HFH this conversion is done in two phases - first the input is turn into a deeper structure built of hashes and arrays - then then the processor goes recursively from the leaves and at each node applies a conversion if it is defined for that node.

There is always a question when we should do the checks - sometimes it is easier do do them on the input data sometimes on the internal values. A good example is DateTime - we can check if the day field is a number between 1 and 31, but we also need to check if the whole date is correct and the business rules could add another requirement that the day of the week is Monday or some other operation that is most convenient on the DateTime object. In FormHandler all of this is possible:



How this works? As explained above the processor starts with the leave nodes - and checks the day, month and year fields (here they don't define any transformations only checks, but they could). Then it goes to the parent node date_time and there it has the hash of values from the leave nodes ready for the next transformation and checks - it is passed to:

sub { DateTime->new( shift ) }

If that subroutine triggers an exception, for example when someone passes

{ year => 2009, month => 2, day => 30 }

to it - then the error is recorded with the message This is not a correct date. If it works - the value returned from it, a DateTime object, is then passed to the next check, and later presumably saved to the database. The checks and transformations are taken from a single list - so you can interweave them as you need.
Instead of transform and check in the apply attribute we can also use the Moose types and coercions and also we can mix these two styles (for example coerce some value using Moose types and then check it again). If someone needs even more power - he can also write his own field class with it's own validation method - and do both validation and transformations in there, that method would receive the same hash of child values.

To save the full data structure of values that can contain many inter-related records HTML::FormHandler::Model::DBIC uses
DBIx::Class::ResultSet::RecursiveUpdate. I separated it so that HFH is not related to the hacks I made there :) Or maybe I hoped that perhaps other form processors could reuse my module. Using it you can update a company record together with all it's related addresses in one form.

Wednesday, October 07, 2009

Barriers to entry

Chromatic calls us to Remove the Little Pessimizations, to sand off the tiny, rough corners of our software that individually are small, ignorable distractions. I've read somewhere that each barrier to entry reduces the number of users by some percent - and it is a good model to think about it. Each one reduces the number of possible uses and users and the effect is cumulative - hundreds of small barriers make the tool useful in only a narrow niche, but even just one small barrier can make a tool not usable in a particular setting. The question is when fixing these little pessimizations becomes needlesly pedantic.

I believe that fixing CPAN installation issues, each one usually a small issue occurring in some special cases is needed (at least for the most popular modules) if we don't want to confine CPAN users, and since CPAN is the killer application of Perl, then it follows that also Perl users, to a narrow niche.

Another thing is how off-putting are the pessimizations that seem gratuitous, you immediately feel neglected and you imagine how the arrogant developers just don't care about your case. Then there are also pessimizations that look small if you have a big project - but are impenetrable barriers for small projects - this is how Perl lost the place of the most used language for WWW programming to PHP.

Tuesday, October 06, 2009

CPANHQ

There is a project to build a Catalyst based CPAN search engine, with the idea to make it more hackable and ultimately incorporate some more 'social' features into it. The original effort was started by Brian Cassidy, he created a cpanhq github repository and wrote some initial code, later there were some features added by Shlomi Fish (in his repo) and now I am working on it. In my repo you'll find a version that finally has a search page and it even works. It uses the SQLite FTS3 full text search engine.

The biggest problem for me is that I had to modify the database quite extensively and now the loading scripts don't work any longer. To show something I decided to load the database into the repo - so now it can take awful long time to download the software, but in return right after the download you should be able to try out the searching. It is not at end user quality - you need to know some internals to use if effectively (like using 'me.name desc' in the order field to sort the packages by name), but it should be reasonably fast. Now I am thinking what are the most useful search strategies.

Tuesday, September 29, 2009

warszawa.pm

Witam,

Co prawda zwykle piszę tutaj w języku angielskim - ale reklamówkę naszej grupy Perl Mongers postanowiłem napisać po polsku, żeby było widać, że u nas też się coś dzieje. Mamy już swoją wiki: http://warszawa.pm.org/. Jeszcze nie wszystko działa, ale miejmy nadzieję, że administrator sie postara i naprawi. Żeby dostać konto na wiki trzeba poprosić na liście dyskusyjnej, niestety ze względu na spam nie mogliśmy wiki zrobić zupełnie otwartej. Zapraszamy na wiki, listę i spotkania.

Tuesday, September 22, 2009

Optional Dependencies Are Going Out of Fashion

I was reading the Changes document for the latest DBIC release and I spotted:

- Remove the recommends from Makefile.PL, DBIx::Class is not
supposed to have optional dependencies. ever.

I am noting this so that it will not missed by the general public that optional dependencies are going out of fashion.

The problem here is that if an optional dependency enables some useful feature (and otherwise why would you add it?) - then you should expect that someone writes a library using this feature and uploads that library to CPAN. What that other guy should add to his dependencies? How is he supposed to know which of your optional prerequisites are needed for his module? And what if this is not about just two modules in dependency relation but rather a whole chain of them? Then it becomes a mess.

Monday, September 14, 2009

How they do it in Python (core libraries).

I don't read much from the Python blogosphere - so maybe it is not representative at all - but it seems that Python 'batteries included' approach to core libraries is also approaching its limits. I am sure that debate will happen in any popular Open Source language after reaching some maturity level. It is interesting what kind of compromise is proposed by the author of that piece:

I would first prune the standard library down to a more core focus of libraries that require cross-platform expertise, using widely accepted standards, and stuff needed to simply get a script to run. In terms of cross-platform stuff, that would mean os, multiprocess, etc. These are modules that a typical developer would have a hell of a time getting working on all the OSs that CPython runs on.

Saturday, September 12, 2009

How to install Catalyst::Authentication::Credential::OpenID

This post is meant to be both a quick instruction and another data point for the CPAN installation problems thread of my blog posts.

For those impatient the winning sequence of commands (for Debian/Ubuntu) is:

sudo apt-get install libgmp3-dev
cpanp -i Math::BigInt::GMP

and now you can:

cpanp -i --force Catalyst::Authentication::Credential::OpenID


Now the story - at my pristine installation of Perl 5.10.1 on my Ubuntu box I discovered two problems with C:A:C:O installation. First was a trivial POD error, detected with the new pod tests - this is why you need to do that --force (it is fixed in the dev version). The second is more interesting - installing one of the prerequisites - Crypt::DH seemed to take forever. Fortunately by chance I found a good explanation of the problem in Crypt::DH::GMP docs - apparently for any reasonable performance Crypt::DH needs Math::BigInt::GMP which is not in the prerequisites. Without it it still works - but it can take serveral hours to pass the tests (reported first 3 years ago). So next thing is install Math::BigInt::GMP - unfortunately this is not automatic either - you need to install gmp.h header file first (on Debian/Ubuntu it is the libgmp3-dev package). This is why you need to execute the first two commands.

I hope that will help someone.

Monday, September 07, 2009

Change Management and FOSS authors obligations

The two recent controversies making rounds in the Perl blogging community made me thinking about what obligations FOSS authors have to those using their code. The traditional answer is none, but on the other hand there are some customs to the contrary - like maintaining API compatibility to at least one version back. Some people have expectation that software not marked upfront as 'experimental' will follow that custom. This is a good custom, because complete freedom for the authors can mean too much chaos for those using their code, but it is both not general enough and too radical in rejecting the traditional no obligations rule. It is not enough - because not all API changes can be made with a backcompat layer, and as shows the example of Dave porting Array::Compare to Moose - there are also non API changes that can make downstream software unusable. And it is too radical because of being an implicit, 'by default' rule. And it is also too ambiguous in it's formulation because it does not state when the new releases can be published. What we need is more general and explicite policies.

Unexpectedly changing a library's API can be like pulling a rug from under the feet of authors of software using that library. In the past that was a rare event - and usually there was time to warn all the parties involved and let them fork the project if needed. Now in CPAN we have such huge dependency tries that even if it is still rare for the individual leaf nodes - it becomes quite frequent for the root nodes where all the changes cascade from the leaves. These deep dependency hierarchies also makes forking a lot more problematic as it can mean forking not just one library - but a whole path of them leading from the leaf up to the root. CPAN might be still rather unique in it's size among distributed library projects - but with the overall exponential growth of FOSS software we can expect that to happen in other projects soon. That's why if we want to continue the process of lego-like assembling software components out of more basic libraries we need those libraries to adhere to some change management policies.

I am inviting FOSS authors to formulate their change management policies (and in particular API deprecation policies) and voluntarily add them to the manuals. I want to stress it - it is not about a top down rule forcing authors to do something - this is an invitation to voluntarily give up a bit of the authors freedom in leading the project in order to make it more useful. The bulk of FOSS can still be published as a gift to the humanity with no strings attached. I just want to show that additional option and also create some pre-canned policies that could be cut and pasted into the documentation just like the FOSS licences are.

Wednesday, September 02, 2009

What really fails in installation?

I've finally installed Perl 5.10.1 and I had to reinstall most of the modules I use. I set PERL_MM_USE_DEFAULT=1 and did make for a module that requires half of the modules on my computer. Everything was installed automatically - with no additional intervention from me. The morale of this is that if you are using Ubuntu and if you have installed all the external prerequisites - then the installation is really rather automatic. The problem with systems other than a special flavour of Linux is both hard to solve and rather unsophisticated - we just need more people working on those systems and reporting and solving bugs. Or at least that is the solution I can imagine. But the problem with external prerequisites is where we could do better with just a bit of programming.

I remove memcached from my system and try

zby@zby:~/progs/Enzyme$ cpanp i Cache::Memcached::Managed
Installing Cache::Memcached::Managed (0.20)
Running [/home/zby/local/bin/perl /usr/local/bin/cpanp-run-perl /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/Makefile.PL]...
No executable memcached found: No such file or directory
Running [/usr/bin/make test]...
PERL_DL_NONLAZY=1 /home/zby/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
# Looks like you planned 176 tests but ran 184.
t/001basic.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 176 subtests passed
(less 177 skipped subtests: -1 okay)
t/002inactive.t .. ok
t/003multi.t ..... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
t/010fork.t ...... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server(s)
t/020grab.t ...... ok

Test Summary Report
-------------------
t/001basic.t (Wstat: 65280 Tests: 184 Failed: 8)
Failed tests: 177-184
Non-zero exit status: 255
Parse errors: Bad plan. You planned 176 tests but ran 184.
Files=5, Tests=36330, 18 wallclock secs ( 6.12 usr 0.03 sys + 3.86 cusr 0.20 csys = 10.21 CPU)
Result: FAIL
Failed 1/5 test programs. 8/36330 subtests failed.
make: *** [test_dynamic] Error 255
[ERROR] MAKE TEST failed: Bad file descriptor PERL_DL_NONLAZY=1 /home/zby/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
# Looks like you planned 176 tests but ran 184.
t/001basic.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 176 subtests passed
(less 177 skipped subtests: -1 okay)
t/002inactive.t .. ok
t/003multi.t ..... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server
t/010fork.t ...... ok
Can't exec "memcached": No such file or directory at /home/zby/.cpanplus/5.10.1/build/Cache-Memcached-Managed-0.20/blib/lib/Cache/Memcached/Managed.pm line 843.
#
# Stopped memcached server(s)
t/020grab.t ...... ok

Test Summary Report
-------------------
t/001basic.t (Wstat: 65280 Tests: 184 Failed: 8)
Failed tests: 177-184
Non-zero exit status: 255
Parse errors: Bad plan. You planned 176 tests but ran 184.
Files=5, Tests=36330, 18 wallclock secs ( 6.12 usr 0.03 sys + 3.86 cusr 0.20 csys = 10.21 CPU)
Result: FAIL
Failed 1/5 test programs. 8/36330 subtests failed.
make: *** [test_dynamic] Error 255



The tests for 'Cache::Memcached::Managed' failed. Would you like me to proceed anyway or should we abort?

Proceed anyway? [y/N]:
[ERROR] Unable to create a new distribution object for 'Cache::Memcached::Managed' -- cannot continue

*** Install log written to:
/home/zby/.cpanplus/install-logs/Cache-Memcached-Managed-0.20-1251915361.log

Error installing 'Cache::Memcached::Managed'
Problem installing one or more modules

zby@zby:~/progs/Enzyme$


Yeah - the information is there - but now imagine that this was a deep dependency tree with lot's of similar external prerequisites. You try to install the module - it fails, you view the log, locate the problem, install the external dependency, run the installation again and it fails again. Everytime you need to parse the really long installation log and try to understand what happened. It is not that complicated (although it can become complicated when there aren't so explicite error messages, or when there are other installation failures mixed together) - but it is intimidating and really time consuming. Definitively it is not something that a newcomer would be able to do. But external dependencies are not something extraordinary - they are rather standard, this procedure I described here is not exactly a failure mode - it is anticipated.

Someone could argue that when I install Cache::Memcached::Managed I should know that it needs the memcached binary - but it is only obvious for those that know that module, and if that module is somewhere deep in the dependency tree you might not even realize that you are installing it.

Now imagine this:

zby@zby:~/progs/Enzyme$ cpanp i Some::Module
Computing prerequisites.
You have unmet external prerequisites:
memcached
xslt version 1.1.18
...
Please install the prerequisites first then run the installation again.

wouldn't that be helpful?

PS. How about

requires_external memcached => sub { -x "memcached" }

in Makefile.PL?

Sunday, August 30, 2009

CPAN installation failures

The "Chance of all tests passing" at the CPAN dependencies and test results checker is not a scientific measure - but it is great in making it visible that installing from CPAN is still a game of chance with quite big odds of failure. The linked above Catalyst dependencies page quotes the 'probability' of installation failure as 22%, and that is for the most advertised Perl web framework. The important thing in that picture is how this relatively big failure chance comes from the individual small failure chances of the Catalyst prerequisites.

So what can we do? One thing that I would propose is to stop using optional features and dependencies because they really get into the way of automatic installs. We can also simplify the dependency graph of some more important modules, make better statistics about the failure rates and they origins, but probably the most effective way will be to be just more vigilant about the test failures in our modules and in the prerequisites. The important lesson here is that a 1% failure rate seems like nothing important - but it can make our module inadequate to become a prerequisite for some more complex library.

Sunday, August 23, 2009

Let's write some deprecation policies

Sebastian writes about version numbers and backwards compatibility and asks:

Should a project have deprecation policies as soon as it hits CPAN with a development release?

I think the answer should be yes - there should always be a deprecation policy. But this deprecation policy does not need to be the full monty of always keeping the backcompat to at least one major version, using strict deprecation cycles etc - it just needs to be stated explicitely and not relying on the version numbering convention as they are not really universal. There is no negative side to putting a few sentences into the docs, only a bit work to formulate them. And here is my idea:

Let's write some deprecation policies

Let's have a list of well formulated policies covering as much as we can of possible circumstances (from experimental to mature, and from one-person projects to team work) that could be copied to the CPAN distribution docs with minimal effort. For the start - here are some ideas:

  • Experimental - no policies at all, use at your own risk
  • Experimental with mailing list announcements of new versions and promise to take into account problems encountered by people testing the new code
  • Backcompat to one version, with a specific minimal time between versions
  • Backcompat to one major version, with a specific minimal time between major releases

Update:
In related news mst gives an overview of techniques that can be used for maintaining back-compat - sometimes these are really simple things.

Friday, August 14, 2009

Dissecting the Rails Screencast

Most comments to my previous post on marketing for the Perl community focused on the visual identity of Perl web sites. It is very important, but marketing is not only that. Ruby on Rails would not be what it is now if not for the RoR screencast. It was:

Simple

Writing a blog engine - what can be simpler?

Unexpected


The first surprise is the medium - that was the one of the first popular screencasts to circulate the Internet. But of course the main load of unexpectedness was in the fact that a simplistic, but still functional, blog engine can be build in 15 minutes. Not months or years of a team of programmers, a designer and a HTML coder - but 15 minutes of one person - that is surprising.


Concrete


That was not an abstract plan, framework with holes to be filled in later - but a concrete implementation. And once again - blog engine - what can be more concrete for the web afficionados?


Credible


You can see it for yourself - it is not a description of how to do it - it is filmed the whole process of doing it. And later you can also try it on your own.


Emotional


Not really - but you don't need to cover all of the points.


Story


This is the most important thing - the screencast is a story. The viewer can easily imagine himself writing his own web application with the same efficiency - this is how stories work.



PS. Don't read this as an argument for (more) screencasts. Yes - they can be effective - but it all depends. What I am advocating here is using that checklist for all marketing projects.

Tuesday, August 11, 2009

Marketing for Perlers

Sources on the intertubes report that marketing is becoming a hot topic around the Perl community. It might be a good time to learn a bit about that subject and perhaps exchange some recommendations. I don't want to retract anyone from hiring a professional marketer, to the contrary I think this is the mature thing to do. We need to admit that our own knowledge in that area is limited and seek professional help. But learning a bit more will not hurt.


The book that I would like to recommend is Made to Stick. Why Some Ideas Survive and Others Die ... - it is about creating messages that will be remembered. It is build around a simple list:

  • Simple
  • Unexpected
  • Concrete
  • Credible
  • Emotional
  • Stories

these are the ingredients of an effective marketing message (and yeah the first letters do form an mnemotechnic acronym).

The message needs to be simple, so that the basics are easily understood and can be easily spread around. I think this will be well understood to programmers. It is not about dumbing it down - it is about finding the real core. It needs to be unexpected to catch peoples attention. To hold attention the book advices to use curiosity gaps. Next quality is concrete - this is about involving things the audience already knows well, tangible things and painting a mental image. Credibility comes from inside or from an outside authority, inside credibility comes from using vivid details, statistics but also can be gained by inviting the audience to test for themselves. Emotional is about speaking to emotions. Finally the best messages are stories - the power of story telling comes from the fact that that people reach the conclusions for themselves and they easily absorb them as their own.

Apparently this was tested by the ads business:

The final group was trained for two hours on how to use the six creative templates. Once again, the fifteen best ads were selected by the creative director and tested with consumers. Suddenly these novices sprouted creativity. Their ads were rated as 50 percent more creative and produced a 55 percent more positive attitude toward the products advertised. This is a stunning improvement for a two-hour investment in learning a few basic templates! It appears that there are indeed systematic ways to produce creative ideas.

from excerpts.

Friday, August 07, 2009

The Catalyst API

Now, after the Moose port is completed, it looks like a good time for reviewing the Catalyst API. It's not a secret that in my opinion some of the Catalyst practice is a cargo cult (for example using forward instead of a subroutine call), other parts could be greatly simplified. In How I Use Catalyst the interesting part is how Dave also does not use some important parts of the Catalyst API, and Marcus in his reply, even though he disagrees with Dave, still he does agree that there might be a more suitable API for some things.

It's time to review the API, check how Moose can improve it, remove the redundancies and make it slim.

One more piece of food for thought: the thing that always bothered me is the first parameter every action receives:



I've never seen it used. I even was never sure if that thing is a controller object or if it is the controller class (i.e. if the actions are class or object methods), is it ever explained in the docs? Now I tested it and the output was:



so at least that action was an object method. I've reviewed the whole Catalyst Tutorial and, even though this parameter is passed to every action example there it is used in only a couple places (as $self->action_for('list') and $self->roles). This does not seem like the best Huffman coding.

Sunday, August 02, 2009

Dear TPF - don't send money, send thank you letters

I have already commented at the Proposed Payments for Clearing Perl 5 Bugs blog post from The Perl Foundation - but now I think I have an even better idea. I am now unemployed, maybe I could try this bug hunting as a way to get some income during these hard times, but the money, that is most probably at stake, could hardly solve any problem for me. Still there is one thing that could have much impact on my situation - I am sure you already know what I have on mind - a printed thank you letter from The Perl Foundation that I could take with me to the interviews. This would not cost much, even for something printed on a nice paper - but it could have a great impact on peoples careers.

Even if the money rewards are already something decided - and no change can be made - TPF could at least accompany them with such a thank you letter. And there should be a way to make it signed by Larry Wall himself - this is crucial for the impressiveness factor.

Friday, July 31, 2009

Fixing dependencies

Fixing other people code is not easy - but this is what we need to do if we want to clean our dependencies. In answer to John's invitation here is what I did.

I wanted to write some code using Catalyst::Authentication::Credential::OpenID unfortunately one of it's dependencies fails it's tests. Here I'll document how I analysed that and produced a patch, it's not yet approved by the author - but the tests pass.

Step first - checkout the ParanoidAged from it's repository (I had the repository address from an email to the Catalyst mailing list). If you don't have access to the repository of your failing dependency - then just download the source from CPAN unpack it and work from there. Then run the tests:

zby@zby:~/progs/pa$ perl -Ilib t/00-all.t
.
.
.
ok 27
5 second tarpit (tolerance 2)...
not ok 28
# Failed test at t/00-all.t line 180.
3 second tarpit (tolerance 4)...
ok 29
Killing child pid: 7669
# Looks like you failed 1 test of 29.

OK - the failing test is at line 180. Let's see what is there:

I changed that to:

ran the tests and save my debug output:
zby@zby:~/progs/pa$ perl t/00-all.t 2>debug.
What I got there is:



Now the info is there - but of course I did not spot it at once. I did a lot of more debugging and testing other hypothesis before I focused on: 'x-died' => 'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394.'. Yeah interesting - something died, but no error was reported. The line in question is:



To get some more evidence I changed that to:



Now the debug output changes to:



And again - it is there - but to spot it I had to play with the debugger first. I don't remember what exactly I did - but finally I sow it - there are two evals in this stack trace, the first one is in LWP::Protocol and the second in lib/LWPx/ParanoidAgent.pm line 313. I checked that second eval - and yeah - it tried to retrieve the error already catched by the first one. So when there was a timeout the agent was aborting the retrieval - but the error was being cleared and not reported. I wrote following patch:



and now all the tests pass. Finally I uploaded the patch to RT.

Wednesday, July 29, 2009

On dependencies

Ovid writes about Mojolicious::Lite:

The fact that it has no dependencies makes it even more compelling.It's the sort of thing one could more easily attract new developers with. Can you imagine rewriting my old CGI course with something like this? Or maybe a way to introduce new users to Perl? There are many excellent Perl frameworks out there which I would, nonetheless, hesitate to start new users on. Just trying to set up CPAN can be difficult for them; working through even one test failure is often too much. For people who want to jump in and learn "Web programming", though, this might work.

On the other hand Jay advices to stop worrying about dependencies The lie of independence - or - how I learned to stop worrying and love the dependency chain. There is truth in both statements. I have the impression that the problem of CPAN dependencies have improved much recently. The visibility of installation problems directly from the cpan search engine (cpan testers matrix, dependencies review) is a great step ahead. But still installing Catalyst and DBIC and TT is a big hurdle - and I can understand why Ovid is not enthusiastic about teaching Perl novices how to do that.

For me Mojo sounds like iPod - slick and beautiful and really useful - but you are not supposed to change the battery. It is a nicely integrated product - but somehow closed. The author apparently tries to make it perfect, tries to polish every detail - what perhaps would not be possible if he had to compromise with the visions of the authors of the other packages. I can feel his frustration here. But rejecting all dependencies and rewriting everything in his own way sounds a bit extreme and not welcoming.

The big question here is how much can we trust fellow CPAN authors to make the right decisions, can we rely on their work? Can someone, who put's a lot of attention for the details of his API, rely on modules with imperfect APIs, perhaps spoiling his own work and not become insane? The story of the phalanx project shows that there are no easy answers to these questions, but I believe we can do better than rewriting everything from scratch each time.

Sunday, July 26, 2009

Google Wave

One, sometimes feverishly defended, rule of our technical irc channels is about not pasting code into the conversation. To show your code you are supposed to use some web based paste application and then copy the link into the channel. The need to use two separate tools, irc client and web browser, for one task, code review, is just a small annoyance - but eliminating those small annoyances is the only way towards really usable software. Many people have internalized the procedure so much, that they don't think about it any more and I am sure there will be some of them arguing that this is not annoying at all. But there are additional steps to do: choose the tool, paste the link, switch the windows and all this is a burden to the brain, even if it is processed in the background - it still consumes some brain resources.

And now imagine that you have it in one tool - not only those additional steps go away - but it also opens the way for new features like seeing a pointer to the part of the code that the other person is talking about, seeing his edits in real-time and commenting on them immediately etc.

This is why I believe in convergence of communication tools.

Friday, July 17, 2009

Nothing is forever

I have been writing about the scaffolding metaphor - but after some conversations with my colleagues I realized that I did not do a good job there. I did not conduct well the main idea: nothing is forever. No library and no program is forever and we should not shy off from writing libraries that are useful only at some phase of the application construction. It is still valuable if you can quickly get to the point of having a preview, a simplified prototype of the functionality you need even if eventually you'll remove all of that prototype code from the project. This is very 'agile' and it is also 'worse is better (if you can have it early)' and those libraries, those tools that you use only temporarily, I call the scaffolding.

Now one would ask why write a library that is useful only temporarily? Why not include at once all the parts needed in the final product? There can be several reasons:

  • something that is only a part of the scaffolding in one project can stay in the final solution in another one
  • the author of the library might not yet fully understand the problem, and it can take years to reach the right abstraction
  • the distribution of possibly needed modification can be so uniform that there is no reason to concentrate on one of them more than on another one and it is virtually impossible to cover them all
  • and finally: nothing is forever - you never know where a project will go in the future and what you'll need to change

A good example (of the third point) are code samples from manuals - you copy them to quickly get something working - but later you modify it and sometimes you change every character from the original. Another one can be the scaffolding code in Rails - you can debate if it fits the first or the second or other points - but it was useful to a lot of people.

Every library can be used as scaffolding - so why not accept this and help the developers in using it that way?

Monday, July 13, 2009

CPAN and code reuse

CPAN is wonderful thing - it is a great achievement of the whole Perl community and the thing that makes it apart from other languages. But somehow it works mostly for libraries. And what about the other modes of code reuse? How about CPAN for code examples? One can say we don't need CPAN for code examples - everyone is using code examples all the time and they do that without any special infrastructure. But what I see around is that some of CPAN modules are actually code examples in disguise. Some of them would be better without any code in the package and only documentation and tests validating the documented techniques. Hmm maybe we don't really need a separate CPAN for that - maybe we can keep them at the existing one - and just mark them in a special way? And maybe we need some social acceptance for packaging such 'empty' libraries?

And what with applications?

Thursday, July 09, 2009

Code reuse styles

There is much buzz around PHP tools like Drupal and Joomla. They are ready-made solutions, but in fact there is also a lot of tweaking the templates and writing add-ons. It seems to work very well - the clients get something usable from the beginning - and later they can modify it to fit their purpose better. This is very agile. In contrast the reuse in Perl is mostly around libraries. This is much more flexible - you don't tweak a ready solution - you assemble it from the thousands of CPAN packages. But it is also less agile.

I wonder why it is like that. There are some technical differences that lead developers in one or the other direction - but there is also a cultural/psychological cause. Doing some self-analysis I must admit that I am not really enthusiastic about contributing to a Perl CMS (like for example Bricolage) - in my view all the already-build solutions use obsolete libraries. I'd rather contribute to building the perfect library - then code an extension to Bricloage. I have the feeling that I am not alone in that crazy perfectionism around Perl hackers. PHP clearly wins here with more pragmatic approach.

Monday, July 06, 2009

Deprecated code analyzer for HTML::FormHandler

So the other day I've read Deprecated code analyzer for perl and I thought - this is a GREAT idea! And I stole it for HTML::FormHandler. Now with new releases our users will have a tool to find all the deprecated API calls in their code.

Saturday, July 04, 2009

Is using the Catalyst 'forward' method a cargo cult?

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:

$c->forward( 'some_action' )

and the standard Perl:

$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.

Wednesday, July 01, 2009

Functional versus Unit testing

Benefits of automated functional testing (was: Why unit testing is a waste of time) is a comprehensive list of arguments for functional and against unit testing. It is a bit long and maybe it is preaching to the choir as I have never seen unit testing in CPAN modules, but it might be handy if you needed to convince your co-workers.

Also discussed at Hacker News

Monday, June 29, 2009

InstantCRUD and FormHandler

The new version of Catalyst::Example::InstantCRUD is on CPAN. This version uses FormHandler, is a bit nicer visually (I hope) and has a proof of concept experimental REST interface.

Friday, June 26, 2009

Iron Man Blogging Challenge

First I need to confirm that I really like the idea. Perl needs more publicity. Some may say that quality is more important than quantity but there is no better way to improve your writing skills than practice. So yeah - it's a great idea. In general I think we need more competitions - but that is really for another post.

There is just one small nitpick - if the goal is: "to promote the language to the world outside the Perl echo-chamber" then the rule that the all the posts need to be about Perl is misguided. If we want to talk to "the world outside the Perl echo-chamber" then we need to talk about the subjects that they care about. I can understand why we need some focusing - and posts about what you had for dinner are not the best way to promote the language - but it will be much easier to engage the outside audience if we wrote about general programming subjects. Look at advertisings - they don't talk about the advertised brand all the time, the more subtle the link is, the more engaging they are.

Update: I should have added that this post was inspired by Gabor Szabo: http://szabgab.com/blog/2009/06/1245788806.html

Saturday, June 20, 2009

CatalystX::Comments - RFC

In Packaging cross cutting Catalyst features I promised to start developing a generic Catalyst comment sub-system. Now I have incorporated the code samples into another experiment - ravlog(a blog engine by Gerda), and I would like to ask you what you think about it before releasing it to CPAN.

Here is the current API - any comments are welcome. First you need to declare the controller using the new Moose style:

package RavLog::Controller::View;

use Moose;
BEGIN {
extends 'Catalyst::Controller';
with 'CatalystX::Comments::ControllerFormRole';
}

has model_name => ( is => 'ro', isa => 'Str', default => 'DB' );


The important part for us is of course with 'CatalystX::Comments::ControllerFormRole', the model_name attribute should hold the name of the model CatalystX::Comments will use to store the comments.
After the declarations you can use CatalystX::Comments to put the comment form on the stash:

sub view : Chained('base') PathPart('view') Args(0)
{
my ( $self, $c, $id ) = @_;

...

$self->stash_comment_form( $c, $self->article->id );
}

That's the whole controller part - what it handles is generating the form for display and saving the submitted comments. In the view templates you need to add:

[% comment_form.render %]

For displaying the comments themselves ravlog already had some template code and I did not change it.

Next is the model part. The library assumes that the DBIC model contains a Comment Result class like that one in ravlog. Of course it does not need to have all of the columns, and can also have other columns - but these are the columns that will be filled by the CatalystX::Comments form. Ideally that model part should be a Result base class - so that it can be subclassed and adapted to local circumstances (for example the belongs_to relationship to article needs to be changed), but for now I don't know how to do that.

Now some more details. Thanks to html_prefix all parameters sent from this form are prefixed with the form name - so this form can be added to pages with other forms, it will recognize it's parameters. The comments are saved only when the HTTP method used is POST, and after a successful comment creation the user is redirected to the same page (this seems a bit constraining - but see next paragraph - this is only meant to be temporary solution - I try to keep the API simple).

Of course I understand that this can never cover the needs of a comment sub-system in a mature social web site. I am thinking about it as more of scaffolding - code that let's you quickly develop a feature, see how it integrates with the rest of the user experience, formulate a more complete requirements list - and replace it part by part.

Friday, June 19, 2009

On Scaffoldings

Scaffolding is a wonderful metaphor. Was it invented in Ruby on Rails? I am not sure about that - but nevertheless it is RoR that popularized it and undoubtedly it was one of the engines of it's popularity. It is very "Agile" - like a programming sprint - it implementa a feature in the most simplistic way that can possibly work - but instead of a programming team doing the work the code for that feature is generated.


Scaffolding is a wonderful metaphor - and the brilliance of it shows in how it transcends the original area of code generation where it was introduced in Rails. It can be applicable to any reusable software component meant to be temporary support and to be replaced later. An example of that are code samples in manuals - you copy them just to have something working, but then you modify it and nearly completely replace the original code. And since nothing is universal and no library is ideal - it can be used as a generic measure of how well a library supports this iterative development style. And this brings me to the question - what are the qualities of a good scaffolding? One is that it needs to be gradually replaceable.

Saturday, June 13, 2009

Module Rank and Informed Choice

There is much talk recently about CPAN and how to improve it. I firmly believe that initiatives like CPAN Top 100 will eventually lead to some good, Page Rank like, measures of if not module quality then at least module utility. People vote with their legs - they use other modules in their own productions and we don't need any other voting mechanisms. After all Google does quite well with Page Rank, much better than the human edited Internet Directories (like the original Yahoo or Open Directory).

But it all still relies on people choosing the right modules for their own projects - I believe this is a good assumption, the popular "The Wisdom of Crowds" shows why - but still it can be improved by people doing more informed choice. This was my motivation behind my API design blog post. Design of APIs is tricky and subtle and there are things that you just don't spot until it is too late, or alternatively there are deficiencies that are invisible for the seasoned user because he is used to the quirks, he internalized them and he uses the API automatically without thinking.

Sunday, June 07, 2009

Form rendering and CSS

I tried a few times to learn CSS, but not really seriously, I guess, since I still know only the most basic things. So I am a bit lost now when I need a nice and universal CSS for FormHandler forms in Catalyst::Example::InstantCRUD. Or perhaps I should say HTML::FormHandler::Render::Simple forms instead of just FormHandler forms as theoretically the HTML rendering part is separated from the main FormHandler code, so that if this Simple renderer turns out really too simplistic I can also write a HTML::FormHandler::Render::NotSoSimple to get the right HTML.

I started reviewing the options and it seems that there are three general approaches to having aligned forms:

  • tables
  • divs
  • lists
  • no additional markup

Plus some appriopriate CSS. I think the 'no additional markup' option could never accommodate for error messages, so I leave this one out. This leaves me with three options. I am sure that not everything can be done in each one - but I think we can add a 'style' parameter to the renderer to make it configurable.

Next thing is the recently popular CSS grids - anyone using them? They seem like a nice option for someone needing some universal defaults and the SenCSS example even contains a form styled in two ways (vertical and horizontal alignments) - but would it work with error messages? And the Blueprint one even shows a div with the error class - but how that is to be inserted into the form? I guess each one requires it's own way of HTML structuring.

Friday, June 05, 2009

Packaging cross cutting Catalyst features

A recurring subject on the Catalyst mailing list is about packaging functionality that is not entirely comprised in one of the Model View or Controller parts of the framework (and is not a full application on itself). This is hard and still there aren't many CPAN libraries that do that. One solution to that is writing "helpers" that generate the integration code into your application files. As with all code generation this solution creates many new problems - and ideally we would like to avoid it.


To try out what can be done about that I decided to write a generic comment system for Catalyst (+DBIC, TT and FormHandler). I've already posted the the first code samples to the Catalyzed wiki. The form code and the controller stuff seems closed - it does not need to refer to things from outside but in the DBIC declarations the belongs_to relation of the comments needs to relate to an external table and it's primary key (I assume that comments belong to an article/post/something). How that can be made configurable?

Wednesday, May 27, 2009

API design

In the past programming was more about writing from scratch - now code reuse take more and more prominent place in development. In the Perl sphere this is clearly visible in the growth of CPAN, but what is also visible is that this quantity is not everything.


There are many aspects of quality - here I would like to focus on the APIs. The problem with APIs is that their shortcomings are much more subtle than let's say bugs. Many of them are just inconveniences - things that you stop to see once you get accustomed to them, and all of them require some experience or theory just to understand that it can be done better. Arguments about APIs can become dangerously subjective. We are in dire need of API theories, objective measures to evaluate them - and here I try to compile a list of on line resources on that subject.


How To Design A Good API and Why it Matters (from Google Tech Talks)"

- see also an ACM paper with the same title and a nice abstract in the form of bullet points.

My take-away points (the most non-obvious or potentially useful in arguments):

  • write examples from the very start and build the API from use cases
  • good names are important - API is language
  • "when in doubt leave it out"
  • implementation somtimes leaks out to the API
  • for inheritance document self-use pattern (how methods use one another)
  • "don't make the client do what the module can do"
  • you can't please everyone so aim to displease everyone equally.


The Little Manual of API Design



Characteristics of good design:

  • Easy to learn and memorize
  • Leads to readable code
  • Hard to misuse
  • Easy to extend
  • Complete


Again stress on writing use cases and code using the API during the design process - and one surprising but seemingly valid point: "prefer specific names to general names".

API Design Matters



This one is mostly arguing why API design matters - but there are a few points worth noting:

  • always design the API from the perspective of the caller (rather obvious)
  • again start with use cases
  • there used to be much more programming from scratch - now we use more libraries and designing API becomes more and more important, but this is not yet reflected in university curricula (this is exactly the intuition that led me to this blog post)


Designing and Evaluating Reusable Components



The needs of the programmer changes as her project proceeds and it is important to account for that fact when writing the API. In particular more advanced projects usually need more granular APIs while at the beginning they are only an obstacle.

API design guidelines

- my question on Reddit.

Update: Read Curing Python's Neglect - it's not really about Python in my opinion - and it is about how people get used to bad APIs and stop noticing the ugliness of them.

Tuesday, April 21, 2009

Constraints and Types (in Moose)

Moose::Manual::Type says that Moose types are mostly just constraints with a name. When designing new constraints API for Formhandler we decided to use Moose types - but with some playing around it also exposed the fact that not all constraints are types. I am not an expert in Type Theory - but I checked wikipedia and it seems to support my intuition that type is an attribute of data value - while when validating we often need to check not just the value - but also it's relation to the whole system.


If that is not yet clear - let's take the example of checking the age of a application user (from his provided birth date). The question if a user with a given birth date can legally participate in a web forum cannot be answered without knowing what is current date - it is not a property of the birth date alone - but of it's relation to the current date. I think that using the term 'type' for this kind of constraints can be confusing, even though technically in Moose this is possible to declare this.

Saturday, April 04, 2009

Roles for FormHandler

This is a more concrete version of the question from my last post. This is a form defined in FormHandler:

package BookDB::Form::BookWithOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'Author' );

has_field 'title' => ( type => 'Text', required => 1 );
has_field 'publisher' => ( type => 'Text', required => 1 );
has_field 'owner' => ( type => 'BookDB::Form::BookOwner' );

This should be quite self-explanatory for anyone working with forms (and DBIC).
Now - I think Model::DBIC should be just a role here, but that is just a side track. The real question is about the fields here - ideally I would want that doing:

with 'HTML::FormHandler::Model::DBIC';

(this is after it is made a Role) - would add appriopriate methods to the fields - an example here could be validation of fields that need a unique value. I would like to not need to change the definitions above to:

has_field 'title' => ( type => 'Text::DBIC', required => 1 );

Fields like 'owner' above which are Compound fields and work mostly like a form itself and need to load the options for SELECT boxes from the model, set the values of fields from a database row etc:

package BookDB::Form::BookOwner;

use HTML::FormHandler::Moose;
extends 'HTML::FormHandler::Field::Compound';
with 'HTML::FormHandler::Model::DBIC';

has '+item_class' => ( default => 'User' );

has_field 'user_name';
has_field 'fav_cat' => ( label => 'Favorite Book Category' );
has_field 'fav_book' => ( label => 'Favorite Book' );
has_field 'occupation';
has_field 'country' => ( type => 'Select' );


Ideally this definition should not need 'HTML::FormHandler::Model::DBIC' - applying this role to the root form should be enough.