Fixing dependencies

Fixing other people code is not easy - but this is what we need to do if we want to clean our dependencies. In answer to John's invitation here is what I did.

I wanted to write some code using Catalyst::Authentication::Credential::OpenID unfortunately one of it's dependencies fails it's tests. Here I'll document how I analysed that and produced a patch, it's not yet approved by the author - but the tests pass.

Step first - checkout the ParanoidAged from it's repository (I had the repository address from an email to the Catalyst mailing list). If you don't have access to the repository of your failing dependency - then just download the source from CPAN unpack it and work from there. Then run the tests:

zby@zby:~/progs/pa$ perl -Ilib t/00-all.t
.
.
.
ok 27
5 second tarpit (tolerance 2)...
not ok 28
# Failed test at t/00-all.t line 180.
3 second tarpit (tolerance 4)...
ok 29
Killing child pid: 7669
# Looks like you failed 1 test of 29.

OK - the failing test is at line 180. Let's see what is there:

	# dying in a tarpit
	print "5 second tarpit (tolerance 2)...\n";
	$ua->timeout(2);
	$res = $ua->get("$HELPER_SERVER/1.5");
	ok(! $res->is_success);

view raw gistfile1.pl hosted with ❤ by GitHub

I changed that to:

	# dying in a tarpit
	print "5 second tarpit (tolerance 2)...\n";
	$ua->timeout(2);
	$res = $ua->get("$HELPER_SERVER/1.5");
	ok(! $res->is_success) or warn Dumper( $res ); use Data::Dumper;

view raw gistfile1.pl hosted with ❤ by GitHub

ran the tests and save my debug output:
zby@zby:~/progs/pa$ perl t/00-all.t 2>debug.
What I got there is:

	$VAR1 = bless(
	{
	'_protocol' => 'HTTP/1.0',
	'_content' => '[1/5]
	[2/5]
	',
	'_rc' => 200,
	'_headers' => bless(
	{
	'client-date' => 'Fri, 31 Jul 2009 08:56:17 GMT',
	'x-died' =>
	'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394.',
	'content-type' => 'text/plain',
	'client-response-num' => 1,
	'client-peer' => '127.66.74.70:9001',
	'client-aborted' => 'die'
	},
	'HTTP::Headers'
	),
	'_msg' => 'OK',
	'handlers' => {
	'response_data' => [
	{
	'callback' => sub { "DUMMY" }
	}
	]
	},
	'_request' => bless(
	{
	'_time_begin' => 1249030575,
	'_content' => '',
	'_uri' => bless(
	do { \( my $o = 'http://127.66.74.70:9001/1.5' ) },
	'URI::http'
	),
	'_headers' => bless(
	{ 'user-agent' => 'libwww-perl/5.825' },
	'HTTP::Headers'
	),
	'_method' => 'GET',
	'_uri_canonical' => $VAR1->{'_request'}{'_uri'}
	},
	'HTTP::Request'
	)
	},
	'HTTP::Response'
	);
	Killing child pid : 7731

view raw gistfile1.pl hosted with ❤ by GitHub

Now the info is there - but of course I did not spot it at once. I did a lot of more debugging and testing other hypothesis before I focused on: 'x-died' => 'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394.'. Yeah interesting - something died, but no error was reported. The line in question is:

die "read timeout" unless $self->can_read($timeout);

view raw gistfile1.pl hosted with ❤ by GitHub

To get some more evidence I changed that to:

confess( "read timeout" ) unless $self->can_read($timeout); use Carp 'confess';

view raw gistfile1.pl hosted with ❤ by GitHub

Now the debug output changes to:

	'x-died' => 'read timeout at lib/LWPx/Protocol/http_paranoid.pm line 394
	LWPx::Protocol::http_paranoid::SocketMethods::sysread(\'LWPx::Protocol::http_paranoid::Socket=GLOB(0x8a22d4c)\', \'\', 4096) called at /usr/local/share/perl/5.8.8/Net/HTTP/Methods.pm line 236
	Net::HTTP::Methods::my_read(\'LWPx::Protocol::http_paranoid::Socket=GLOB(0x8a22d4c)\', \'\', 4096) called at /usr/local/share/perl/5.8.8/Net/HTTP/Methods.pm line 541
	Net::HTTP::Methods::read_entity_body(\'LWPx::Protocol::http_paranoid::Socket=GLOB(0x8a22d4c)\', \'\', 4096) called at lib/LWPx/Protocol/http_paranoid.pm line 352
	LWPx::Protocol::http_paranoid::__ANON__ called at /usr/local/share/perl/5.8.8/LWP/Protocol.pm line 157
	eval {...} called at /usr/local/share/perl/5.8.8/LWP/Protocol.pm line 99
	LWP::Protocol::collect(\'LWPx::Protocol::http_paranoid=HASH(0x8a227dc)\', \'undef\', \'HTTP::Response=HASH(0x8a2fdc0)\', \'CODE(0x8a324b4)\') called at lib/LWPx/Protocol/http_paranoid.pm line 358
	LWPx::Protocol::http_paranoid::request(\'LWPx::Protocol::http_paranoid=HASH(0x8a227dc)\', \'HTTP::Request=HASH(0x8a22548)\', \'undef\', \'undef\', \'undef\', 2) called at lib/LWPx/ParanoidAgent.pm line 314
	eval {...} called at lib/LWPx/ParanoidAgent.pm line 313
	LWPx::ParanoidAgent::send_request(\'LWPx::ParanoidAgent=HASH(0x8645148)\', \'HTTP::Request=HASH(0x8a22548)\', \'undef\', \'undef\') called at /usr/local/share/perl/5.8.8/LWP/UserAgent.pm line 255
	LWP::UserAgent::simple_request(\'LWPx::ParanoidAgent=HASH(0x8645148)\', \'HTTP::Request=HASH(0x8a22548)\', \'undef\', \'undef\') called at /usr/local/share/perl/5.8.8/LWP/UserAgent.pm line 263
	LWP::UserAgent::request(\'LWPx::ParanoidAgent=HASH(0x8645148)\', \'HTTP::Request=HASH(0x8a22548)\', \'undef\', \'undef\', \'undef\') called at lib/LWPx/ParanoidAgent.pm line 219
	LWPx::ParanoidAgent::request(\'LWPx::ParanoidAgent=HASH(0x8645148)\', \'HTTP::Request=HASH(0x8a22548)\') called at /usr/local/share/perl/5.8.8/LWP/UserAgent.pm line 391
	LWP::UserAgent::get(\'LWPx::ParanoidAgent=HASH(0x8645148)\', \'http://127.66.74.70:9001/1.5\') called at t/00-all.t line 179',

view raw gistfile1.pl hosted with ❤ by GitHub

And again - it is there - but to spot it I had to play with the debugger first. I don't remember what exactly I did - but finally I sow it - there are two evals in this stack trace, the first one is in LWP::Protocol and the second in lib/LWPx/ParanoidAgent.pm line 313. I checked that second eval - and yeah - it tried to retrieve the error already catched by the first one. So when there was a timeout the agent was aborting the retrieval - but the error was being cleared and not reported. I wrote following patch:

	--- lib/LWPx/ParanoidAgent.pm (revision 18)
	+++ lib/LWPx/ParanoidAgent.pm (working copy)
	@@ -314,11 +314,12 @@
	$response = $protocol->request($request, $proxy,
	$arg, $size, $timeout);
	};
	- if ($@) {
	- $@ =~ s/ at .* line \d+.*//s; # remove file/line number
	+ my $error = $@ || $response->header( 'x-died' );
	+ if ($error) {
	+ $error =~ s/ at .* line \d+.*//s; # remove file/line number
	$response = _new_response($request,
	&HTTP::Status::RC_INTERNAL_SERVER_ERROR,
	- $@);
	+ $error);
	}
	}
	else {

view raw gistfile1.diff hosted with ❤ by GitHub

and now all the tests pass. Finally I uploaded the patch to RT.

On dependencies

Ovid writes about Mojolicious::Lite:

The fact that it has no dependencies makes it even more compelling.It's the sort of thing one could more easily attract new developers with. Can you imagine rewriting my old CGI course with something like this? Or maybe a way to introduce new users to Perl? There are many excellent Perl frameworks out there which I would, nonetheless, hesitate to start new users on. Just trying to set up CPAN can be difficult for them; working through even one test failure is often too much. For people who want to jump in and learn "Web programming", though, this might work.

On the other hand Jay advices to stop worrying about dependencies The lie of independence - or - how I learned to stop worrying and love the dependency chain. There is truth in both statements. I have the impression that the problem of CPAN dependencies have improved much recently. The visibility of installation problems directly from the cpan search engine (cpan testers matrix, dependencies review) is a great step ahead. But still installing Catalyst and DBIC and TT is a big hurdle - and I can understand why Ovid is not enthusiastic about teaching Perl novices how to do that.

For me Mojo sounds like iPod - slick and beautiful and really useful - but you are not supposed to change the battery. It is a nicely integrated product - but somehow closed. The author apparently tries to make it perfect, tries to polish every detail - what perhaps would not be possible if he had to compromise with the visions of the authors of the other packages. I can feel his frustration here. But rejecting all dependencies and rewriting everything in his own way sounds a bit extreme and not welcoming.

The big question here is how much can we trust fellow CPAN authors to make the right decisions, can we rely on their work? Can someone, who put's a lot of attention for the details of his API, rely on modules with imperfect APIs, perhaps spoiling his own work and not become insane? The story of the phalanx project shows that there are no easy answers to these questions, but I believe we can do better than rewriting everything from scratch each time.

Google Wave

One, sometimes feverishly defended, rule of our technical irc channels is about not pasting code into the conversation. To show your code you are supposed to use some web based paste application and then copy the link into the channel. The need to use two separate tools, irc client and web browser, for one task, code review, is just a small annoyance - but eliminating those small annoyances is the only way towards really usable software. Many people have internalized the procedure so much, that they don't think about it any more and I am sure there will be some of them arguing that this is not annoying at all. But there are additional steps to do: choose the tool, paste the link, switch the windows and all this is a burden to the brain, even if it is processed in the background - it still consumes some brain resources.

And now imagine that you have it in one tool - not only those additional steps go away - but it also opens the way for new features like seeing a pointer to the part of the code that the other person is talking about, seeing his edits in real-time and commenting on them immediately etc.

This is why I believe in convergence of communication tools.

Nothing is forever

I have been writing about the scaffolding metaphor - but after some conversations with my colleagues I realized that I did not do a good job there. I did not conduct well the main idea: nothing is forever. No library and no program is forever and we should not shy off from writing libraries that are useful only at some phase of the application construction. It is still valuable if you can quickly get to the point of having a preview, a simplified prototype of the functionality you need even if eventually you'll remove all of that prototype code from the project. This is very 'agile' and it is also 'worse is better (if you can have it early)' and those libraries, those tools that you use only temporarily, I call the scaffolding.

Now one would ask why write a library that is useful only temporarily? Why not include at once all the parts needed in the final product? There can be several reasons:

• something that is only a part of the scaffolding in one project can stay in the final solution in another one
  
• the author of the library might not yet fully understand the problem, and it can take years to reach the right abstraction
  
• the distribution of possibly needed modification can be so uniform that there is no reason to concentrate on one of them more than on another one and it is virtually impossible to cover them all
  
• and finally: nothing is forever - you never know where a project will go in the future and what you'll need to change
  

A good example (of the third point) are code samples from manuals - you copy them to quickly get something working - but later you modify it and sometimes you change every character from the original. Another one can be the scaffolding code in Rails - you can debate if it fits the first or the second or other points - but it was useful to a lot of people.

Every library can be used as scaffolding - so why not accept this and help the developers in using it that way?

CPAN and code reuse

CPAN is wonderful thing - it is a great achievement of the whole Perl community and the thing that makes it apart from other languages. But somehow it works mostly for libraries. And what about the other modes of code reuse? How about CPAN for code examples? One can say we don't need CPAN for code examples - everyone is using code examples all the time and they do that without any special infrastructure. But what I see around is that some of CPAN modules are actually code examples in disguise. Some of them would be better without any code in the package and only documentation and tests validating the documented techniques. Hmm maybe we don't really need a separate CPAN for that - maybe we can keep them at the existing one - and just mark them in a special way? And maybe we need some social acceptance for packaging such 'empty' libraries?

And what with applications?

Code reuse styles

There is much buzz around PHP tools like Drupal and Joomla. They are ready-made solutions, but in fact there is also a lot of tweaking the templates and writing add-ons. It seems to work very well - the clients get something usable from the beginning - and later they can modify it to fit their purpose better. This is very agile. In contrast the reuse in Perl is mostly around libraries. This is much more flexible - you don't tweak a ready solution - you assemble it from the thousands of CPAN packages. But it is also less agile.

I wonder why it is like that. There are some technical differences that lead developers in one or the other direction - but there is also a cultural/psychological cause. Doing some self-analysis I must admit that I am not really enthusiastic about contributing to a Perl CMS (like for example Bricolage) - in my view all the already-build solutions use obsolete libraries. I'd rather contribute to building the perfect library - then code an extension to Bricloage. I have the feeling that I am not alone in that crazy perfectionism around Perl hackers. PHP clearly wins here with more pragmatic approach.

Deprecated code analyzer for HTML::FormHandler

So the other day I've read Deprecated code analyzer for perl and I thought - this is a GREAT idea! And I stole it for HTML::FormHandler. Now with new releases our users will have a tool to find all the deprecated API calls in their code.

Is using the Catalyst 'forward' method a cargo cult?

I think most Catalyst new-commers believe that to execute a Catalyst action you need to forward to it - this is the effect of all the examples in the manual doing that instead of using the standard Perl syntax of calling a method on a class. Of course there are some differences between those two - but I am sure that for most cases the standard Perl way of doing it is perfectly enough.

And when it is not enough? What are really the differences between:

$c->forward( 'some_action' )

and the standard Perl:

$self->some_action( $c );

As far as I know the only two differences are that the first call is called in an eval and that it is logged in the debug output (if it is switched on), maybe there are some other minor differences, but, I am sure, nothing dramatical. It can helpful/needed sometimes (and othertimes that eval can be a nuisance), but it is also easy to add. Of course this is the simplest case - 'forward' signature is more complex and it does amazing things for finding the action "by it's private name" - but for most of the cases you ain't gonna need it and you can use ye old Perl syntax to make the call and spare yourself learning how to pass parameters to the forward call.

Related reading: Premature Flexibilization Is The Root of Whatever Evil Is Left - a reddit discussion.

Functional versus Unit testing

Benefits of automated functional testing (was: Why unit testing is a waste of time) is a comprehensive list of arguments for functional and against unit testing. It is a bit long and maybe it is preaching to the choir as I have never seen unit testing in CPAN modules, but it might be handy if you needed to convince your co-workers.

Also discussed at Hacker News