Guillermo Roditi - Perl stuff

I am officially sick of Vox. I think this account has at  best a couple of weeks left to live while I switch over to Blogger or Typepad or whatever is free and ad-free these days. I would use my use.perl journal, but I think SLASH is a piece of shit, I don't like their discussion system, and I constantly lose my fucking session cookie. Can anyone recommend a decent place to blog? I just want the basics. Hell I may just set up my own software because at this point ANYTHING is better than looking at the ads all over this website. I like that Vox looks pretty, but why is it that it is so fucking slow? Between the ads, all the eye candy images, the javascript and this stupid richtext editor that continually lags two keystokes behind my hands I have just about had it.

Please someone, recommend to me a place where I can blog, and it's relatively simple to include code formatted such that it doesn't look like complete trash.

Also: why doesn't Vox have a fucking preview button? or a "save draft" button? I don't care about fancy WYSIWYG formatting options, but I would like to be able to save drafts to work on them over several sessions. And even more, I would like to preview my fucking post.

I HATE THE INTERNET. WHY DOESN"T ANYONE GET IT RIGHT?
Make it usable, THEN make it pretty. It's just not THAT hard.

I just subscribed to The Perl Review, and I already hate it. I know it is probably understaffed and limited in funds, but why is the website so fucking awful? This is a publication representative of the Perl community and it is just god-awful. Their scheme where subscribers all have the username of "subscriber" and share a global password which changes every month is horrible, first of all. How hard is it to implement a simple application with a user system these days? We have Catalyst and Jifty, we have a wealth of database tools, we have modules capable of managing all types of authentication schemes and stores in the blink of an eye. Why on earth is this website using HTTP authentication and passing the username and password via plaintext in the URL? (http://subscriber:secret@www.theperlreview.com) It isn't so much that I am bothered by not having my own personal account, where I could easily set preferences, store and edit my mailing and billing information and even keep track of my favorite articles. No, that I can live without. But why do I need to change the password stored in my browser's password manager every time a new issue comes out? That, sucks.

I have not yet read the magazine, but here's to hoping the content is a little better than the website.

So, while writing the Reflector's code generator I thought to myself "Why don't I just generate the documentation stubs too and save my self the 10 minutes or so per file of writing the POD skeleton?" The thought Obviously morphed into "How can I automatically generate POD skeletons for my projects?", which led to the easy answer "With a couple of templates and Class::MOP". Well, about 20 or so hours of coding (non-consecutive) later, I came up with MooseX::AutoDoc. It's not yet on the CPAN, and it may be a while before it gets there, but it is available in the moose svn repo (http://code2.0beta.co.uk/moose/svn/MooseX-AutoDoc/).

What AutoDoc does is simply use Class::MOP and Moose to introspect classes or roles and generate as much information as possible for you. Unfortunately there is a lot it can't do, but I like to think it is fairly clever. For example:

It will generate stubs for your attributes and, if applicable, make sub-entries for related (e.g. predicate, accessor, clearer) methods. It will read type constraints and refer to them by name and, if possible, link to where they are defined, be it Moose::Util::TypeConstraints or a MooseX::Types-based library. It will tell you if the attribute is read-only or read-write and omit information of initialization arguments and accessors if they are share a name with the attribute.  For the most part, the auto-generated description is not only accurate, but complete. You can even attach additional documentation to attributes by using the documentation attribute for attributes in your 'has' statements.

Generated, constructor and accessor methods are ignored. Entries for the methods meta and new are automatically generated along with their descriptions and a clause about setting attributes through the constructor.

Role-application is short circuited to prevent AutoDoc from generating entries for attributes or methods generating from consumed roles, although that may change in the future. A list of roles consumed is generated and linked to, so that you don't have to duplicate documentation. Composed roles are decomposed to their original names to make linking accurate.

Most of this process is customizable. The core of the module simply creates a small data structure representing the package to document and hands it to a View class and the view class does all the text generating. So far, I have only created a Template Toolkit-based view, but it is relatively easy to create one. The default TT View has a hash of BLOCKs that you can edit to provide customized output. At the moment you can only create documentation for an entire package at a time, but that will change too.

Still to be done
Well, a whole lot. I'd like a command line application to automatically generate the POD and append it to a file; I'd also like to break the generator down so that you could create DOCS for only certain items; I'd like to somehow keep the data structure stored serialized and stored somewhere in the POD so that the command line application could detect  when there are changes to the package and  only add docs for new methods or attributes. Unfortunately POD has some limitations, so I think I may need to develop an intermediate storage form in something like YAML and split the generator into two parts. one to make the intermediate file and write it (which could then be edited) and one to generate POD from this intermediate file.

Splitting the process into these two parts would allow for greater flexibility because I could simply digest the individual sections of the intermediate file and store them as matadata so that I know not to overwrite user-edited entries upon regeneration. Additionally, it would be a breeze to thaw the file out, add some elements to it, reserialize it, and then regenerate the POD. I could then handle the actual replacing of the POD on the file by using Pod::POM.

All in all, I am pretty excited about the possibilities there is for taking grunt work out of writing documentation and I am hoping someone else will also find this tool useful.

So, when I get to talk to the other Reaction developers I will undoubtedly have to answer the question of why the Reflector code generation doesn't just introspect the created classes to build the data representations of the IM classes. It would have been simpler to simply create a Role that introspected the classes created to generate code instead of refactoring the existing reflector and having to deal with the possibility of introducing new bugs and hard-to-debug code. The short answer is "closures".

I have tried to stay away from excessive use of closures during the Reflection operation because we simply can't easily get the code back. Moose and Class::MOP allow us to introspect classes very deeply and accurately, but after I do something like this:

has timestamp => (is => 'rw', required => 1, default => sub { DateTime->new } );

There is no simple way for me to do something like:

 my $default = __PACKAGE__->meta->find_attribute_by_name('timestamp')->default->as_source;

In other words, I can't get the source code back, which is actually something I need for certain parts of the equation, like some methods wrapped at runtime, a couple of methods added, and the

default

and

valid_values

attributes. Originally this would have been impossible to do because of the way these closures were formed, but when I wrote the new Reflector I made sure that these closures were all free of outside lexicals so, when the time came, I could always write them out as source code and return them as a string to be evaled.

Currently, the Loader--for lack of a better term--is returning all code meant to be evaled as a scalar ref to a string of source code. I lifted this little piece of behavior from SQL::Abstract's behavior because I figured it would make it easy to differentiate between code and strings.

An additional thought
It is worth considering here  how custom metaclasses could cut a lot of this code out. Special  attribute metaclasses for columns and different types of relations could come with predefined behaviors  to fetch defaults, valid_values or builder methods. Another thing to consider would be using custom method metaclasses for certain methods which, as an additional attribute keep a string representation of their source. Such a thing wouldn't be particularly hard to implement using the new trait support in Moose.

The Reflector is a fine piece of software and an integral part of Reaction, but it has one glaring and horrible shortcoming: m2m relationship building on the interface model requires that your link relation and your m2m relationship have specially formated names, which I hate. Originally this was adopted as a GoodEnough approach because, since M2M relationships are not actually relationships, but merely a set of shortcut methods it was impossible to reflect them by introspecting DBIC. Along with this, it is obvious that this is another part of Reaction that is not up to par with the rest of the codebase. Adding Moose attributes to DBIC classes to aid the reflector works, but is incredibly hacky and generally looks awful. It also doesn't work as well as it could.

My proposed solution
I am planning on creating a special class, R::IM::DBIx::Class that will export a small set of sugary keywords that will thinly wrap the traditional DBIC methods (as well as take care of some of the quirks of making Reaction-friendly DBIC classes) and generate the columns, relationships, and reaction attributes without having to duplicate calls. I started playing with a prototype earlier today and I think I found it to be not only feasible, but generally simple. All it really takes is a wrapper that both declares a column and an IM attribute. Additionally, it will allow us to improve the CRUD functionality of reaction by using more specific TCs by simply looking at column attributes like unsigned => 1, size => 7, etc etc. It is not going to be magic, but it will make it a whole lot easier to declare Reaction-friendly DBIx::Class Schemas and  Objects.

To accomplish this I will need to put a couple of things in place:

1. The sugar-exporting file
2. A special metaclass for DBIC sources
3. Special attribute metaclasses for DBIC columns and relationships
4. Minor extensions to the TCs to allow for finer grained restriction which we can introspect. so that we can dynamically translate some of these into JavaScript when the time comes. Initially I am going o stick with very simple things like unsiged => 1 and max lengths for strings.
5. A modified version of the reflector which can exploit this data to create more accurate representations of the DB model. I think a role would be good for this extension

It is no secret that over the last 6 months a large amount of the work in reaction has gone toward improvements in the
"V" part of MVC. Other than my initial implementation of the Reflector there really has not been a lot of work on the M side of the Acronym. This weekend that will change.

Today I took the plunge and created a module with an API identical to that of the Reflector that, instead of using Class::MOP to create the classes at runtime, creates abstract data structures that represent the class. This allows me to do three important things.

1. Translate the data structures into source code to create a Loader-style InterfaceModel generation tool.
2. Use the same back end (the data structure generator) for both runtime class creation via MOP and code generation
3. The data structures will be much, much easier to write simple unit tests for. The Reflector has very good test coverage right now, but the problem is that the tests are not narrow enough and when there is a failure in one of the tests it still takes too much digging to pinpoint the cause.

I am really excited about this because right now the reflector is a major guilty party when it comes to the long startup times. While the improvement won't be drastic, it should be noticeable. Also nice will be the demystification of the Reflector. because it is very poorly documented and very very very flexible the reflector still baffles most of the people who have been brave enough to try out Reaction. It really takes very little code to do a whole lot with the reflector, but because it generates classes and not code it is hard to just test small changes and see what changed. Finally, it will make working with very complex InterfaceModels much easier.

Right now I have the data structure generator completed. It was very simple because I have been planning on doing this since i first wrote the reflector, so I made sure wrote it from the get-go in such a manner that would allow me to do this later on. I am hoping tomorrow i will finish 2 Roles for the Reflector, one which will take the data structure and create the classes at runtime using Class::MOP and another one which writes source code to files in directories. I don't know how I am going to do this last one, I am tempted to use TT so I can achieve some kind of decent looking files but I don't know if the work is worth it, after all, I could just use PerlTidy. The most exciting part of this all, however, is that because of how the reflector works it will be really easy to autogenerate stubs of documentation for all the classes created by the reflector. These stubs will actually be very useful and should not need a lot of editing to go from stub to good-quality documentation. Stay tuned for more on this!

Earlier today I uploaded DBIX::Class::EncodedColumn to PAUSE. As I mentioned before, EncodedColumn is the  scheduled replacement for DigestColumns.  DigestColumns was a fine module and it did what  it was supposed to do, but it had unnecessary limitations and a slightly awkward API, in my opinion. Not only is the amount of code necessary to keep encoded columns smaller with EncodedColumn, it also behaves a lot more sanely and supports new features, like a wider array of ciphers and salted password support. Some of this stuff is already on the POD, but I'll repeat it here for those of you who don't care to look at it.

Ways in which EncodedColumn is superior:

• Supports multiple encoded columns per table, and each may have a different encoding cipher and format.
• Support for additional encoding engines, including built-in support for Crypt::Eksblowfish::Bcrypt
• Support for automatically generating and applying random salts to Digest-encoded passwords
• Inlined check methods for speed, although the difference is likely negligible
  
• Values are encoded at set and new time, instead of insert and update time. This means you are guaranteed to receive an encoded value every time you read the value.
• Copying a row with the copy method does the right thing.
• Shorter, more expressive, syntax to accomplish the same things
• Better default values.
• No name space pollution. Only one method is added.
• Better tests and documentation.

How Digest salts are handled
After doing some research onto how salts are handled by most people I realized that most people either store the salt on a different column, or they use a static salt. I didn't want to force users to add an additional column and I really didn't like the idea of using static salts, so I decided the salt and the digest should ideally be stored as a single string. MST suggested to simply append the salt to the digest because, after all, digests have fixed lengths so a little toying with substr in the check method could ensure compatibility not only with random salts, but also with salts of varying lengths. This is a shinning example of DTRT and I am really pleased with the results. For your convenience, the Digest wrapper comes with a table of common digest algorithms and their lengths in various formats.

Why you should use this module
You should use EncodedColumns if you keep a table with one or more values encoded using Digest or a similar module. It allows you to quickly add automatic encoding support to your application as well as the automatic creating of a digest check method, useful for authentication. This works in Catalyst applications and any application where you authenticate by comparing a plain text string to a digest value.  If you have any questions, or cipher requests please feel free to contact me.

EncodedColumns on CPAN - http://search.cpan.org/~groditi/DBIx-Class-EncodedColumn-0.00001/
EncodedColumns on SVN - http://dev.catalystframework.org/repos/bast/trunk/DBIx-Class-EncodedColumn/
EncodedColumns on SVNWeb - http://dev.catalystframework.org/svnweb/bast/browse/trunk/DBIx-Class-EncodedColumn/

Reaction
There's been a lot of good work done here over the last couple of weeks. Lately, development has slowed down as mst works on the Vienna WoC project and while I focus on some $work stuff. However, edenc and wreis have done a great job at keeping development going with a few nice contributions and some important fixes.

POE::Component::DirWatch & POE::Component::DirWatch::Object
The latter is being deprecated in favor of the earlier. I rewrote a large part of it and I am really pleased with the results. The only hold back so far is a lack of testers. The smoke tests from CPAN Testers are still showing failures, but the output is nearly useless so I have been unable to reproduce the failing scenarios. The most notable improvement here is the ability to use AIO automatically if it is present on the system. A full post on this will be coming up.

DBIx::Class::DigestColumns & DBIx::Class::EncodedColumn
Once again, the latter replacing the first one. I inherited DigestColumns and I finally got fed up with it's crappy API and  limited functionality. The new module has better tests, better documentations, should be faster, and is a lot more flexible. A full post on this will be coming up.

DBIx::Class
I am strongly considering reviving my work on what was the "FilterColumns" branch. The idea would be to expose hooks to perform data transformations, much like InflateColumns does, but without requiring inflation to an object. This is still very much in the planning stages.

MooseX::Object::Pluggable
I released a new version that clears up any compatibility issues with newer versions of Moose and deprecated the extension system.

Finance Modules
There's a lot of work to be done here still, but this will probably be one of my bigger pieces of work on the CPAN. The NFS AccountBalance is on CPAN and stable, but there is still a lot of work left to be done with regards to providing a storage medium and better auxiliary tools. DST FAN Mail needs some testing, but the code is essentially complete for the file downloader and parsers. Currently, the Schema needs to be designed and a couple of glue scripts need to be created to tie in the whole package. I am actually very excited about how this is coming along.

Cantella::DJabberD
Coming soon to a repository near you, this is Cantella's customized DJabberd server driven by a DBIx::Class Model and featuring a Reaction-based front end.

A couple of weeks ago I took it upon myself to restructure the reaction ViewPorts. I started by changing names to kill off some of the bad practices we started with. ObjectView became Object; ActionForm became Action; DisplayField became Field; Field became Field::Mutable; and GridView became Collection::Grid.

In addition to the namespace changes I analyzed what each item represented, what was duplicated logic and places where we could use inheritance and roles to our advantage. Over the course of a few weeks and after a lot of discussion in #reaction I came up with what is now the new_viewports branch in the Reaction repository. I am currently pulling in the changes to the widgets from mst's recent overhaul of the Widget layer. Once that is done I will have to apply the same changes to the templates, and finally,  make some changes to make the mapping of templates to widgets more DWIMy. In other words, I will be causing myself a lot of big headaches so that our users can have things JustWork.

This refactoring and reorganizing is important because it ties up all the loose ends me and mst have added  since I began helping him with Reaction. I naively added the ObjectView and DisplayField namespaces with little thought to structure and our future needs. Part of it was that I didn't fully grok the idea behind ViewPorts and their place in the big layer cake that is Reaction (we didn't have Widgets or LayoutSets yet!). I really believe these changes are one of the final major changes to our UI layer, and though they may seem small, they represent the culmination of a lot of hard work and a lot of thought by the core Reaction developers / contributors / interested parties (notably: mst, castaway, edenc and myself)

I recently checked in changes to trunk which got rid of the jifty-like names for controllers (RootController and CRUDController). Instead I added Controller::Root, Controller::Collection, and Controller::Collection::CRUD. Controller::Collection is a new controller and superclass of CRUD which has only list and view actions. A sort of display-only controller. With the new changes to the CRUD controller this was really easy to accomplish. Additionally I added to private actions which allow you to easily tie in your custom actions to the model at hand with a minimal amount of code. This is the first step in the restructuring / refactoring of the so-far neglected Controller layer. I am pretty excited about it, it works marvelously so far and is simpler than the previous system.

On ViewPorts:
I spoke previously about the need to refactor ListView and how I came up with GridView. Lately I've been thinking that  the display requirements of collections would be better suited by 2 different viewports, Collection and Collection::Grid. It makes perfect sense and it would allow me to reuse the Action, Ordered and Pager roles accross both. Collection::Grid would just be a new name for GridView, with a possible new superclass and Collection would be a very simple viewport which would just inflate a collection into an array reference (just like GridView does) and render each object as an ObjectView, which also needs to be renamed to InterfaceModel::Object. To be honest, I'd like to really clean up that whole section of the tree and give everything appropriate names, but I'd also like to wait until mst finishes the Devel::Declare work so the merge can be simple. Back to viewports though, the idea here is that Collection would just be a very very basic version of GridView that could display a heterogeneous collection of objects, which GridView can not do. Ideally I'll be able to implement a caching scheme to avoid all the attribute -> field lookups (they are kind of expensive). This would mark the end of the work I have planned for ViewPorts for the time being and allow me to dig into improving FocusStack, Window and CRUDController. so we can stabilize namespaces and hopefully APIs. There is a lot of documentation and test writing to be done as well as assorted cleanups of code no longer being used.

What's next?
Hopefully I will be able to clean up some of my internal code and add it to the Reaction code base, particularly SoftDelete stuff as well as little things related to user authorization, authentication, etc. Also on my radar is importing the PageContent / ContentProvider mst wrote for the Shadowcat site to make using partially static content an easier task. This also ties into efficiency because an approach like this would allow us to transparently render certain dynamic pages and flush the result of the viewport to HTML. This, in essence, would allow us to enable caching all the way to the ViewPort level and perform cache management through callbacks from model actions. Like I said before, things are starting to shape up and I have a feeling we have something pretty great on our hands.