Xor News

JNA is surely deserving of all the praise it has been getting. It’s being used on some pretty high profile projects like JRuby with great success. After having done JNI the hard way, the painful, tortuous, despicable, bang-head-on-keyboard-while-wondering-if-ReleaseStringUTFChars-applies-here and why-the-jvm-is-segfaulting-again way, well I have a deep appreciation for JNA.

Still who wants to go write a bunch of useless Java interfaces for stuff that already exists in built into the native library itself? Not me. So that’s where Jython and JRuby come in.

This week I needed Jython/Python access to some native modules, namely ssdeep for fuzzy hashing. There’s already a pretty nice solution for connecting pure python to native libraries — you can either use swig or pyrex. The pyrex piece for ssdeep has been mostly written here. Needed to add the fuzzy_hash_buf method into that mix but it was nice and easy. From inside pure python with ssdeepmodule.so (via pyrex) and libfuzzy.so (from ssdeep) (or .dylib for Mac or .dll on Windows) sitting there on your LD_LIBRARY_PATH, you get to do this coolness:

  from ssdeep import ssdeep
  import sys, os
  f = open("/bin/ls","rb")
  data = f.read()
  f.close()
  ss = ssdeep()
  fuzzy_hash = ss.fuzzy_hash_buf(data)

Pretty nice you have to admit. But from my pure java p2p data-driven workflow framework, I really wanted to do this from Jython to keep from having to start the interpreter up in a subprocess over and over. Pyrex extensions do not work in Jython. Makes perfect sense. M’kay. I could write a whole bunch of lame JNi code to hook libfuzzy.so in there. Or I could use JNA and write some non-dry interface in java and figure out all the details of the types and so forth. Or … I could just push all that code down into the python module that I’m going to call.

from com.sun.jna import NativeLibrary, Function, Memory
import sys,os

class ssdeep:
    fuzlib = None
    hash_func = None
    FUZZY_HASH_SIZE = 116

    def __init__(self):
        self.fuzlib = NativeLibrary.getInstance('fuzzy')
        self.hash_func = fuzlib.getFunction('fuzzy_hash_buf')
        pass

    def hash_data(self,data):
        ptr = Memory(self.FUZZY_HASH_SIZE)
        i = self.hash_func.invokeInt([data,len(data),ptr])
        return ptr.getString(0,False)

With a class and method conveniently named exactly the same as the pyrex module I can make it all flexible enough to work either way:

try:
   # try the pyrex extension module
   from ssdeep import ssdeep
except ImportError:
   try:
     # try the jna wrapper when in jython
     from ssdeepjna import ssdeep
    except ImportError:
      # write tmp files and just exec the dumb thing

Which is all pretty nice I think. Not too many worries about creating interfaces or other crazy things. Seems very efficient, a little extra packaging and we are good to go.

Well there was one problem in getting different values from the hashes when the data was binary. Turns out the JNA layer needs to be told how to convert data with -Djna.encoding=8859_1 on the JVM command line. Since I usually run with -Dfile.encoding=UTF-8 and in a UTF-8 locale, this made all the difference.

If that is inconvenient or you want to encode things differently only sometimes, the extra steps in the python layer would be something like

  from java.lang import String
  def wrap_hash_buf(self,data):
    javastr = String(data,"8859_1")
    jbytes = javastr.getBytes("8859_1")
    return hash_buf(jbytes)

The same type of thing would work just as well from JRuby.

Welcome to the sweet spot. Code on, baby!

Project of the day was to revive an old SVN repo project and get some changes made.
I no longer have all of the mod-svn+apache stuff working and didn’t want to work on that.
I did have an “svnadmin dump” of the old SVN repo though. The project name is emissary.

1. cd /tmp; svnadmin load emissary < emissary.dump
2. cd ~/projects; git svn clone file:///tmp/emissary -t tags -b branches -T trunk emissary
3. cd emissary; ant all test
4. Converted branches to tags, Removed some obsolete branches, using git branch -r -d
5. add “github” stanza to .git/config, and create the empty repo on github.com
6. git push github
7. git push –tags github

Double plus un-bad.

That’s what I wish I had done, anyway. Turns out git saved the bacon in more ways
than one. Initially I didn’t read the man page for “git-svn” far enough to get
the tags, trunk and branches. Started hacking on my shiny new git repo and did
a half-day’s work, committed, pushed to github, then realized I didn’t have any
of the historical tag info from svn. Bummer.

Here’s what I did at that point.

1. git svn dcommit
2. remove and recreate the git repo using the proper clone command
3. verify all work still in place — yes it is!
4. remove and recreat the github repo

Thanks, git.

The books in these pictures are free for the asking. If you know me or are local, just
let me know if you want any or all. Otherwise I plan to take them to The Book Thing of Baltimore in a couple of weeks. When the stack of books to get rid of
reaches up to the light switch, I guess it is time to take action! Caveat: most of the books have
my name scrawled across the top of the page edges, but are otherwise fairly pristine.

The big project our team has been working on for the last couple of months if finally public. It’s a fairly substantial rails app, about 40 models, lots of cool stuff. Most of the site is subscription based but you could checkout some of the info and look at the home page:

http://globalsports.net/

It’s a social networking site centered around sports videos and young athletes.

Rails controllers are just super nice. My MVC experience predates STRUTS by several years. So having done it by hand several times, and seen pretty much everything out there, I just find that Rails has a lot of nice features that make it super convenient to use: controller inheritance, before_filter, default routes, and much more.

But this week I got in a real jam with security.

I am inheriting a bunch of controllers from a plugin. All of the plugin controllers extend BaseController, which the plugin also supplies. In order to add or change some functions, I just create a controller by the same name, do what I want, extend BaseController and everything just works great.

Except when the controller in the plugin has a line like this:

before_filter :login_required, :except => [:edit, :destroy]

If in my version of the controller I need to change it so that the edit and destroy actions are also protected by the login_required filter, well that gets tricky real fast. If you think you can just add

before_filter :login_required

and you are covered, think again. If you want to try

before_filter :login_required, :only => [:edit, :destroy]

well that doesn’t work either. Some people have a proposed (with diff and tests no less) that an :exclude and :include tag. This seems sensible and would be a solution to the problem I had.

What I chose to do is to wrap the filter I wanted to apply more broadly in a new name.

class BaseController < ApplicationController
  before_filter :my_login_required

  protected

  def my_login_required
    login_required
  end

end

What that does is allow me to use my filter without regard to where it might collide with specifications from the inherited controllers. This makes it much easier to audit security in my application, is much less verbose, and pretty easy to understand.

In this case the filter that I wrapped was pretty lightweight. It didn't do any database lookups for normal cases. That's good because rails now doesn't know that login_required and my_login_required are really the same filter. So I'm paying the overhead of running it twice when I require my_login_required and the super class uses login_required. If it was a heavyweight method that would have caused a problem I would have to hack a little more to make sure that one of them could bail out early.

On a recent project we served up a flash/flex based client from a Rails app. The flash client needed to call back into the app to retrieve xml to populate some displays. Nice. Rails is good that way, especially if you have started from scaffolded resources which have all the format.xml responders already in them.

Unfortunately the dashes that rails ActiveSupport::CoreExtensions::Array::Conversions uses
(same goes for AS::CE::Hash::) takes the model attribute names and puts dashes into them
to create the xml tags.

So if you have a User model with user_name attribute and you just want to return
@user.to_xml, you are going to get

   <user>
      <user-name>value of name</user-name>
   </user>

Which looks pretty. But apparently the xml parser in flex can’t handle that. I couldn’t find any reference in the documentation on how to fix this, but when in doubt UTSL, baby. Which shows that there is a :dasherize option that can be provided.

  @user = User.find(params[:id])
  @user.to_xml(:dasherize => false)

Now the generated xml will have underscores instead of dashes. Flex developers everywhere should send DHH a nickel for that one. Ok, I don’t know who made the actual commit for it, but you’s should give David the nickel anyway.

In a Rails system with a pretty standard user model, views, and controller, I recently found that the users_controller was getting pretty large. There was a pretty significant amount of code that I wanted to add to deal with the users using a different set of views under some circumstances — and I didn’t want to add all of that to the users_controller.rb. What I wanted was to move these functions to a staff_controller.rb, use restful routing on it and just treat the user object differently.

This didn’t work out so well initially. Trying to use form_for in a view always reflects on the type of object you have and sends the results right back to the users_controller — not what I wanted. I tried a bunch of stuff to force the form_for to do what I wanted but no joy. What I wanted I thought was a way to alias the user model.

class Staff < User
end

Problem solved.

The User model had some nice named scopes for finders, for example:

  named_scope
          :recent,
          :conditions => ["created_at > ?", 2.weeks.ago]

And now I can do all of the following

  Staff.first
  Staff.recent
  Staff.find(params[:id])
  @staff = Staff.recent.find(:all,
              :conditions => ['name = ?',"jim"])
  @staff.last_login_at = Time.now
  @staff.save!

.. which since reflection now says they are staff objects, causes
all the view helpers that reflect on stuff to work properly and doesn’t
break any of the ActiveRecord stuff.

I may write up a better review of the new Canon A650 that I got for my birthday once I figure out a little more about how to use all of the fine features and get a little more experience. I’ve been too busy to take very many pictures recently, but I did experiment with the panorama stitching feature while on vacation in Waves, NC.

The image is large (9MB) so don’t click if you are on dial-up. Does dial-up still exist?

Upon putting the camera into panorama mode, the next step is to indicate whether you will shoot moving left or moving right. The camera then lets you take up to 26 pictures and keeps showing the last 1/3 of the previous picture on the large LCD screen as an aid to lining up the next shot.

Having taken all the pictures they should then be downloaded to a folder on the computer. This panorama comprises 13 images shot at 3 MP resolution (the camera does 12.1 MP) and then stitched using the Canon software on Mac OS X. The stitching process took about 4 seconds. I think it did a pretty reasonable job and I’m excited to try it out some more.

I got sucked into a trap. In a Rails migration it’s so easy to use an ActiveRecord::Base model object to update some stuff, and it looks nice and DRY. But it is extremely fragile. For example suppose your domain has a user and sponsor model. Each user has a sponsor. You start out this way:

   def self.up
      create table :users do |t|
         t.string :name
         t.string :password
         t.string :sponsor_name
         t.timestamps
      end
   end

Later you want to insist that every user in the system have a sponsor. You already have a bunch of test data that you are intellectually invested in, so you whip up a quick migration to make sure.

  def self.up
     User.find(:all, :conditions => 'sponsor_name IS NULL').each do |u|
         u.sponsor_name = "The Default"
         u.save!
      end
  end

Cool. Works good. You can go up and down. But some time later you realize the supreme error of your ways. You should have had a sponsor model and the user should have a sponsor_id, not a sponsor_name. Doh! What was I thinking. Time for another migration:

   def self.up
      add_column :users, :sponsor_id, :integer
      drop_column :users, :sponsor_name
   end

and the corresponding self.down method too, and the necessary belongs_to, has_one in the user and sponsor model classes. But now you can’t migrate the database up from bootstrap any more because your use of the the User model with u.sponsor_name above will not work. The user no longer has a sponsor_name.

The root of the issue is that the model objects are not versioned along with the database as it goes up and down. Since fixtures just use the same model objects that doesn’t help one bit.

The only reliable way we have come up with is to inject data using raw sql. So step two above
could change to

  def self.up
     execute 'update users set sponsor_name = "The Default" where sponsor_name IS NULL'
  end

That works, up and down the line, because the database sql IS in sync with the database at every step.

Have a better way to write less brittle migrations? Let me know in the comments.

The typical Ruby on Rails flash area is only drawn when there is a message to flash. I think something like this is fairly common (this example is haml, but that’s not the point) usually done in a partial, something like app/views/shared/_flash.html.haml:

- %w(info notice error).each do |type|
  - if flash[type.to_sym]
    .warning.flash_message{:class => "#{type.to_s}"}
      %span= h(flash[type.to_sym])
      %small
        %a{:href=>"#", :onclick=>"$$('div.flash_message')[0].hide();return false"} Close this message
 

The problem with this is that if you are Ajaxing a bunch of stuff you could use the flash area for user interaction and highlighting messages, but if it isn’t there, that could be a problem. So how about doing this instead so that the flash area is always drawn, just not displayed:

- %w(info notice error).each do |type|
  .warning.flash_message{ :class => "#{type}", :id => "flash_#{type}", :style => "display: #{flash[type.to_sym] ? 'block' : 'none'}" }
    %span
      - if flash[type.to_sym]
        = h(flash[type.to_sym])
    %small
      %a{:href=>"#", :onclick=>"$$('div.#{type.to_s}')[0].hide();return false"} Close this message

The flash closer is still in there so we don’t have to worry about the complexity of building that again every time, and the message is in a separate span element so it is easy to fill in. So then we can add a little bitty application helper (app/helpers/application_helper.rb)

  def flashnow(page,msg)
    page.select("#flash_notice span").first.replace("#{msg}")
    page.select("#flash_notice").first.show
  end

You might need to change the replace method for replaceHtml
depending on the version of prototype you are working with.
and call it from any view that is building an rjs response:

  flashnow(page,'Yeeha, cowboy, it worked!')