Chris’ Musings

Everytime I sit down to implement more of the paypal interface, I feel like I am heading into the wilderness with a rough outline of the territory. The latest venture was to add the ability for our ecommerce solution to process recurring payments via paypal. We allow this for credit cards already; but paypal is a different beast.

This time I sat down and after reading the documents quickly to get my bearings, I did the design and then started coding. What I left out this time was the exact protocol pieces. This time I was going to learn the protocol via experimentation.

I was not disappointed. While the paypal docs get their own protocol about 90% of the way there, they left off critical things and it will not work until you get it 100% right – nothing new there. The first is, what should the date format be. They say the date should be sent; but do not specify the format. Fortunately a quick google found several examples. For future reference the first billing date format is: yyyy-MM-ddT00:00:00Z (I just hard coded the time to midnight).

Second, the documentation is confused about what is required and what is not. For example, the docs say the address fields are required. However, I did not put them in for my first experiment and everything worked out just fine. I’m guessing the address fields are only needed if you put in an address.

Third the docs are just wrong! The response field with the status of the recurring profile is called STATUS in the docs. Hey paypal, it’s PROFILESTATUS. Easy enough to figure out, but come-on.

Now that I can create a basic recurring profile in paypal, I’m off to create a LOT of test cases. I’m particularly concerned about what happens when using direct payment (i.e., credit card gateway mode) and recurring payments. There is some vague language about this being different, so I need to find out what they mean. For starters, all paypal tx pass back a token id. However, the docs say not to use that with the use case. Instead, we need to pass all credit card info back to paypal. Geez, what a pain. Paypal already has the information linked to that token id, it would be simpler and more secure to just use it as the reference – just like we do for the non-credit card recurring payments. Guess I’ll have to create a 2nd similar code path to account for this….

Overall, I’m pleased with what I see as far as functionality provided by paypal. I just wish thier documentation was correct and clear on the specifics.

Actually, it wasn’t too bad once I worked through how I wanted to load test a GWT app. We have lots of tests for the GUI portion which obviously also test the RPC/server side; but we wanted to really stress test a few RPC paths.

Fortunately, we are not the first. We needed a mechanism to simulate RPC calls and with a little digging (actually, it was pretty hidden in a mess of less than helpful posts), we found a rough and ready class that we used – easier to borrow something than to build from scratch. Here is the project page:

http://code.google.com/p/gwt-syncproxy/

Once we adapted the code to our specific case and removed the app engine stuff, we were good to go with a quick framework that lets us write RPC test cases that look like:

RPCService service = 
       (RPCService ) SyncProxy.newProxyInstance(RPCService.class, 
                                                "http://mydomain/my-web-app/",
                                                "path");
String result = service.executeRemoteMethod(param1, param2);

We use the SyncProxy to instantiate the RPC service, than call a method on the service interface and wait for the result. That’s my definition of easy to use. The great thing is the test code is synchronized so we don’t have to worry about ansyc calls in the test code and can evaluate the result for correctness immediately.

Next up is to use Apache JMeter to simulate load. This again is relatively easy. By implementing the JavaSamplerClient of JMeter, we can add our test case as a java class that gets executed. There is sample code available – take a look at SleepTest included in the source download.

We did have to modify the source code to add the ability to include the CookieManager in the JavaSamplerClient. This required a bit more digging around and modifying of JMeter but is doable in an hour or so and changes 3-4 JMeter files in the java protocol package. If you want to do this, look at the HTTP protocol package. We did this because our code retrieves cookies via a standard HTTP call first and then calls our java test case which needs to access the cookies. Other than that, we used the stock framework and can easily load test our app.

In fact, with these two pieces (SyncProxy and JavaSamplerClient) we quickly write test cases for the critical parts of our app that we want to load test. That’s my task for next week – yeah!

The great part is after a day of testing different components, our performance and scalability is even greater than what we thought it was (ok we did make one minor change to remove some contention) and we have lots of room for growth with our current architecture.

The only part I couldn’t figure out was how to get JMeter to add our jar files to it’s internal classpath. We tried using the search_paths and user.classpath parameters, but nothing we did seemed to allow JMeter to pickup our jars from a different location. It’s just an annoyance to have to build the components and then move the jars to the ext directory of JMeter. Would be nice if we could get the search_paths parameter working. Oh well, maybe another day.

All in all, it took about a day to become familiar with these tools and write our test cases to cover our most frequently used code paths.

We have an external API that can be called from any programming language. In this day and age, who doesn’t. For a couple of the calls we have some parts that use encryption within the message. When we original designed the system, we did for internal consumption only. Now we are going back and opening those parts up to external users. Yeah! However, this is where the fun starts.

For the encryption, we use the Cipher: AES/ECB/PKCS5Padding

One of our clients uses php for their application and couldn’t figure out how to do the encryption/decryption with this particular cipher. So I got to explorer php encryption today. The popular mcrypt function seems to be what everyone uses with php. It’s built in, so no surprise there.

The difficult part is padding is not supported. There is some information out in the ether, but nothing seems to put it all together for us. So here is a quick primer.

This php program will take a message and a key and produce an encrypted string which matches our java implementation:

<?php
echo "\n";
 
function hex2bin($hexdata) {
  $bindata = "";
 
  for ($i = 0; $i < strlen($hexdata); $i += 2) {
    $bindata .= chr(hexdec(substr($hexdata, $i, 2)));
  }
 
  return $bindata;
}
function pkcs5_pad ($text, $blocksize){
    $pad = $blocksize - (strlen($text) % $blocksize);
    return $text . str_repeat(chr($pad), $pad);
}
 
function pkcs5_unpad($text){
    $pad = ord($text{strlen($text)-1});
    if ($pad > strlen($text)) return false;
    if (strspn($text, chr($pad), strlen($text) - $pad) != $pad) return false;
    return substr($text, 0, -1 * $pad);
}
function encrypt($str, $key) {
  $key = hex2bin($key);
  $td = mcrypt_module_open("rijndael-128", "", "ecb", "");
  mcrypt_generic_init($td, $key, "0000000000000000");
  $size = mcrypt_get_block_size('rijndael-128', 'ecb');
  $str = pkcs5_pad($str, $size);
  $encrypted = mcrypt_generic($td, $str);
 
  mcrypt_generic_deinit($td);
  mcrypt_module_close($td);
 
  return bin2hex($encrypted);
}
 
echo encrypt("4", "ERT345");
 
echo "\n";
?>

The key points are:
1) we have to pad the message ourselves in encryption function (or remove it from the decrypted string)
2) the IV needs to be 16 0s (or 32 if using 256bit encryption)
3) AES is refered to as rijndael in php
4) we have to translate the key using hex2bin

The above function matches what we do in java:

Key secretKey = new SecretKeySpec(fromHex(key),"AES");
Cipher aesCipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
aesCipher.init(Cipher.ENCRYPT_MODE, secretKey);
String encryptedMessage = toHex(aesCipher.doFinal(message.getBytes()));

While short and sweet, it actually is not the most straight forward thing to do in php. Dam padding . Anyway, hope this helps someone someday.

We want to let our customers host a form on their web pages (so they can control the form L&F using whatever mechanism they want) and “post” the form to our servers. We also want to provide robust validation and error handling for the form. We created a little gwt app that can be downloaded (cross site script).

Everything worked great until we tried to post the results to our server (actually that worked) and get a result back to show to the user (this didn’t work). Getting the result back turned out to be really hard. We first tried to use a gwt form and modified the form to use the window.name tricks for the returned value; but couldn’t get it to work in all of the browsers. It was probably a mistake on our part; but after a couple of hours we did not get it to work.

So we ended up crafting a specific URL with the form fields as parameters, redirecting the browser to that URL and then the server “redirects” the user to the next page on success or back to the form page on error (with parameter values appended that the gwt app can read and display error message for).

We’re not entirely happy with the solution as there is a lot going on in the address bar; but it works.

We will revisit this shortly and try to get the window.name trick working. In the meantime I would like to hear about others experiences.

Love hibernate / hate hibernate.

I love it because we have not had to code boilerplate SQL in.. well … years. It gives us all kinds of flexibility and quick application development.

There are times I hate it though (that might be too strong). We are in the process of converting a large app and upgrading the UI. At the same time, we are removing all of the technical debt we have built up over time. Today we found a bug and the error from hibernate was:

org.hibernate.type.SerializationException: could not deserialize

Not very helpful. Turns out we are not the only ones who have had this issue. Gotta love search!

The gist of everyone’s problem is incorrect mappings. Once we were able to turn on debuging, we narrowed it down to a class and then to the specific field. Once we corrected the mapping, all was good with the world again.

Now you might look at this and wonder why it was such a big deal. We’ll, what was an hour exercise, should have been 5 minutes at most (the actual fix was 10 seconds once identified). First, the issue was we were missing a mapping in the configuration file (the mapping was correct on the object). So I would have though it would have been caught during hibernate startup. The actually problem was a missing @Type declaration for an enum.

At worst, the error message should be better in Hibernate (we are working with 3.3.2.6) – hibernate knows which object/property it had issues with.

Anyway, moral of the story is to always check your field mappings and the config file to make sure it is correct also.

Update:
Found another gotcha (again my fault). Watch association mappings. My index property on a list mapping was being updated with null. I had the following:

   @OneToMany(mappedBy = "form")
   @IndexColumn(name="sequence", base=0)
   public List<FormField> getFields(){
      return fields;
   }

It should have been:

   @OneToMany
   @JoinColumn(name = "form")
   @IndexColumn(name="sequence", base=0)
   public List<FormField> getFields(){
      return fields;
   }