Chris’ Musings

Everytime I sit down to implement more of the paypal interface, I feel like I am heading into the wilderness with a rough outline of the territory. The latest venture was to add the ability for our ecommerce solution to process recurring payments via paypal. We allow this for credit cards already; but paypal is a different beast.

This time I sat down and after reading the documents quickly to get my bearings, I did the design and then started coding. What I left out this time was the exact protocol pieces. This time I was going to learn the protocol via experimentation.

I was not disappointed. While the paypal docs get their own protocol about 90% of the way there, they left off critical things and it will not work until you get it 100% right – nothing new there. The first is, what should the date format be. They say the date should be sent; but do not specify the format. Fortunately a quick google found several examples. For future reference the first billing date format is: yyyy-MM-ddT00:00:00Z (I just hard coded the time to midnight).

Second, the documentation is confused about what is required and what is not. For example, the docs say the address fields are required. However, I did not put them in for my first experiment and everything worked out just fine. I’m guessing the address fields are only needed if you put in an address.

Third the docs are just wrong! The response field with the status of the recurring profile is called STATUS in the docs. Hey paypal, it’s PROFILESTATUS. Easy enough to figure out, but come-on.

Now that I can create a basic recurring profile in paypal, I’m off to create a LOT of test cases. I’m particularly concerned about what happens when using direct payment (i.e., credit card gateway mode) and recurring payments. There is some vague language about this being different, so I need to find out what they mean. For starters, all paypal tx pass back a token id. However, the docs say not to use that with the use case. Instead, we need to pass all credit card info back to paypal. Geez, what a pain. Paypal already has the information linked to that token id, it would be simpler and more secure to just use it as the reference – just like we do for the non-credit card recurring payments. Guess I’ll have to create a 2nd similar code path to account for this….

Overall, I’m pleased with what I see as far as functionality provided by paypal. I just wish thier documentation was correct and clear on the specifics.

We’ve had they paypal direct pay API implemented for some time. We just ran our automated tests and they ALL failed. Turns out paypal changed their sandbox environment (again). If you use this api you need to make sure to always pass the cvv2 code; otherwise the payment request will fail.

It’s not a big deal that they want to always require the cvv2 code. The big deal is they just did it without letting anyone now, or make it easy to find out what a valid cvv2 code is that can be used. Turns out it is “000″. I only found that because there is a post on the site from one of the pp developers.

https://www.x.com/message/177346#177346

Sigh, I really don’t like commenting on other people’s apps (everyone has problems); but paypal is so difficult to work with I sometimes need to just vent….

Actually, it wasn’t too bad once I worked through how I wanted to load test a GWT app. We have lots of tests for the GUI portion which obviously also test the RPC/server side; but we wanted to really stress test a few RPC paths.

Fortunately, we are not the first. We needed a mechanism to simulate RPC calls and with a little digging (actually, it was pretty hidden in a mess of less than helpful posts), we found a rough and ready class that we used – easier to borrow something than to build from scratch. Here is the project page:

http://code.google.com/p/gwt-syncproxy/

Once we adapted the code to our specific case and removed the app engine stuff, we were good to go with a quick framework that lets us write RPC test cases that look like:

RPCService service = 
       (RPCService ) SyncProxy.newProxyInstance(RPCService.class, 
                                                "http://mydomain/my-web-app/",
                                                "path");
String result = service.executeRemoteMethod(param1, param2);

We use the SyncProxy to instantiate the RPC service, than call a method on the service interface and wait for the result. That’s my definition of easy to use. The great thing is the test code is synchronized so we don’t have to worry about ansyc calls in the test code and can evaluate the result for correctness immediately.

Next up is to use Apache JMeter to simulate load. This again is relatively easy. By implementing the JavaSamplerClient of JMeter, we can add our test case as a java class that gets executed. There is sample code available – take a look at SleepTest included in the source download.

We did have to modify the source code to add the ability to include the CookieManager in the JavaSamplerClient. This required a bit more digging around and modifying of JMeter but is doable in an hour or so and changes 3-4 JMeter files in the java protocol package. If you want to do this, look at the HTTP protocol package. We did this because our code retrieves cookies via a standard HTTP call first and then calls our java test case which needs to access the cookies. Other than that, we used the stock framework and can easily load test our app.

In fact, with these two pieces (SyncProxy and JavaSamplerClient) we quickly write test cases for the critical parts of our app that we want to load test. That’s my task for next week – yeah!

The great part is after a day of testing different components, our performance and scalability is even greater than what we thought it was (ok we did make one minor change to remove some contention) and we have lots of room for growth with our current architecture.

The only part I couldn’t figure out was how to get JMeter to add our jar files to it’s internal classpath. We tried using the search_paths and user.classpath parameters, but nothing we did seemed to allow JMeter to pickup our jars from a different location. It’s just an annoyance to have to build the components and then move the jars to the ext directory of JMeter. Would be nice if we could get the search_paths parameter working. Oh well, maybe another day.

All in all, it took about a day to become familiar with these tools and write our test cases to cover our most frequently used code paths.

We have an external API that can be called from any programming language. In this day and age, who doesn’t. For a couple of the calls we have some parts that use encryption within the message. When we original designed the system, we did for internal consumption only. Now we are going back and opening those parts up to external users. Yeah! However, this is where the fun starts.

For the encryption, we use the Cipher: AES/ECB/PKCS5Padding

One of our clients uses php for their application and couldn’t figure out how to do the encryption/decryption with this particular cipher. So I got to explorer php encryption today. The popular mcrypt function seems to be what everyone uses with php. It’s built in, so no surprise there.

The difficult part is padding is not supported. There is some information out in the ether, but nothing seems to put it all together for us. So here is a quick primer.

This php program will take a message and a key and produce an encrypted string which matches our java implementation:

<?php
echo "\n";
 
function hex2bin($hexdata) {
  $bindata = "";
 
  for ($i = 0; $i < strlen($hexdata); $i += 2) {
    $bindata .= chr(hexdec(substr($hexdata, $i, 2)));
  }
 
  return $bindata;
}
function pkcs5_pad ($text, $blocksize){
    $pad = $blocksize - (strlen($text) % $blocksize);
    return $text . str_repeat(chr($pad), $pad);
}
 
function pkcs5_unpad($text){
    $pad = ord($text{strlen($text)-1});
    if ($pad > strlen($text)) return false;
    if (strspn($text, chr($pad), strlen($text) - $pad) != $pad) return false;
    return substr($text, 0, -1 * $pad);
}
function encrypt($str, $key) {
  $key = hex2bin($key);
  $td = mcrypt_module_open("rijndael-128", "", "ecb", "");
  mcrypt_generic_init($td, $key, "0000000000000000");
  $size = mcrypt_get_block_size('rijndael-128', 'ecb');
  $str = pkcs5_pad($str, $size);
  $encrypted = mcrypt_generic($td, $str);
 
  mcrypt_generic_deinit($td);
  mcrypt_module_close($td);
 
  return bin2hex($encrypted);
}
 
echo encrypt("4", "ERT345");
 
echo "\n";
?>

The key points are:
1) we have to pad the message ourselves in encryption function (or remove it from the decrypted string)
2) the IV needs to be 16 0s (or 32 if using 256bit encryption)
3) AES is refered to as rijndael in php
4) we have to translate the key using hex2bin

The above function matches what we do in java:

Key secretKey = new SecretKeySpec(fromHex(key),"AES");
Cipher aesCipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
aesCipher.init(Cipher.ENCRYPT_MODE, secretKey);
String encryptedMessage = toHex(aesCipher.doFinal(message.getBytes()));

While short and sweet, it actually is not the most straight forward thing to do in php. Dam padding . Anyway, hope this helps someone someday.

…for integrating with their system. What should be a quick job, takes about 3 times longer than it should due to their total inadequate documentation.

I integrated Paypal Web Pro (express and direct) into our system this week. The direct portion wasn’t too bad as we already have other payment processing gateways that we hook into. And there are only so many ways to pass credit card information in an interface.

Their Web Pro Express checkout is a total different experience. Their documentation is haphazard and scattered and total lacking. For example, in the key/value interface, a VERSION key pair is needed. To find what a valid value for that version is though you have to google it and then you find a page on their development site that is pretty deep in the site where a paypal developer says he will maintain the latest version number. For grins, the only documented reference I found was version 2.0.1 (circa 2006/2007), the latest is 62.0 (that’s a lot of version in 4 years).

Their sandbox environment is so slow and having to login to the sandbox with a “master” user/password before running any tests just blows. I’ve automated it know; but just another hurdle that doesn’t really need to be there.

And I almost forgot, setting up a “test” web pro account was an exercise in futility. I spent over an hour trying to get it setup. Their is a form to “apply” for web pro account; but it never liked the email addresses I would give it. Altough this could be user error, I really hope their production process is a bit easier. I finally found a shortcut method to web pro enable one of my existing test accounts – don’t ask how though, I just remember having to search their forums and eventually found something after several false starts. Basically, you have to “login” to the test account and go to a setup tab and click a link on the right (I remember this cause the forum posting had the link on the left and named something else). Things change over time, but is it really too much to ask for a simple checkbox on the test account. I mean, if you make us setup test accounts, why not add a couple of check boxes.

Anyway, once I got the interface going, it wasn’t too bad. I actually didn’t really on the result parameter documentation too much. The call to GetExpressCheckoutDetails returns a lot of fields that are not documented.

And the callback to obtain shipping methods and taxes is nice; but having to specify one method as the default or you won’t use any of the methods is a bit draconian. They could just default to the first one. And I do have a beef with how the display shipping methods. They have two parameters: shipname / shiplabel. The docs are not accurate – or maybe just lacking. It says the shipname is the internal name and the shiplabel is what is shown to the user. Great, I’ll put my internal id in shipname.

Nope, paypal shows shipname shiplabel with a space to the user so you get stuff like:
XX methodname
YY methodname

Arg, and you can’t leave shipname off. I’m still playing with this one a bit; but I will either use a single space or a sequential number. And I would have thought that shipname is what is passed back; but nope again, they pass back shipname shiplabel as 1 string with a space between the two.

Anyway, I feel a bit better and my test suite just finished so time to wrap up this rant. Glad I’m done with it.