While GAE might have great infrastructure for web applications (I’m using the java version), storing large amounts of data is pushing the envelop of what it can do. To store large files, I had to break each file up into a little less than 1MB chunks (1MB is the largest unit of storage for a GAE object). When I started to retrieve the 1MB objects, I would periodically run into GAE datastore timeout issues.
Retrieving large entities does not seem to be worked out yet. As a note, I would get timeout errors retrieving a single 1MB entity by it’s primary key. After fiddling around a bit, I have decided that I’m pushing the GAE infrastructure. So, I’m back to trying out Amazon’s S3. Now that I have a prototype of the application, it shouldn’t be too hard to port to S3.
In fact, I’ve been able to setup a S3 look alike instance on my local hardware using Eucalyptus; but that’s a story for another day.
Can you see what is wrong with this code? I’m sure if your an old hand at file IO it’s obvious. It took me two hours to catch on to what was wrong.
1
2
3
4
| byte[] data = new byte[MY_DATA_SIZE];
InputStream in = get_input_stream_from_someplace();
in.read(buffer);
System.out.println("data[" + new String(data) + "]"); |
The newly created String does not contain all of the data. The call to in.read() was not reading all of the input stream – even though all of the data might have been available. There’s probably some type of internal buffering going on. What needs to happen is the code needs to keep looping until it has read all the data from the stream. Each loop will store the last in.read() into a temporary array and then append the just read data to a permanent byte array. Rinse and repeat until there is no more data. I ended up with something like this:
1
2
3
4
5
6
7
8
9
10
11
| byte[] data = new byte[MY_DATA_SIZE];
byte[] buffer = new byte[10000];
InputStream in = get_input_stream_from_someplace();
int len = 0;
int offset = 0;
while((len = in.read(buffer)) != -1) {
System.arraycopy(buffer, 0, data, offset, len);
offset += len;
}
System.out.println("data[" + new String(data) + "]"); |
Basically you have to read in a chunk of data into the temporary byte array and then copy that data into the permanent byte array. Make sure you size the permanent byte array correctly.
Today we officially turned off all SSL2 support so that we could be PCI compliant. After this we passed the certification scan. Yeah. We bit the bullet and decided that SSL2 has been deprecated for years and the versions of IE6 we could find on different computers all supported SSL3. So, while there isn’t much information out there about SSL2, we will take that as a sign that we will not have an issue. Besides, everyone else who is PCI compliant will have the same issue.
If you want to setup your web servers to be PCI compliant, these are the settings we used:
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
These will ensure that only SSL3 and TLS1 are used with Medium to High Cipher protocols. Also, all “not secure” ciphers are expressly excluded from the acceptable list.
Here is a tool we used to scan our web servers to verify which protocols are being used (much easier to use this then to request a PCI scan from our vendor in order to verify this changed worked).
SSL Digger by Foundstone
At least for the time being I am going to try to create my application using AppEngine. They seem to bill based on actually consumed resources were Amazon would need a cpu running 24×7. I know I am technically consuming a resource by running the instance; but it will not be doing anything for most of that time. It would need to be on for the short periods of time when a user visits the site (one can dream about actually using an instance 24×7).
The next hurdle to overcome is google 1MB limit on things stored in their DB. We are uploading large files (700-1.4 GB). I am going to try and break things into 1MB chunks and see how that goes.
Our recent upgrade to GWT 1.6 went pretty smoothly. We have started to convert our Custom widget Listeners over to the new Event Handling framework. It turns out there is a conflict somewhere with the old incubator DatePicker class. I tracked it down to something to do with EventPreviewing and the PopupPanel is canceling events if the popup is modal. That is important for our code since the DatePicker is pop’ed open from a popup.
I didn’t track it down fully as I needed to get it to work. So I migrated from the incubator DatePicker version to the included GWT version. When that didn’t seem to solve the issue, I opened the DatePicker in a Popup descendant instead of a Dialog. That worked. I didn’t take the time to analyze why the DatePicker didn’t work; but some combination of the new Event Handling code and the Dialog box causes a conflict.