Friday, December 11, 2009

Google App Engine vs. PHP

I've been messing around a bit with ip information, proxy detection, and geolocation at work, and wanted to build some of my own services in order to get a better understanding of them. So far, the only thing up at http://ip.emmesdee.com is a basic replacement for ipchicken and whatismyip, but what's been slowing my experimentation has been a couple of surprising limitations in Google App Engine.

In order to check for a tor proxy via tordnsel, it would be much easier if GAE could perform DNS queries. You have to think this is coming with their recent announcement of public DNS servers. (I wish they would host DNS too! dotster charges $10/yr for dns, and my friends down under have had problems with godaddy DNS)

In order to do basic geolocation, it would be simplest if I could just import a sql database and query it. Unfortunately, it is still far easier to just bang out a little PHP than it is to figure out how to migrate to BigTable.

All things considered, GAE still feels a bit beta, but hopefully these problems will be gone in another year or so of GAE.

Friday, October 16, 2009

Google App Engine as static web server, bazaar vs. git

I've been serving http://www.emmesdee.com as a google app ( http://msdewww.appspot.com ) since august 2008 without any problems, but I'm thinking about switching to DryDrop. My big resistance at this point is actually version control. When I did my research into distributed version control a couple of years ago, bazaar looked more interesting to me than git, partially because it was more like svn to me than git, and partially because it seemed the easiest to host. All you need for a "repository" is just ssh access to a linux box and a directory.

Wednesday, July 29, 2009

grails conditional bootstrapping

I've been fortunate enough to fit grails into a new development project, and it's definitely made things more pleasant. I'm still missing eclipse and autocompletion a bit, but overall I am more productive. Anyway, I've created a bunch of objects in my grails-app/conf/BootStrap.groovy for testing, that also happen to be good initial values for production. However, they're getting created every time I restart the app, causing duplication! I've seen many references to checking GrailsUtil.environment, i.e.
def init = {  
if (GrailsUtil.environment=="development") initData();
}
This is close, but this doesn't do what I want. When I'm doing some of my testing, I don't need my objects created, as I want to persist my data between restarts. The decision to re-create my objects isn't the environment, it's whether I've told my DataSource to delete all data. That happens when the dbCreate property is "create" or "create-drop".
def init = {
// only bootstrap if data was deleted
def dbCreate = org.codehaus.groovy.grails.commons.ConfigurationHolder.config.dataSource?.dbCreate
if(dbCreate.startsWith("create")) {
initData();
}
}
Please forgive the non-grooviness of my code, I still like my groovy code to read like java.

Thursday, June 18, 2009

google app engine monitoring service lessons learned

As mentioned earlier, we were in need of some monitoring for our co-loc. After some experimentation, it turned out to be easiest to build in the co-loc almost entirely in nagios. For the external monitoring component, I deployed a web proxy called mirrorrr onto google app engine to provide us with basic functionality from outside of the co-loc to check dns and other potential network issues. I need to follow up with a single monitor on the co-loc itself from google app engine, but our existing monitor from siteuptime can suffice for now.

Nagios appears to have a couple annoying quirks regarding DNS though. It seems to be rather insistent on quering IP addresses for low level services, rather than doing more of a system-level monitoring.
  • check_http appears to resolve the hostname into an IP address before checking it, unless the -H option is used. This breaks google app engine and anything else that uses a virtual hosts-like mechanism to determine what page to serve.
  • check_smtp does not seem to do an MX lookup for a host. Instead, it resolves to our web server and tries to open the web server for SMTP.
  • mirrorrr has a 1 hour cache by default. It should be minimized or disabled when used for monitoring.
  • TBD: The local mirrorrr install should probably get an IP range filter added to it so that it is more difficult to DOS.
  • nagios isn't too happy about passing messages around machines. My main options appear to involving choosing the least of three evils:
    • adding a private key to the nagios machine and sshing everywhere
    • installing NPRE as a daemon on every machine and querying them (it's not that lightweight and most of the servers need more memory badly)
    • calling ncsa_send from some cron shell scripts to a relatively insecure mechanism on the nagios machine. I opted for ncsa_send, but the port is only visible to the intranet, and there are a couple of machines in a dmz that can't reach it easily.

Wednesday, May 20, 2009

Grails Integration

I finally got the chance to do a bit of green field development, and a good excuse to integrate grails. I've always had trouble justifying the use of it in the past, because it is difficult to extend an existing java project with grails. However, I the opportunity to break off a piece of logic and implement it using grails while using the existing java app to read the results.

Requirements that allowed for easy grails:
New database used to store data
Standalone CRUD GUI
database used as primary form of communication to other components

Inputs:
Reporting component
Job requests through a mysql table
Standalone CRUD GUI

Outputs:
2 mysql tables to replicate to the front end
6 mysql tables to replicate to the reporting server

Tuesday, March 24, 2009

google app engine monitoring service

My plate is too full to start this project, but we need a simple monitoring service for our service, similar to nagios. I'm thinking about throwing this together on the new Google App Engine, because hosting a nagios server in our data center is pointless and it seems like a good fit.

Update: I ended up using nagios after all, and google app engine as a simple web proxy for round trip http testing.

Monday, January 5, 2009

Exporting MySQL to CSV

For some quick reporting, I needed to export some data from a table into csv format, and this proved to be particularly handy.

SELECT * INTO OUTFILE 'output.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM users;

http://forums.mysql.com/read.php?79,11324,11324#msg-11324 contains enough info on both of the common approaches, the other of which uses sed.