Ivan Zuzak – blog

Extending and customizing Google Code project hosting

Posted in web development by Ivan Zuzak on December 14, 2009

This blog has moved to a new domain. You can view this post here.

I’ve used Google Code project hosting for almost every project I’ve started in the last few years. It’s really great and I love both the Web UI and all the functionality-related features (which Google is continuously extending). However, one aspect in which I’ve found Google Code project hosting strangely lacking is customizability and extendability in adding rich content and even external applications to a project site. Wouldn’t it be great if we could add a special “Testing” and “Group discussion” tab or an easy way to embed presentations and JavaScript code into wiki pages? This would boost the usability of project sites and make the visitor experience so much better.

So, I’ve been thinking about this idea and playing with hacks and workarounds for extending Google code sites for my projects, and what I’ve discovered is that I’m not alone – several other projects have been doing this also. So in this post I’ll try to cover two approaches to extending and customizing Google Code project hosting: embedding 3rd party content using Google Gadgets and viewing non-source files in the repository using the SVN mime-types property.

The first idea for extending project sites is to embed 3-rd party content into wiki pages (Google Docs presentations, JavaScript programs and such). Although it is possible to use a large subset of HTML in project wiki pages, <iframe> and <script> tags are not available. However, embedding Google Gadgets is supported. Google Gadgets are ” simple HTML and JavaScript applications that can be embedded in webpages and other apps” (think clock, calendar, twitter and similar widgets) and are, in essence, an HTML <iframe> embedded in the parent page. Although there is a public directory of existing Gadgets which you could add to your wiki pages, what’s more interesting is that you can build a Gadget yourself and that it’s really easy to do. So, finally, the first idea for extending Google Code project hosting is to build your own Google Gadgets which wrap the content you want to embed in your wiki page. Here’s what you need to do:

  1. Create a file containing a blank, content-less XML Gadget specification — you can just copy/paste the code from the Google Gadgets tutorial.
  2. Define the content of the Gadget. Depending on what you want to embed in your wiki page, there are two ways to go here:
    1. If you want to embed an existing web page or rich content available on the Web, the content of the Gadget will be a single <iframe> element with the src attribute pointing to the URL of that web-page.
    2. If you want to embed content which does not exist on the web, you will have to write the HTML + JavaScript code that defines that content yourself and inline it in the Gadget. Be sure to check out the Google Gadgets developer guide.
  3. Save and upload the Gadget XML specification file into your project repository.
  4. Edit one of your wiki pages and create a Google Gadget element that references the URL of the uploaded specification. Note that you must reference the raw-file URL of the XML spec which you can get by viewing the uploaded file in your project’s Source tab and then clicking “View raw file” under File info.

Example 1

Example 1 - PubSubHubBub project

Example 2

Example 2 - FeedBuster project

Example 3

Example 3 - Pmrpc project

Example 4

Example 4 - UrlEcho project

And here are some examples that show the usefulness of this approach:

  1. Embedding Google Docs presentations that visually explain details of your project (example project wiki page and XML spec). To get a Google Docs URL which you can use for embedding, go the presentation’s page in Google Docs, click “Share – Publish/Embed”, and copy the URL under “Your document is publicly viewable at:”.
  2. Embedding a FriendFeed discussion groups for in-place discussions (example project wiki page and XML spec — if you see a blank page, view the source of the page). To get a FriendFeed group URL which you can use for embedding, use the official FriendFeed instructions (basically, it’s http://friendfeed.com/GROUP_NAME_HERE/embed e.g. http://friendfeed.com/feed-buster/embed).
  3. Embedding QUnit testing pages for following project stability and cross-browser testing (example project wiki page and XML spec). This one is a bit complex — first you’ll need to create and upload a HTML page containing the QUnit test page (setting the mime-type property to text/html using instructions below) into your repository, and then link to the raw page URL in the <iframe> inside your Gadget.
  4. Embedding dynamic content and forms (example project wiki page and XML spec). There is no outside content in the Gadget spec here, everything is in the Gadget. Note that the XML spec in the example also contains CSS styles so that the Gadget content has the same look-and -feel as the rest of the wiki page so visitors won’t event know that a Gadget is there.

The second idea for extending project sites involves using the source repository for hosting non-source files which will be viewed by project site visitors. This is useful if you want the freedom of creating custom web pages (not embed them into project wikis) and if you want to keep all those files in the project repository (and not on some other server). The idea here is to use the raw-file URL for files in the repository – visiting the raw-file view URL returns only the content of the file without the UI stuff of Google Code. The problem is that, by default, any file uploaded to the repository is attributed to be of type text, so when you view the raw-file URL in the browser you get the content as text, which is useless if you want to see a HTML page. The good thing is that using custom SVN per-file properties it is possible to set the type of the file when it is uploaded in the repository. So, finally, the second idea for extending Google Code project hosting is to upload your content files into the repository and explicitly set their mime-types to the correct value. Here’s what you need to do :

  1. Create your content and save it into a file on your hard drive in the project repository folder.
  2. Right-click the file and under TortoiseSVN -> Properties click “New”. In the “Property name” drop-down list choose “svn:mime-type” and ender the correct mime type for that file. You can find a list of all mime types here. E.g. for HTML pages it’s “text/html” (this is for TortoiseSVN, command-line version is here).
  3. Upload the file to the repository.
  4. Obtain the URL to the raw-file by viewing the uploaded file in your project’s Source tab and then clicking “View raw file” under File info.
  5. Create a link to the obtained URL from one of your wiki pages.

Example 1

Example 1 - PubSubHubBub project

Example 2

Example 2 - Closure library project

And here are two examples that show the usefulness of this approach:

  1. Creating a protocol specification with custom styling (example project page and source file)
  2. Creating advanced API docs (example project page and source file — note: you will not be able to view the source file, access is being blocked somehow)

And that’s it. Go fancy-up your project sites and let me know if you know any other cool ways of extenting Google Code project hosting. @izuzak

Advertisements

Inter-window browser communication and how to make it better

Posted in web development by Ivan Zuzak on October 10, 2009

This blog has moved to a new domain. You can view this post here.

As promised in my post on named arguments in JavaScript, this post will be about pmrpc – a project I’ve been working on with Marko for the last few months.

But first, a bit of background. From mid 2007. when I was interning at Google up until today and my dissertation-related research, an issue that’s consistently been popping up in anything I worked on is inter-window communication. Inter-window communication (IWC) is the process of transporting data from one window object within a browser to another window object inside the same browser. What’s relevant here is that window objects are both full-size windows and iframes – window objects that can be embedded within other window objects (think of an iGoogle page and all the gadgets in it).

The reason why IWC is an issue is the security model of communication within browsers. Same Origin Policy (SOP), a security policy introduced to reduce security risks, restricts access of processes executing within browsers. In essence, SOP prevents a document or script loaded from one domain from getting or setting properties of a document from another domain. For example, a page loaded from http://www.google.com can not directly access data in an iframe loaded from http://www.yahoo.com.

Although SOP turns Web applications into a set of isolated units, the need for communication between those islands is growing rapidly. With the advancement web technologies, web applications were slowly becoming componentized and many current web applications are based on mashing-up content from different sources – map gadgets, finance widgets, blog commenting toolboxes, games, chat widgets and many, many others. Since widgets, content encapsulated within an iframe, are the most popular method of including cross-domain content on a web page, inter-widget communication is most often implemented as inter-window communication.

A lot of groups and projects are researching models and developing systems that address the problem of communication between widgets and between windows in general. So here’s a short overview of this broad space of R&D. Various widget specifications are trying to formally define and standardize what a widget actually is. Browser security in general and various issues concerning intra-browser communication and mashup security are being heavily researched. Low-level IWC mechanisms are being proposed and implemented, sometimes even as browser hacks, and high-level communication models are being researched and developed. Even patents have been filed for methods of inter-window and inter-widget communication.

Among all of these efforts, one is catching on on a wider scale – the HTML5 cross-document messaging specification aka the postMessage API. The postMessage API has been implemented in all popular web browsers and enables secure message-oriented communication between any window objects – scripts in any window may send messages to any other window and receive messages from scripts in other windows. Security is established by two access-control mechanisms: 1) the message sender may specify the domain from which the destination window must be loaded in order for the message to be delivered, and 2) the message receiver may inspect the domain from which a message was sent and drop unwanted messages.

The problem with postMessage is that it’s a very low-level and general communication mechanism which requires a lot of boiler-plate code every time it is used and even more if one wants to implement advanced communication or access-control models. Have a look at the example on using postMessage in the HTML5 spec to see what I’m talking about. However, it’s exactly this generality and low-levelness that makes postMessage an excellent foundation to build upon! Some of the projects I mentioned above have been working on advanced communication libraries based on postMessage, but none that I know of are open-source and applicable in any context but rather tied to a specific application environment. Furthermore, advanced access control, fault tolerance and many other aspects of inter-window communication have been under-explored. These drawbacks were the main motivators for developing pmrpc.

Pmrpc (postMessage remote procedure call) is a JavaScript library for advanced inter-window cross-domain communication. The library utilizes the postMessage API as an underlying communication mechanism and extends it to a remote procedure call (RPC) model using JSON-RPC. JSON-RPC is a transport-independent protocol that uses the JSON data format for formatting messages. The pmrpc library provides a simple API for exposing and calling procedures from windows or iFrames on different domains.

Here’s a simple example for calling a function in another window:

1. First, a procedure is registered for remote calls in the window or iFrame that contains the procedure:

// [window object A] exposes a procedure in it's window
pmrpc.register( {
  publicProcedureName : "HelloPMRPC",
  procedure : function(printParam) { alert(printParam); } } );

2. Second, the procedure is called from a remote window or iFrame by specifying the window object which contains the remote procedure, name of the procedure and parameters:

// [window object B] calls the exposed procedure remotely, from its own window
pmrpc.call( {
  destination : window.frames["myIframe"],
  publicProcedureName : "HelloPMRPC",
  params : ["Hello World!"] } );

Furthermore, pmrpc also provides several features for advanced communication: callbacks for successful and unsuccessful calls, access control lists on both the client and server side, fault tolerance using user-defined retries and timeouts for each call, registering asynchronous procedures. The features and their usage are explained in detail on the pmrpc API docs page and a short presentation, so I'll just give an example of an advanced pmrpc call: 1. First, a synchronous procedure named "HelloPMRPC" accessible from windows and iframes loaded from any page on "http://www.myFriendlyDomain1.com" or exactly from the "http://www.myFriendlyDomain2.com" page is registered:

// [window object A] exposes a procedure in it's window
var publicProcedureName = "HelloPMRPC";
var procedure = function (alertText) { alert(alertText); return "Hello!"; };
var accessControlList = {
  whitelist : ["http://www.myFriendlyDomain1.com*", "http://www.myFriendlyDomain2.com"],
  blacklist : []
};

pmrpc.register( {
  "publicProcedureName" : publicProcedureName,
  "procedure" : procedure,
  "acl" : accessControlList
} );

2. Second, the procedure "HelloPMRPC" is called with a single positional parameter "Hello world from pmrpc client!". In case of success, the return value will be alerted while in case of errors the error description will be alerted. The call will be retried up to 15 times, waiting 1500 milliseconds between retries. Windows or iFrames loaded from any domain, except from http://evil.com, may process the call:

// [window object B] calls the exposed procedure remotely, from its own window
var parameters = {
  destination : window.frames["targetIframeName"],
  publicProcedureName : "HelloPMRPC",
  params : ["Hello world from pmrpc client!"],
  onError : function(statusObj) {alert(statusObj.description);},
  retries : 15,
  timeout : 1500,
  destinationDomain :  {  whitelist : ["*"],  blacklist : ["http://evil.com*"] }
};

pmrpc.call(parameters);

Pmrpc is an open-source project developed by Marko Ivankovic and myself and is available under the Apache v2.0 license. The library works in all web browsers that implement the postMessage API (Firefox 3, Google Chrome, Internet Explorer 8). For more information please visit the project homepage on Google Code and the API docs page for a full description of the API, feature list and usage examples. I'll end this post with a few ideas on making pmrpc even better: 1. discovery - currently, in order to call a remote procedure one needs to have a reference to a window that exposes that procedure. Sometimes obtaining a reference to a window is not very practical and easy (or possible) and a way to find windows by various criteria is needed. For example, discover all windows with a specific name or location or discover windows that expose a procedure with a specific name. 2. web workers - web workers are another specification being developed in parallel with HTML5. The specification "defines an API for running scripts in the background independently of any user interface scripts". What's interesting is that the postMessage API is used to communicate with web workers and therefor pmrpc could be extended to include support for web workers. 3. publish-subscribe mechanism - there are lots of communication models other than remote procedure call. One of the most popular models is publish-subscribe in which publishers publish events on channels and subscribers subscribe for events on these channels. Messages are delivered from publishers to subscribers without having a direct reference to each other. I'd appreciate any feedback on pmrpc and our ideas for future work. What are your ideas for the future of inter-window communication? @izuzak

A better future for the FriendFeed tools and services ecosystem

Posted in web development by Ivan Zuzak on September 2, 2009

This blog has moved to a new domain. You can view this post here.

FriendFeed is the only social-networking application I get and like using, and one of the reasons why is that it’s very community-oriented. The groups and friends approach of segmentation of attention easily directs both topic-oriented discussions and friends-oriented discussions towards the right people.

One other nice thing about FriendFeed is that, although not as popular as Twitter or Facebook, it has a healthy ecosystem of developers extending and building upon the platform.  Lots of great user scripts, bookmarklets, feed-processing services, sms services and web apps have been developed and recently even more so with the release of the new API.

However, what I don’t like about the ecosystem is the separation of developers which develop tools and their potential users. Developers mostly use the official FriendFeed API Google group and FriendFeed API group for discussing programming issues and the API, while non-programmers mostly join the FriendFeed Google group and the FriendFeed Feedback and FriendFeed for beginners FF groups. This definitely isn’t the best solution for the future of the ecosystem. There is no central hub for the exchange of ideas between users and developers – users have a hard time finding existing tools and suggesting new ones while developers are have a hard time finding out what users need.

Enter FF tools and services – a new FriendFeed group for both developers and users to discuss existing and future FF-related tools and services. The goals I hope the group will help catalyze are:

  1. establish a central place where both developers and users of FriendFeed tools can go to sync up and discuss ideas,
  2. compile a single list of all FriendFeed tools and services – if you know or use a FF-related web app, bookmarklet or user script – just add it to this list and other users can use it to find cool apps,
  3. compile a single list of suggestions for future FriendFeed tools and services – if you have an idea for a cool FF web app or service which doesn’t yet exist – just add it to this list and a developer with some free time might build it.

And that’s it – a place where the FriendFeed tools and services ecosystem can grow. @izuzak

Tagged with: ,

How to supercharge your free AppEngine quota?

Posted in web development by Ivan Zuzak on August 27, 2009

This blog has moved to a new domain. You can view this post here.

AppEngine is awesome! Google’s cloud computing platform is hosting countless web apps, from Google products to innovative personal projects, and is constantly being upgraded with cool new featuresAnd best of all, AppEngine charges for resources (computing, storage and bandwidth) using a freemium pricing model:

“Each App Engine application can consume a certain level of computing resources for free, controlled by a set of quotas. Developers who want to grow their applications beyond these free quotas can do so by enabling billing for their application and set a daily resource budget, which will allow for the purchasing of additional resources if and when they are needed.”

For example, the default free quota has a daily limit of 6.5 hours of CPU time with a maximum rate of 15 CPU-minutes/minute, while the billing enabled quota has 6.5 hours of free CPU time plus a maximum of 1,729 hours of paid CPU time with a maximum rate of 72 CPU-minutes/minute.

Billing for paid resources is controlled with a daily budget which developers set for each of their applications. The daily budget controls the amount of extra resources which may be purchased each day if the app steps over the free quota. The application developer defines how the daily budget is split between each of the billable quotas – outgoing bandwidth, incoming bandwidth, CPU time, stored data and number of recipients emailed. For example, $1 (the minimum daily budget) gets you 5 more CPU hours ($0.10 per CPU hour) and 4 more GB of outgoing bandwidth ($0.12 per GB).

Other than billable quotas, there are also unbillable quotas which increase when you switch to the paid model but are not billed. So, by switching to the paid model, irregardless to how you distribute your budget, your app gets a free bump in the:

  • number of requests it can process from 1,300,000 to 43,000,000 and from 7,400 to 30,000 req/min,
  • number of outgoing HTTP requests from 657,000 to 46,000,000 and from 3,000 to 32,000 req/min,
  • number of memcache calls from 8,600,000 to 96,000,000 and from 48,000 to 108,000 calls/min,

and many others. These are very usable improvements by themselves for apps that need to process a lot of requests (or bursts of requests) which don’t consume a lot of CPU time, or apps that need to make a lot of outgoing HTTP requests that don’t consume a lot of bandwidth.

So here’s the idea for supercharging your free AppEngine quotas:

  1. switch you app to the paid model by enabling billing,
  2. enter the minimum daily budget ($1),
  3. distribute the budget over resources you are 100% sure will not consume their free quota (e.g. if you have a stateless app which doesn’t use the database, put the whole $1 in the stored data quota).

Since you put your budget on resources which won’t consume the entire free quota and since AppEngine doesn’t charge you anything if the app doesn’t step over the free quota – you are essentially getting a better free quota.

EDIT (Sept 5 2009): Yesterday, a week after my original post, Google released AppEngine SDK 1.2.5 which includes a major feature a lot of developers have been waiting for – XMPP support. What’s interesting and relevant for this post is that XMPP has it’s own quotas for the number of API calls, data sent, recipients messaged and invitations sent, and – none of these quotas are marked as billable. Therefore, using the recipe for supercharging your free quota will also get you a significant boost in your XMPP quotas (657k -> 46000k API calls, 657k -> 46000k recipients messaged and 1k -> 100k invitations sent). Pretty cool!

EDIT (Nov 2 2009): I just noticed that it is possible to increase the limit on the number of apps each AppEngine user can create. Currently, each user may create up to 10 AppEngine apps and there was no way to create more when you reached the limit, even by paying (or was there?). However, Google engineers have been approving explicit user requests for increases in the number of apps created on the official AppEngine group. People who have requested an increase have had the limit set to 20 apps.

Are there any other hacks for squeezing more out of AppEngine?  @izuzak

Named arguments in JavaScript

Posted in web development by Ivan Zuzak on August 9, 2009

This blog has moved to a new domain. You can view this post here.

While working on another project (pmrpc, which I will write about soon), the implementation of a feature came down to calling JavaScript functions with named arguments. Unfortunately, the JavaScript language (or EcmaScript to be exact) doesn’t support named arguments, rather only positional arguments.

Usually, JavaScript developers get around this by defining functions to accept a single object argument which acts like a container for all other named parameters:

// params is a container for other named parameters
function callMe(params) {
  var param1 = params['param1'];
  var param2 = params['param2'];
  var param3 = params['param3'];

  // do something...
}

var result = callMe({ 'param3' : 1, 'param1' : 2, 'param2' : 3 });

This solution works nicely when you are in the position to modify every function to be called this way. However, this is not always the case (e.g. when using external libraries), and also isn’t as readable as having a proper parameter list, and introduces code overhead in every function that implements this principle. What I needed for my project is something that enables calling any JavaScript function with named arguments, without modifying the function itself.

What I came up with (code presented below) is based on the following:

  1. using the toString method of the Function prototype, it is possible to retrieve a string representation of any function definition
  2. from the string representation, it is possible to parse the number, order and names of function parameters
  3. from a dictionary of named arguments, it is possible to construct an array of arguments in the correct order using the parsed representation
  4. using the apply method of the Function prototype, it is possible to dynamically call a function, passing an array of arguments instead of an argument list.
// calls function fn with context self and parameters defined in dictionary namedParams
function callWithNamedArgs( fn, self, namedParams ) {
  // get string representation of function
  var fnDef = fn.toString();

  // parse the string representation and retrieve order of parameters
  var argNames = fnDef.substring(fnDef.indexOf("(")+1, fnDef.indexOf(")"));
  argNames = (argNames === "") ? [] : argNames.split(", ");
  var argIndexes = {};
  for (var i=0; i<argNames.length; i++) {
    argIndexes[argNames[i]] = i;
  }

  // construct an array of arguments from a dictionary
  var callParameters = [];
  for (paramName in namedParams) {
    if (argIndexes[paramName]) {
      callParameters[argIndexes[paramName]] = namedParams[paramName];
    }
  }

  // invoke function with specified context and arguments array
  return fn.apply(self, callParameters);
}

// example function
function callMe(param1, param2, param3) {
  // do something ...
}

// example invocation with named parameters
var result = callWithNamedArgs(callMe, this, { 'param3' : 1, 'param1' : 2, 'param2' : 3 });

This could also have been implemented as a Function prototype method and be called directly on a function object, which would simplify usage since the target function parameter wouldn’t have to be passed. But, if the CallWithNamedArgs function indeed was implemented prototype function, could the context parameter be removed also? I.e. is it possible to retrieve the current execution context (this object) of the target function within the CallWithNamedArgs method? The problem is that the execution context within the CallWithNamedArgs method is the Function object on which the CallWithNamedArgs method was called. But what we need is the execution context of the target function, which is the Function object on which the CallWithNamedArgs method was called. That is, what we need is the current execution context of the parent object. Any ideas? @izuzak