Perimeter Sweep

HTTP Caching 101

2011-11-16T09:55:00.005-07:00

This captures a basic learning curve around HTTP Caching, actually just a selected subset of a rather large topic. Another post driven by laziness - written in terse talking-point style, with most information either paraphrased or copied explicitly from the resources listed at end - with a touch of editorializing on my part. My thanks to those authors upfront.

Overview

Benefits: save bandwidth, reduce latency, lighten server load

Types of caches (server's responses can be cached by these)

gateway cache (shared): middle-man, facade in front of a server (for any client). This caches responses from one server for many clients.
browser cache (private): for BACK button, images, etc. This caches responses from many servers for one particular client.
proxy cache (shared): forward-facing towards web, client-side. This caches responses from many servers for many clients.

Here's some decent slideware with boxes and lines illustrating the above, and with some sequence diagrams illustrating the protocols discussed below.

Mechanisms

Expiration, Validation and Invalidation
note that caching is applied only to GET and HEAD
...this is yet another reason not to tunnel operations thru GET, etc abuse - requests won't make it to server w/proxy cache in play
and, another reason to use all verbs as intended: POST, PUT and DELETE should not ever be cached
in fact, POST, PUT and DELETE should result in cached content being invalidated

Pragma: no-cache - don't bother; from http://palisade.plynt.com/issues/2008Jul/cache-control-attributes/:

“pragma: no-cache” is linked to requests, and not responses. The RFC did not specify the behavior of this directive for responses. Hence, this directive does NOT instruct the browser not to cache a page. We often see this tag being misused when a page is served to the browser. Developers mistakenly set this directive expecting that the page will not be cached on the browser.

The sections of the HTTP 1.1 spec (http://www.w3.org/Protocols/rfc2616/rfc2616.html) that discuss caches include 13, 14.9, 14.19, 14.21, and 14.24-29.

Cache Control

Here's a subset of the cache-control mechanism to be discussed here:

non-cacheable:
HTTP/1.1 200 OK
Cache-Control: no-cache,no-store
user-specific private:
HTTP/1.1 200 OK
Cache-Control: private,no-cache,no-store
cacheable:
HTTP/1.1 200 OK
Cache-Control: max-age=3600,must-revalidate
Date: Sun 16 Oct 2011 15:30:00 GMT

Here's a summary on the semantics of the header values listed above ( the first explanation section of each directive comes from http://www.mnot.net/cache_docs/#CACHE-CONTROL, with subsequent text coming from section 14.9 of the HTTP 1.1 spec):

no-cache — forces caches to submit the request to the origin server for validation before releasing a cached copy, every time. This is useful to assure that authentication is respected (in combination with public), or to maintain rigid freshness, without sacrificing all of the benefits of caching. From the spec: This allows an origin server to prevent caching even by caches that have been configured to return stale responses to client requests. (Note that it's possible to specify field-names for this header that can apply more fine-grained meaning to the intended behavior; see section 14.9.1 for details.). However, a point of interest from this article:

“no-cache” directive, according to the RFC, tells the browser that it should revalidate with the server before serving the page from the cache..."no-store" tells the browser not only not to cache the page, but also not to even store the page in its cache folder. Whenever you’re serving a sensitive page, this is the cache control directive to use. Notice that of late, “cache-control: no-cache” has also started behaving like the “no-store” directive. To be on the safer side, we recommend that you use both “no-cache” and “no-store” when serving sensitive pages.

no-store — instructs caches not to keep a copy of the representation under any conditions.

private — allows caches that are specific to one user (e.g., in a browser) to store the response; shared caches (e.g., in a proxy) may not.

max-age= --- specifies the maximum amount of time that a representation will be considered fresh. Similar to Expires, this directive is relative to the time of the request, rather than absolute. is the number of seconds from the time of the request you wish the representation to be fresh for. From the spec: implies that the response is cacheable (i.e., "public") unless some other, more restrictive cache directive is also present. If a response includes both an Expires header and a max-age directive, the max-age directive overrides the Expires header, even if the Expires header is more restrictive. (Note that max-age can also be specified in a request from the client; the meaning here is, from section 14.9.3, that the client is willing to accept a response whose age is no greater than the specified time in seconds. There are also max-stale and max-fresh directives that further refine the user-agent's intentions; see section 14.9.3 for details.). Note that setting max-age to zero ensures that a page is never served from cache, but is always re-validated against the server.

must-revalidate — tells caches that they must obey any freshness information you give them about a representation. HTTP allows caches to serve stale representations under special conditions; by specifying this header, you’re telling the cache that you want it to strictly follow your rules.

What follows here is a general discussion of the Expiration and Validation models.

Expiration

With expiration, you tell the cache how long should a response be considered "fresh", i.e. on expiration it becomes "stale". This is more efficient than validation since cache need not go back to origin server until staleness.

Expires

This is useful for long-lived static images, and things that change on regular schedule. It's not without its problems:

server/client clocks must be in sync
times are in one-second resolution
you might forget to update date after expiration
spec prohibits Expires date more than one year in future
it's not useful for dynamic content

Cache-Control (i.e. max-age, s-max-age (shared caches only))

to address limitations of Expires
relative vs absolute time frames

Validation

With the validation model, the cache asks the origin server on each request if its cached response is still valid. This is probably the best option for dynamic content.

ETag/If-None-Match

You can think of ETag as a hash, fingerprint, etc. unique ID for a given page, resource, etc. Use this when it's inconvenient to store modification dates, or when the 1-second resolution of HTTP expiration model is not good enough, etc.

An ETag can be strong or weak - in a nutshell, strong means the response can be used to compose a larger response, i.e. it's valid as a partial piece of content; whereas weak means you should expect to use the response to reliably compose larger pages (etc.). If a request contains an If-None-Match header with a value equal to the ETag for the given resource, the response simply contains a 304-Not Modified response status with an empty body; this is useful for bandwidth savings. If any intermediary cache must go back to the origin server for validation, a useful implementation to consider on that origin server involves saving the ETag (in database, key-value store, in-memory hashmap, etc.) on a per-URL (etc.) basis, to additionally save on CPU (i.e. the ETag need not be recomputed on each request).

Note that there is a fair amount more to the ETag-based validation model than discussed here. Please consult the RFC for more details.

Last-Modified/If-Modified-Since

A date-based approach to validation can be done with the Last-Modified/If-Modified-Since protocol. Similar to ETag, the server might save the modification dates for a given resource (URL, etc.) and compare that with any request header value for If-Modified-Since, again returning 304-Not Modified if the resource has not been modified since that given date. Again, savings in bandwidth can be gained here.

Combining ETag with Last-Modified

There's a fair amount of ambiguity in blogs and web articles around what the behavior should be when requests send both types of validation headers and the server side supports both. To make things worse, the RFC 2616 is not the easiest specification to understand (in fact, there's an effort underway to improve that with a rewrite). I'm basing my interpretation on my own careful read of RFC 2616; you are invited to correct me if you believe I've got it wrong - note in the following that "server must execute the method" is similar to saying there was a "cache miss":

- etag matches, date is fresh: return status 304
- etag matches, but date is stale: origin server must execute the method and return that response
- etag does not match: origin server must execute the method and return that response, regardless of any date-based validation information in request
- etag matches, and there is no date-based header: return status 304
- no etag in request; date is fresh: return status 304
- no etag in request; date is stale: origin server must execute the method and return that response

For reference, here are the sections of the RFC that I used to determine the above recipe.

Invalidation

The third mechanism supporting caching (beyond Expiration and Validation) is invalidation. However, here you shouldn't need to do anything special with your web application -- it should be handled implicitly by any gateway or proxy cache that is in place. What you should do is return ETag and Last-Modified information in any response for a given resource (including GET, POST, PUT, DELETE) - such that any intermediary cache can not only cache as needed (for GETs), but also evict/update/otherwise-deal-with URLs that have been changed (via POST, PUT and DELETE).

Design Discussion

Checking for ETag alone is not as effective a caching solution as might be accomplished with use of If-Modified-Since. This is because the cache must validate the value of the ETag before returning a 304. In the case of a dynamic system where state changes frequently or unpredictably, however, ETag might be the best approach. In other cases where it is known e.g. that state won't change until a given time, use of If-Modified-Since is likely to be the most scalable solution, since the application can offload requests to reverse proxies.

Use of the Expires header can also be used in systems where it is known that state will change at precise times; however, this calls for client and server clocks to be sync, and requires explicit updates to the Expires header value when expiration occurs (which, if forgotten, leads to permanent staleness and zero benefit from a cache).

Note that date-based validation is limited to one-second precision. So it's feasible that a client could fetch a resource, saving the last-modified date in the response, then change the resource (or some other client could change it) and subsequently request it again immediately, but receive a 304 response since all of this took place within the same second. As such, I'd endorse use of the ETag as the preferred caching protocol if your service expects anything resembling high concurrency and requires strict correctness.

If proxy/reverse caches are expected to be part of the ecosystem and you want to control their behavior independently, consider using these headers (see RFC 2616, section 14.9.3 for details):

s-maxage= - similar to max-age, except that it only applies to shared caches.
proxy-revalidate - similar to must-revalidate, except that it only applies to proxy caches. From this article:

...useful when an authenticated page is being cached in the browser. You don’t want the proxy to cache and serve the page, whereas it’s fine for your browser to cache and serve the page.

From my understanding of the protocol so far, I'd also suggest adding the public header value to responses intended to be cacheable if HTTP authentication is involved:

public --- marks authenticated responses as cacheable; normally, if HTTP authentication is required, responses are automatically private. From HTTP 1.1 spec: Indicates that the response MAY be cached by any cache, even if it would normally be non-cacheable or cacheable only within a non- shared cache.

Here's additional information from HTTP 1.1. spec, section 14.8:

A user agent that wishes to authenticate itself with a server – usually, but not necessarily, after receiving a 401 response – does so by including an Authorization request-header field with the request. When a shared cache (see section 13.7) receives a request containing an Authorization field, it MUST NOT return the corresponding response as a reply to any other request, except that it MAY use the response in a subsequent request if s-maxage, must-revalidate or public headers are included in the response (see section 13.7 for details).

Other

Here are some things you can do to make your website more cache-friendly and cache-aware. Not all of these will necessarily apply to your context, especially when dealing with dynamic content.

Note that one assumption made in this writeup is that there is only one representation for any given URL. This may not be the case, however; representations may vary as per JSON vs XML encoding, use of compression or not, special markup for a given browser (or other user agent), and etc. This is where the Vary header comes into play; here's a nice explanation of this pattern.

There are many open source/commercial proxy and reverse-proxy caches available, notably Squid and Varnish (let alone Apache, Nginx, ... etc.).

Resources

http://en.wikipedia.org/wiki/Web_cache
http://www.peej.co.uk/articles/http-caching.html
http://tomayko.com/writings/things-caches-do
http://www.web-caching.com/welcome.html
http://www.mnot.net/cache_docs/
http://www.visolve.com/squid/whitepapers/ViSolve_Web_Caching.pdf
http://stackoverflow.com/questions/4360283/http-cache-control
http://www.w3.org/Protocols/rfc2616/rfc2616.html
http://wiki.squid-cache.org
http://symfony.com/doc/2.0/book/http_cache.html
http://hc.apache.org/user-docs.html
http://www.slideshare.net/rtomayko/https-bestkept-secret-caching

RESTful Design - Benefits, Patterns

2011-11-16T09:44:00.000-07:00

This captures odds-n-ends around RESTful design - why bother, what are the benefits, what are some patterns, etc. This is written in a terse "talking points" style, with most content either paraphrased or explicitly copied from the footnoted links at end of post; I've added some of my thoughts here and there.

An opening thought around what might be the benefit to understanding and leveraging aspects of RESTful design: since REST describes the way the web works, and the web is the single most scalable application ever known, we might do well to understand and embrace aspects of RESTful style.

REST describes a Resource-Oriented Architecture (ROA): the web is based on resource exchange, not on sending commands.

Selected excerpts from Roy Fielding's thesis[1]:

REST provides a set of architectural constraints that, when applied as a whole, emphasizes scalability of component interactions, generality of interfaces, independent deployment of components, and intermediary components to reduce interaction latency, enforce security, and encapsulate legacy systems.

The central feature that distinguishes the REST architectural style from other network-based styles is its emphasis on a uniform interface between components. By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved.

What makes HTTP significantly different from RPC is that the requests are directed to resources using a generic interface with standard semantics that can be interpreted by intermediaries almost as well as by the machines that originate services. The result is an application that allows for layers of transformation and indirection that are independent of the information origin, which is very useful for an Internet-scale, multi-organization, anarchically scalable information system. RPC mechanisms, in contrast, are defined in terms of language APIs, not network-based applications.

HTTP is not designed to be a transport protocol. It is a transfer protocol in which the messages reflect the semantics of the Web architecture by performing actions on resources through the transfer and manipulation of representations of those resources. It is possible to achieve a wide range of functionality using this very simple interface, but following the interface is required in order for HTTP semantics to remain visible to intermediaries.

Principles

as per Tilkov[2]:

Everything has its own URI as an identifier
Link things together
Use a set of standard methods (in the manner they're intended)
Resources have multiple representations
Assume stateless communication

Benefits from Principles

Identifiers:

flexibility, extensibility: bookmarkable, pass between different apps, facilitate new mashups; support versioning (version # is part of URI)
lowered dev costs, extensibility: familiar programming model (ala browser); apply web-centric security constraints; leverage HTTP redirects; apply different rules to different URIs for logging, statistics, auditing, etc.
ease of evolution: retrofit things like auditing, diagnostics, recovery, undo operations, ...

Linking:

scalability: paging via links in the face of many results
lowered dev costs: symmetric, consistent, understandable, maintainable, extensible codebase; familiar programming model (as browser). Clients can "discover" the entire information space dynamically, no need to hardwire drill-down URLs that will later break.
lowered dev costs, extensibility: guide "next valid transitions"; encapsulate URI details via rel (relations) attribute, no need for out-of-band document (WADL, WSDL)
ease of evolution: self-describing server can evolve without breaking clients

Standard Verbs ("Uniform Interface"):

scalability: can leverage (and not get burned by) existing web infrastructure (proxies, gateways, etc...crawlers, etc.)
reliability: GET and HEAD are safe; PUT and DELETE are idempotent - thus clients can resend requests as needed (except for POST - but see patterns below for workarounds). Cache intermediaries can "determine the cacheability of a response because the interface is generic rather than specific to each resource. By default, the response to a retrieval request is cacheable and the responses to other requests are non-cacheable." (Fielding, sec. 5.2.2)
lowered dev costs: no need to invent a new protocol for every application (ala WS).
extensibility: re-use of testing tools/techniques, interoperability between new apps and existing clients, etc.
value-add: facilitate intranet with searchable resources (i.e. crawlers can index GETs without deleting your database...).

Multiple Representations:

ease of evolution: support versioning for backwards compatibility (e.g. application-custom MIME types)
flexibility: client references are not coupled to a particular representation

Stateless:

scalability: facilitates load balancing, distributed caching, clustering, parallel processing and pipelining
reliability: failover
flexibility, extensibility: decoupled from clients
visibility: diagnostics are transparent since each request is self-contained

All of above used together

easier to combine different services (interop)
more consistent coding patterns, better redundancy, faster training/learning curves, faster evolution
symmetric, understandable, maintainable codebase
From Fielding, section 5.3.1: "(RESTful constraints) allow intermediaries - proxies, gateways, and firewalls - to be introduced at various points in the communication without changing the interfaces between (client and server), thus allowing them to assist in communication translation or improve performance via large-scale, shared caching. REST enables intermediate processing by constraining messages to be self-descriptive: interaction is stateless between requests, standard methods and media types are used to indicate semantics and exchange information, and responses explicitly indicate cacheability."

Patterns

Scalability

Caching - leverage validation, expiration, etc so response transfers data only when it's changed; server-side caching to minimize repeating expensive computations and/or to handle increased demand from multiple clients
Cache control - server specifies which responses are cacheable; client has option to re-use the cacheable data
Response code 409 - to leverage optimistic locking patterns
Asynch request pattern - POST a query that is costly on server side; client receives a "future" in Location header, later does a GET to that URI (404 means not done yet...)
Header information - specify what encoding is acceptable, server can compress e.g. into gzip to save bandwidth
Instead of sessions (impacts scalability), make the shopping cart a resource
Provide "collection" resources - coarse-grained interactions
Provide paging of large results - e.g. 20 at a time - with links on NEXT and PREVIOUS, etc.
- provide this link in a header to enable linking from non-text media types, e.g. an image[7]
- use this "header link" pattern for other linking needs, e.g. providing the "next valid state transitions" (i.e. what valid things can client do from here)
Transactional behavior
- make the txn a resource. GET txn, do stuff to it, finally PUT it at the very end
- use BASE and compensating txns (PUT, DELETE, POST) as needed
- server provides links that facilitate compensating actions
Conditional GET: response has ETag and/or Last-Modified set; client later requests same resource with header including Etag and/or Last-Modified value as value for If-None-Match and/or If-Modified-Since, respectively; then server decides if resend is needed ("validation"). If not, response code is 304.
If you must use cookies, store all app state on client side (i.e. don't use a session ID pointing to data on server) - else, you'll sacrifice scalability
Leverage intermediaries - using HTTP as intended in a RESTful style facilitates interoperation with network components that provide load balancing, caching, security policies, etc. As per Fielding: Within REST, intermediary components can actively transform the content of messages because the messages are self-descriptive and their semantics are visible to intermediaries.

Design/Coding

New URIs are created by server, returned to server via the Location header after a POST creates it
Version the service with URI - /v1/service/resource, or even as part of the host - v1.myservice.twc.com, v2.myservice.twc.com, etc.
Keep in mind that HTML5 will support PUT/DELETE. But not all firewalls allow these through...so, tunnel the method using header or hidden form field, or just use XHR
Provide canonical representations (text/plain, HTML) - supports easier debugging, scraping
Get around idempotence of POST:
- instead of retry when uncertain about success, do a PUT
- Post-Once-Exactly: to get around non-repeatable POSTs - GET returns a server-side link representing a resource not yet created, then client POSTs to that URL to create new resource; subsequent POSTs to the same resource URL return 405 (not allowed).

"Conditional PUT (POST)": before submitting a large amount of information to server that might not be able to handle it at the moment - PUT without resource but include Content-Length and Expect headers. If response has same code as Expect value, client proceeds, else it does not.
Use POST to support large queries, to get around length limits imposed by servers, clients and proxies. However this results in loss of cacheability if response is sent synchronously; instead, use the async request pattern from above - server returns new resource with 201 response code, client then GETs the answer to the request.
Content negotiation - client uses accept, accept-encoding, accept-language headers; vary and/or location headers in response. This supports "late binding" in determining content representation as function of request.
Evaluate extensibility vs visibility if considering "code on demand" (applets, javascript) - while this simplifies clients (extensible at runtime), it reduces visibility (maintenance, understandability, diagnostics, ...)

References:

[1] Fielding, Roy Thomas. Architectural Styles and the Design of Network-based Software Architectures. Doctoral dissertation, University of California, Irvine, 2000:http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
[2] Stefan Tilkov presentation (video): http://www.parleys.com/#st=5&id=1397
[3] Richardson, L. & Ruby, S. (2007) RESTful Web Services. Sebastopol, CA:O'Reilly Media, Inc.
[4] Allamaraju, S. (2010) RESTful Web Services Cookbook. Sebastopol, CA:O'Reilly Media, Inc.
[5] HATEOAS, the scary acronym: http://css.dzone.com/articles/hateoas-scary-acronym
[6] RESTful Web Services: http://imyousuf-tech.blogs.smartitengineering.com/2011/02/restful-web-services.html
[7] IETF RFC 5988 - Web Linking: http://tools.ietf.org/html/rfc5988

Create and Detect Thread Deadlocks

2011-10-27T15:37:00.004-06:00

Just the code:

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;
import java.util.Date;
import java.util.Scanner;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

/**
 * Example of how a deadlock might be created, and how to detect it. Two threads
 * are launched, each acquiring the same two resources but in a different order.
 * A delay is introduced in the first thread before acquiring the second
 * resource to set up a race condition such that a deadlock *may* be introduced.
 * The deadlock is not guaranteed to occur; it depends on the length of the
 * delay, which is controlled by the user executing the program. Empirically, on
 * my desktop, I've found that values of 100 sometimes are adequate, while
 * values of 1000+ become increasingly reliable.
 * 
 * Note that looking for deadlocks with the JMX mechanism used here (ThreadMXBean) is
 * described in the javadocs as an operation that "might be expensive". As such,
 * unless you're confident that the cost is not high...I'd recommend not putting
 * this into operation, rather using it in dev/test environment when trying to
 * reproduce a situation observed in production where a deadlock occurred. Note
 * that in production, issuing SIGQUIT (ctl-backslash) will provoke a thread
 * dump (without killing the process), in which information on deadlocked
 * threads will also appear.
 */
public class DeadlockedThreadFinder
{
   // done becomes true when either thread completes; note that if one thread 
   // completes, then the other one will also
   public boolean done;

   /**
    * Launch two threads, each acquiring a lock on two different resources in a
    * different order. The first thread pauses some time T as a function of the
    * given delay.
    *
    * @param delay is the given delay
    */
   public void createDeadlock(final int delay)
   {
      System.out.println("First thread will pause for time T as function of " +
         delay);

      // the resources in contention - if threads use the synchronized keyword, 
      // you could use something like the two Object declared (and commented 
      // out) here; else, use the ReentrantLock if in java.util.concurrent 
      // land.
      //      final Object resource1 = new Object();
      //      final Object resource2 = new Object();
      final Lock lock1 = new ReentrantLock();
      final Lock lock2 = new ReentrantLock();

      // first thread
      Thread t1 = new Thread()
      {
         public void run()
         {
            long id = this.getId();
            lock1.lock(); // alternately use synchronized (resource1) without 
            // the try-finally block
            try
            {
               System.out.println(id + ": obtained first lock");
               // instead of using Thread.sleep(), just create a whole bunch 
               // of expensive objects
               for (int i = 0; i < delay; i++)
               {
                  new Date();
               }

               System.out.println(id + " waiting for 2nd lock...");
               lock2.lock(); // or use synchronized (resource2)
               try
               {
                  System.out.println(id + ": obtained 2nd lock");
               }
               finally
               {
                  lock2.unlock();
               }
               System.out.println(id + " is done!");
               done = true;
            }
            finally
            {
               lock1.unlock();
            }
         }
      };

      // second thread
      Thread t2 = new Thread()
      {
         public void run()
         {
            long id = this.getId();
            lock2.lock(); // alternately use synchronized (resource2) without 
            // the try-finally block
            try
            {
               System.out.println(id + ": obtained 2nd lock");
               System.out.println(id + " waiting for first lock...");
               lock1.lock(); // or use synchronized (resource1)
               try
               {
                  System.out.println(id + ": obtained first lock");
               }
               finally
               {
                  lock1.unlock();
               }
               System.out.println(id + " is done!");
               done = true;
            }
            finally
            {
               lock2.unlock();
            }
         }
      };

      t1.start();
      t2.start();
   }

   /**
    * User must enter an integer that specifies the delay. Values of 10000+ are
    * likely to cause a deadlock. The larger the value entered, the more likely
    * the program will loop a few times detecting no deadlock, and then, once
    * the first thread has satisfied the delay, the program will detect a
    * deadlock until the end of time or user termination of program, whichever
    * comes first.
    */
   public static void main(String[] args) throws Exception
   {
      // the JMX deadlock finder
      ThreadMXBean mxbean = ManagementFactory.getThreadMXBean();

      // user specifies the delay
      System.out.print("Enter delay: ");
      Scanner sc = new Scanner(System.in);

      // launch the threads
      DeadlockedThreadFinder finder = new DeadlockedThreadFinder();
      finder.createDeadlock(sc.nextInt());

      System.out.println("Waiting for either thread to be done...");
      do
      {
         // the JMX deadlock finder will return null if no deadlock detected. 
         // Note that if the threads are using traditional synchronization (i.e. 
         // using the synchronized keyword), then you could also use the 
         // findMonitorDeadlockedThreads method here; but since the threads 
         // might uplevel to use java.util.concurrent, it's advised to instead
         // always use findDeadlockedThreads, which will work for either situation.
         long[] deadlocked = mxbean.findDeadlockedThreads();
         if (deadlocked != null)
         {
            System.out.print("Deadlocked threads: ");
            String comma = "";
            for (long l : deadlocked)
            {
               System.out.print(comma + l);
               comma = ",";
            }
            System.out.println();
         }
         else
         {
            System.out.println("No deadlock detected");
         }
         Thread.sleep(1000);
      }
      while (!finder.done);

      // uh, done
      System.out.println("Done");
   }
}

Create and Initialize a Utility Class with Spring

2011-03-08T15:16:00.001-07:00

Let's say we want to encapsulate a collection of read-only configuration parameters into a class (so, we want immutability). There are numerous parameters in the configuration, perhaps dozens - so we don't want to provide some ugly multi-parameter constructor or factory method. In fact, we want to provide clients with the convenience of static getters, so we know this will be a utility class (i.e. no instances needed or desired). While we could initialize a list of static fields with their configuration values to facilitate this, we would rather factor these value out into some external file.

We'd like to use Spring - but Spring works on instances, and we don't have a constructor for Spring to use, nor do we want or need an instance anyway, nor do we want to provide setters on this class (since we would like to stay immutable). So, how to use Spring?

One solution is to use a variation on the Builder pattern - the recipe goes like this:

create the utilty class with static fields for each configuration parameter
give it a private constructor
provide static getters for each parameter
add a static inner class with the same fields, although these will be instance fields (not static)
this inner class also has a private constructor
provide setters for each field - although here we deviate from the standard Builder pattern: these are true setters, returning void
the utility class now provides a static "init" method that accepts an instance of the inner class as an argument - and this behaves just like the regular Builder pattern, where the top-level class initializes its fields from the correspond fields in the argument object
the inner class provides its own "init" method which calls the utility class init method with itself

Spring wires all of this together like this:


    <bean id="builder-variant" class="com.mybiz.MyUtilityClass$MyInnerClass"
            init-method="initUtilityClass">
        <property name="foo" value="bar"/>
        <property name="bar" value="foo"/>
    </bean>

Using this, Spring will create an instance of the inner class, setting its properties as given, and then call the inner class' initializer. As mentioned in the above recipe, that initializer will then call the utility class' initializer with itself as the argument. That facilitates initializing the utility class properties, and we're done.

Integrate MyBatis 3.x with Spring 3.x

2011-02-17T13:39:00.000-07:00

Since last I wrote about integrating Spring with iBatis, and about using the very useful auto-rollback in Spring Test Framework, things have changed. The iBatis project has shifted to Google and is now called MyBatis, and includes numerous API changes. But, this was done after Spring 3.0 was released - so full integration with Spring was left undone at that time. However, a Spring-MyBatis integration component is now available, and here I'll write up how to use it to very simply use MyBatis, preserving the auto-rollback feature.

Get a Random Value Within a Given Range

2011-02-09T14:44:00.001-07:00

Just the code:


    public int getRandomValue(long min, long max) {
        return (int) Math.max(min, (int) (Math.random() * max) + 1);
    }

Use this to (obviously) generate an integer value between (inclusively) the given minimum and maximum - which is useful also to fetch random elements e.g. from a List.

Getting Around the Double-Checked Locking Problem

2011-02-08T12:30:00.002-07:00

This is just a summary of alternatives to the double-checked locking pattern, which has been shown to have potential problems. The alternatives are taken from a CERT wiki article. The motivations for the pattern come from three concerns:

avoid constructing an object that is expensive to construct, unless-until it is needed
avoid more than one instantiation of that object, and
avoid the overhead of synchronization for the more common case of fetching the object.

Getting Around the Double-Checked Locking Problem

2011-02-08T12:30:00.001-07:00

avoid constructing an object that is expensive to construct, unless-until it is needed
avoid more than one instantiation of that object, and
avoid the overhead of synchronization for the more common case of fetching the object.

Thread-Safe Singleton Using ThreadLocal

2011-02-08T10:27:00.001-07:00

Let's say a given component in your application has these constraints:

it needs a reference to an object that is expensive to construct
the expensive object state can be set just once, i.e. it never changes
it executes in a multi-threaded context
maximum throughput is required

Constraint #1 seems to call for an instance variable - i.e. you don't want to use a method-local field since the object construction is expensive, as such you don't want to do it each time the object is needed. Constraint #2 implies that a singleton will serve our needs. However, constraint #3 means the singleton must be used in a thread-safe manner - and constraint #4 tells us that we want to avoid synchronization if possible.

Get Notified When Your Periodic Task Throws an Exception

2011-02-02T17:42:00.000-07:00

I've written up a how-to on a periodic task with a watchdog; now I want to handle exceptions and notify interested observers when exceptions occur. The watchdog is in place in case exceptions don't get handled at all, in which case subsequent executions of the scheduled thread will be suppressed. in this case, the watchdog knows that some problem occurred - but it doesn't know just what that was. As an alternative (or in addition) to the watchdog, we simply catch exceptions in the Runnable passed to our Executor, handling it by both logging the information and notifying interested listeners. This helps pass along exception information from a different stack of execution, since the problem occurs in a different thread:

Use Spring to Wire Up Varargs

2011-02-02T17:18:00.000-07:00

This describes a technique to provide N objects to a constructor using Spring. Nothing real new or innovative here; just an idiom I wanted to save for future reference. Let's say we have a constructor that uses varargs:


    public MockDao(SummaryData...summaries) {
        // save the summary data for downstream use...
    }

And, we want to use Spring to provide test summary data to our mocked-out DAO.

Encapsulate SSL TrustStore Configuration

2011-02-02T13:27:00.001-07:00

Here I'll present the steps of a recent exercise I completed, around configuring SSL on the client side, encapsulating the configuration for transparency. For some background on SSL and the topics discussed in this article, here are some resources:

SSL Protocol Overview
SSL Certificates HOWTO
Transport Layer Security

The Security Now podcast series also has some excellent deep-dives into SSL and many other security-related topics.

Enable Assertions with JUnit Tests in IDEA

2011-02-02T10:18:00.000-07:00

My test cases have recently been converted from TestNG to JUnit4, and to my surprise every single test I wrote from that point forward passed without a problem - if I ran the tests in my Intellij IDEA 9.0.4 IDE. Mysteriously, executing them via Maven did not have that same result (whether from command line or from IDEA). As it turns out - apparently JDK assertions are enabled by default when I use TestNG in IDEA, but not so much when using JUnit.

Fix this problem by setting the -ea VM argument for all existing tests, and setting that as a default for JUnit tests created from this point forward.

Arrrggghhhh.

Basic Authentication with Apache HttpClient

2011-01-28T11:41:00.000-07:00

This is the general idiom used for BASIC authentication with Apache's HttpClient, version 3.1.

Periodic Task and Watchdog - Reusable and Testable Versions

2011-01-27T13:31:00.000-07:00

In my previous post, I presented a simple utility for a periodically executing task and a watchdog to make sure it runs forever. Now I'll refactor that prototype code to make things generic and reusable.

Schedule a Periodic Task with a Watchdog

2011-01-27T11:36:00.000-07:00

This demonstrates a simple mechanism to start up a thread that executes some task periodically, with a watchdog to ensure it keeps running. It is assumed that the periodic task involves some kind of I/O or remote communication that can result in an exception. The watchdog is needed because the java.util.concurrent class used for the task will not proceed with subsequent executions after one of its executions encounters an exception.

Test Your CXF REST Service by Examining Application Object State

2011-01-14T15:14:00.000-07:00

While I've previously described using CXF to unit test a REST service, it's a limited technique since the object returned from the HTTP request is a response string - and any assertions you could make against that string are going to be brittle, since it relies on the decisions around how the encoding (XML or JSON) is done. What we really want is a more object-oriented mechanism for testing against application object state...which means you need a REST client that calls the service and can then look at the returned application object, as opposed to a Response object.

Portfolio Entry #1: Massage Business in Erie, Colorado

2011-01-10T09:31:00.002-07:00

Tempo Software and Consulting (my LLC) happily announces the publishing of its first website, for my client running a massage therapy business in Erie, Colorado. Please have a look - comments and feedback are invited.

And if in fact you live in the Boulder County area, Aimee is a very gifted therapist who has taken care of my aches and pains for many years. Should you be looking for a massage therapist, I would very sincerely recommend her.

Full disclosure: Aimee is my wife, but I submit that my endorsement remains objective - since she was my massage therapist of choice for many years before we even started dating. And, yes, I billed her at a discounted rate.

Her business is Hands On Harmony Massage. Enjoy.

Versioning XML and JSON Models in a CXF REST Service

2011-01-06T17:01:00.000-07:00

Following up on my previous post around how simple it was to add JSON support to my CXF-based REST service...now I'll add versioning support to my XML-annotated model and see what happens.

Add JSON Support to your REST Service

2011-01-04T15:47:00.000-07:00

Continuing from my previous post about a standalone unit-test mechanism for your RESTful service, now I'll add JSON support to that service, which already gives you XML. This turns out to be a pretty simple thing - all we need is:

Tell the RESTful service to produce both XML and JSON
Add the runtime dependency you'll need
Add some tests to confirm things

Unit Test Your REST Service

2011-01-04T15:19:00.001-07:00

Here's an approach for unit-testing your RESTful service.

Populate Immutable Objects with iBatis Mapping

2010-12-16T10:33:00.001-07:00

In a previous post, I provided iBatis configuration and mapping that supported populating Java objects on a per-property basis, i.e. the classes in use had to provide setters for this approach. Here I describe another iBatis pattern that populates immutable objects with more than a few properties.

Notes on the Builder Pattern

2010-12-16T09:49:00.000-07:00

This post simply describes the Builder Pattern, for my own future reference. This pattern is useful in constructing immutable objects that have numerous properties, some required and some optional, without using numerous constructors to support various permutations. And, of course, since the object is immutable, there are no setters or non-private fields to support post-construction alterations. With the Builder pattern, one can ensure objects are not only immutable, but have completed state initialization before use.

Running an Effective Agile Retrospective

2010-11-17T14:40:00.003-07:00

This is a summary of a presentation given by Brad Swanson of Propero Solutions on November 9, 2010, with permission from Brad. The topic was around effectively leading an agile retrospective meeting.

Features Gained with SOAP UI Pro Version

2010-11-03T14:11:00.001-06:00

This documents differences between the free and Pro version of SOAP UI, as determined solely from browsing the SOAP UI website. This might be useful in determining if the free 14-day trial of the Pro version is worth your time.

Specify Database Connection via Maven Resource Filtering

2010-10-25T13:46:00.004-06:00

This is an example usage of Maven's resource filtering facility. While Paolo Predonzani's excellent article got me pointed in the right direction, it didn't quite work out for me. Here is what I did to get what I needed.

Basic GWT-RPC Application

2010-10-01T15:12:00.002-06:00

After stepping through this introduction to a simple GWT-RPC example, I had to experiment a bit to (1) get it running and (2) get some clarity around the naming hooks. What I'll present here is my own variation on that article's code examples, and the understanding I gained around what to keep straight around the names of the various classes and files.

Tip on Specifying GWT-RPC Remote Service Relative Path

2010-10-01T13:43:00.001-06:00

Robert Hanson's excellent introduction to a GWT-RPC application got me started, but I bumped into this:


com.google.gwt.user.client.rpc.StatusCodeException:
HTTP ERROR: 404

NOT_FOUND

RequestURI=/relative-path

iBatis Mapping for One-to-Many Relationship

2010-08-26T15:04:00.001-06:00

Perhaps you had a problem, like I did, understanding the iBatis writeup on loading complex collection properties, i.e. populating a list-based field in a domain object where that list is the many-side of a one-to-many relationship. I'll admit it took me a few iterations to totally grasp what was being said, so for my future reference and yours, I'll write down my notes here.

Monitor Changes to a File

2010-08-23T09:58:00.001-06:00

There are undoubtedly many options out there for file-system change monitors - i.e. a component that will notify your Java object when a given file or directory has changed, among many others I'm sure: jnotify and jpathwatch, the latter of which is based on upcoming Java 7 NIO enhancements.

I gave jpathwatch a try recently, at a time when I was under a tight deadline and didn't want to reinvent a wheel. This presents a perfectly fine API that worked out quite well and quite quickly for me in my Windows environment, but alas when I deployed to Linux, I bumped into an unsatisfied link error. The problem was around libc.so.6 and GLIBC_2.4, and I gave it a reasonable first effort to try quickly finding the resolution - assuming I'd deployed incorrectly, or my Linux box was out of date, etc. It was neither of these - OK, in fairness, it could be an out-of-date Linux box, but my experiment was to just try it out on our customers' target system - and the same problem occurred. So out-of-date becomes a moot point.

As I mentioned, I was under a tight deadline, so I began some quick prototyping to see if I could reinvent something but without relying on native libraries (as jpathwatch did). That would give me the added advantage of a smaller runtime footprint, which was another customer requirement. Since we are really talking about an Observer pattern, here's how I started:

public interface FileChangeListener {

    void fileModified(String file);

    void fileDeleted(String file);

    void fileCreated(String file);
}

That specifies the observer. Here's a simple monitor interface:

public interface AbstractFileChangeMonitor {

    void watch(FileChangeListener listener, final String filename);
}

This one is a bit limited, since it supports just one listener (observer) for one file. But my goal is not (yet) to provide a full-featured framework - I just need to knock out the problem at hand without any gold-plating. I'm a firm believer in doing the least I have to for a given problem - first make it work, then make it fast, then extend it, ... etc., but only if subsequent steps are called for. In either event, here's an implementation of the monitor - this one polls the file in question to detect changes, running in a thread so the client process isn't blocked:

public class PollingFileChangeMonitor implements AbstractFileChangeMonitor
{
    private final Logger logger = LoggerFactory.getLogger(getClass().getSimpleName());
    private FileChangeListener listener;
    private boolean done;
    private Thread watch;
    private int pollingInterval;

    public PollingFileChangeMonitor(int interval) {
        pollingInterval = interval;
    }

    public void watch(FileChangeListener theListener, final String filename)
    {
        if (watch == null) {
            listener = theListener;
            watch = new Thread() {
                public void run() {
                    try  {
                        init(filename);
                    } catch (Exception e) {
                        throw new IllegalStateException(e);
                    }
                 }
             };
            watch.start();
        }
    }

    private void init(String filename) throws Exception
    {
        boolean exists = false;
        long modTime = -1;
        File file = new File(filename);
        if (file.exists())
        {
            exists = true;
            modTime = file.lastModified();
            logger.info("====> File '" + filename + "' exists; change monitor is running...");
        } else
        {
            logger.info("====> File '" + filename + "' does NOT exist; change monitor is running...");
        }

        while (!done) {
            try {
                watch.sleep(pollingInterval);
            } catch (InterruptedException e) {
                // ignore for now
            }

            if (!exists && file.exists()) {
                exists = true;
                logger.info("====> File '" + filename + "' has been created; notify listener...");
                listener.fileCreated(filename);
            } else if (exists && !file.exists())  {
                exists = false;
                logger.info("====> File '" + filename + "' has been deleted; notify listener...");
                listener.fileDeleted(filename);
            } else if (exists && file.exists()) {
                long timestamp = file.lastModified();
                if (timestamp > modTime) {
                    modTime = timestamp;
                    logger.info("====> File '" + filename + "' has been modified; notify listener...");
                    listener.fileModified(filename);
                }
            }
        }
    }
}

What the listener does when notified is not really important in the context of this post; it can be anything. While I've to a certain extent "reinvented" something here, I've gotten away from reliance on native code and the potential deployment headaches around that, I've reduced my runtime footprint, and for that matter I've solved the problem with about the same amount of code I needed for the boiler-plate suggested by jpathwatch.

Handling Partially Failed Transactions

2010-08-23T09:58:00.000-06:00

Requirements for a web application I've developed include submitting a form that results in many database operations, including a mix-and-match of updates, inserts and deletes. Since there can be hundreds of such operations, my initial approach was to execute them in a single batch for a reasonable response time. Now, if what you want is fast execution and truly an all-or-nothing transactional behavior, using batch execution is a good idea. However, on iteration with my customers, the requirements evolved into something a bit more sophisticated: if any operations in the submitted set fails - e.g. attempting to insert a duplicate row with a unique index - we want all other operations to succeed, and we want to identify the operations that did not succeed.

What's I've described of course is not strictly "transactional" - i.e. the all-or-nothing constraint is not in play, but for the application's needs, this is perfectly acceptable. And since what I describe involves removing the batch execution, the overall set of operations will take longer to execute - but again, this is acceptable for the application's goals. You won't always find these trade-offs to be acceptable, but when you do, I'll describe one mechanism for making it work - my strategy includes the following:

1 - Execute the collection of operations in a single transaction, albeit without batch
2 - Catch and deal with exceptions around operations that fail
3 - Provide return values that includes lists of successes and failures

Here are some snippets that provides these things. I start with an enumeration of the operation types I care about:

    private enum OP {add, update, delete}

Next I provide a result object that can be examined by the client to identify successes and failures. I use generics for type-safety - the Key and Value types represent your application-specific database key and object classes:

public class OperationResult {
    private Map<Key, Value> successes = new HashMap();
    private Map<Key, Exception> failures = new HashMap();

    public OperationResult(Map<Key, Value> succeeded, Map<Key, Value> failed) {
        successes.putAll(succeeded != null? succeeded : new HashMap());
        failures.putAll(failed != null ? failed : new HashMap());
    }

    public Map<Key, Value> getSuccesses() {
        return successes;
    }

    public Map<Key, Exception> getFailures() {
        return failures;
    }
}

Finally I provide the workhorse method that executes all operations, capturing exception details in the face of any failures, and proceeding in either event so it can commit as many successes as possible. Due to the way this API is set up, clients can execute only one operation type at a time:

    private OperationResult execute(Map<Key, Value> map, String query, OP operation) {
        Map<Key, Value> successes = new HashMap();
        Map<Key, Exception> failures = new HashMap();
        try {
            try {
                sqlMapClient.startTransaction();
            } catch (SQLException e) {
                throw new IllegalStateException("None of the operations succeeded - txn could not be started",
                    e);
            }

            for (Map.Entry<Key, Value> entry : map.entrySet()) {
                try {
                    switch (operation) {
                        case add:
                            return sqlMapClient.add(query, entry.getValue()); break;
                        case update:
                            return sqlMapClient.update(query, entry.getValue()); break;
                        case delete:
                            return sqlMapClient.delete(query, entry.getKey()); break;
                    }
                    successes.put(entry.getKey(), entry.getValue());
                } catch (Exception e) {
                    failures.put(entry.getKey(), e);
                }
            }

            try {
                sqlMapClient.commitTransaction();
            } catch (SQLException e) {
                throw new IllegalStateException("None of the operations succeeded - txn could not be committed",
                    e);
            }
        } finally {
            try {
                sqlMapClient.endTransaction();
            } catch (SQLException e) {
                throw new IllegalStateException("Problem ending the txn - you should check whether the operations succeeded or not",
                    e);
            }

            return new OperationResult(successes, failures);
        }
    }

I'm using iBatis 2.x, as you might be able to guess from the SQL Map Client reference; it is assumed that the client passes in the correct query ID for the given operation.

One guideline from my Computer Science education that always stuck with me is "make it right first, then make it fast". The solution provided here is, as noted, not going to be as fast as batch execution - but it does meet the requirements correctly. If there is still a need to make it "faster", we could consider using an asynchronous API so that the response time from the user's point of view is improved. This type of approach is in fact used in many enterprise settings, leveraging so called BASE behavior instead of the traditional ACID approach.

Open a PDF Resource in Browser Tab

2010-08-10T13:34:00.001-06:00

Though the title refers to a PDF, the MIME type of the resource is not crucial to this. My goal was to open a PDF file in a new browser tab, and with IceFaces I initially tried the outputResource widget. This seemed like the right way to go, and according to the TLD I should have been able to open the resource directly in the browser as opposed to simply downloading it - but I could not get that to work as expected.

Instead, I backed off to a more straightforward link approach, i.e. use of the outputLink control, as follows:

<ice:form>
    <ice:outputLink value="doc/some-PDF-file.pdf"
                    target="_blank">
        <ice:outputText value="Open PDF"/>
    </ice:outputLink>
</ice:form>

The plain vanilla HTML is dirt-simple:

<a target="_blank" href="doc/some-PDF-file.pdf">Open PDF</a>

This assumes the "doc" directory location is under your webapp root context.

SELECT Menu Doesn't Grey-Out When Disabled in Firefox 3.6.8

2010-08-10T13:19:00.000-06:00

The title of this post pretty much says it all, just in case you've been Google'ing it to find some kind of reference to this. Since I found nothing, and since I solved my problem by manually changing the background-color on my SELECT item, I write it up here.

The problem does not occur in Firefox 3.5.11, and only occurs in 3.6.8 when you specify the SIZE attribute in a SELECT menu with MULTIPLE selectable items. Leave off the SIZE attribute, and the desired greying-out happens just fine, albeit with a 20-item window into your menu items. If that's not acceptable, you can use some JavaScript to convince the menu to grey out and then come back to normal as you disable and enable it. For example, given some HTML that defines some menu items, which should be disabled when a given checkbox is selected:

<input type="checkbox" name="items" onclick="enableDisable(this, formname)" .../>
....
<select id="menu" multiple size="4">
....
</select>

...here's the JS that can be used:

function enableDisable(input, formName) {
    if (input.checked) {
        disable(formName);
    } else {
        enable(formName);
    }
}
function disable(theForm) {   
    if (theForm.menu != null) {
        theForm.menu.disabled = true;
        // address the Firefox problem - grey it out manually:
        theForm.menu.style.backgroundColor="#DBDBDB";
    }
}
function enable(theForm)
{
    if (document.forms[theForm].menu != null) {
        document.forms[theForm].menu.disabled = false;
        // address the Firefox problem - reset to normal color, manually:
        document.forms[theForm].menu.style.backgroundColor="#FFFFFF";
    }
}

Disclaimer: As is typical with my code snippet posts, I've extracted the relevant statements from my production code, manually editing field names, method names, etc. to protect my client's IP. Since I've not taken the time to confirm the extracted code (I'm lazy, I'm too busy, it's left as a reader exercise, etc.), I can't guarantee that your copy-paste exercise will "just work" - but I do believe I've given you the basic building blocks to make it work for your application.

Get Full Filepath from IceFaces Webapp

2010-08-10T12:48:00.000-06:00

Just a quick little snippet - if you need the "real path" for a resource residing within your webapp directory tree, here's how I've done it in IceFaces:

        FacesContext fc = FacesContext.getCurrentInstance();
        ExternalContext ec = fc.getExternalContext();
        ServletContext servletCtx = (ServletContext)ec.getContext();
        String filePath = servletCtx.getRealPath("/directory-under-webapp-root/somefile.pdf");

In principle the same idea applies regardless of IceFaces or any other web framework - you just need to get the ServletContext. That said, it appears that J2EE servers are not required to return anything beyond null from the getRealPath method.

My setting is with an exploded WAR - but if your webapp remains unexploded, here are some tips to get around that.

Log SQL Statements with iBatis

2010-07-28T11:54:00.001-06:00

Here's what I've done to enable logging of SQL statements in an iBatis context. This adds to the various bits of advice I've found on the web.

Perhaps it's due to the environment I'm using (Java 1.6, iBatis 2.3.4, SLF4J 1.5.11, Log4J 1.2.15), but the guidance offered in the iBatis Dev Guide is, in my experience, not quite right. In fairness, something must have changed since last I tried that advice (was I using Java 1.5 back then? I'm not sure), since, as I recall, it resulted in the SQL logging I needed. However, at the moment I find that simply configuring this:

<logger name="java.sql.PreparedStatement">
    <level value="debug"/>
</logger>

...yields no SQL output at all.

On target is the advice given in the Apache Bookstore, i.e. to specify this:

<logger name="java.sql">
    <level value="debug"/>
</logger>

This yields all of connection information, prepared statements, values inserted as prepared statement parameters and result sets, something like this:

Connection
{conn-100008} Preparing Statement:  select name, description  from mytable where name like ?      
{pstm-100009} Executing Statement:  select name, description  from mytable  where name like ?               
{pstm-100009} Parameters: [name%]
{pstm-100009} Types: []
{rset-100010} ResultSet
{rset-100010} Header: [NAME, DESCRIPTION]
{rset-100010} Result: [name2, Desc for name2]
{rset-100010} Result: [name3, Desc for name3]

You'll possibly get even more output, depending on your app - since the java.sql namespace is more than just these things. If this is a concern, use the following:

    <logger name="java.sql.Connection">
        <level value="debug"/>
    </logger>

    <logger name="java.sql.PreparedStatement">
        <level value="debug"/>
    </logger>

    <logger name="java.sql.ResultSet">
        <level value="debug"/>
    </logger>

I've personally found that the output from PreparedStatement and ResultSet are the most useful - but omitting Connection appears to preclude these log statements from appearing. The Connection logger also appears to control the output level of PreparedStatement and ResultSet - i.e., setting Connection to INFO results in zero output from any of them. I begin to suspect that Connection has a parent relationship to the other two; perhaps this is what's changed since last I did this.

In either event, here are some SQL-interceptor tools recommended by various posts (though I've yet to try either one):

p6spy: http://www.p6spy.com/
jdbcdslog: http://code.google.com/p/jdbcdslog/

And here are all the web references I used for this post:

Stack Overflow:
http://stackoverflow.com/questions/3014318/how-to-do-ibatis-version-of-show-sql
http://stackoverflow.com/questions/2635058/ibatis-get-executed-sql
Apache Bookstore: http://www.apachebookstore.com/confluence/oss/display/IBATIS/How+do+I+get+SqlMapClient+to+log+SQL+statements
iBatis Dev Guide: http://ibatisnet.sourceforge.net/DevGuide.html#d0e2600

Generic Conversion Between List and String

2010-07-27T10:53:00.001-06:00

Here's a reusable generic "converter" interface for converting between comma-separated strings and Lists of a given type:

/**
 * Converts between comma-separated string and lists of type T.
 */
public abstract class AbstractConverter<T> {

    protected abstract T getInstance(String s);

    /**
     * Converts a list of items into a comma-separated string
     */
    public String toString(List<T> items)
    {
        String s= "";
        String comma = "";
        for (T item : items) {
            s += comma + item;
            comma = ",";
        }
        return s;
    }

    /**
     * Converts a comma-separated string into a list of items.
     */
    public List<T> toList(String csv)
    {
        if (csv == null)
        {
            return new ArrayList<T>();
        }

        String[] items = csv.split(",");
        List<T> theList = new ArrayList<T>();
        for (String item : items) {
            T thisItem = getInstance(item); 
            theList.add(thisItem);
        }
        return theList;
    }

    /**
     * Converts a comma-separated string into a list of items, excluding any items in the given
     * exclude list.
     */
    public List<T> toList(String csv, List<T> excludes)
    {
        if (csv == null)
        {
            return new ArrayList<T>();
        }

        String[] items = csv.split(",");
        List<T> theList = new ArrayList();
        for (String item : items) {
            T thisItem = getInstance(item);
            if (!excludes.contains(thisItem))
            {
                theList.add(thisItem);
            }
        }
        return theList;
    }
}

I can now provide concrete converter classes of types as needed, for example:

/**
 * Converts between comma-separated string and lists of MyType objects.
 */
public class MyConverter extends AbstractConverter<MyType> {

    // provide an instance of the custom type
    protected MyType getInstance(String s)
    {
        return new MyType(s);
    }

    // optionally enforce a singleton, if needed
    private MyConverter() {
        // EMPTY
    }

    // provide a convenience for static utility classes
    private static AbstractConverter<MyType> cvt = new MyConverter();
    public static AbstractConverter<MyType> inst()
    {
        return cvt;
    }
}

Since I want to make things convenient for clients of this, I now provide a static utility class that precludes the need to instantiate anything:

public class MyHelper {

    private static AbstractConverter cvt = MyConverter.inst();

    public static String toString(List<MyType> items)
    {
        return cvt.toString(items);
    }

    public static List<MyType> toList(String csv)
    {
        return cvt.toList(csv);
    }

    public static List<MyType> toList(String csv, List<MyType> excludes)
    {
        return cvt.toList(csv, excludes);
    }
}

Now my clients can call e.g. MyHelper.toList("x, y, z") instead of MyConverter.inst().toList("x, y, z"). A small advantage, but it does help with readability and reinforces the notion that this is a collection of utility functions.

When Log4j Logs Messages Twice...

2010-07-23T10:03:00.003-06:00

Since my Google'ing only gave me a clue instead of the complete answer on this one, I post it here for future reference.

When a given logger in a log4j configuration is outputting messages twice to your appender, you might try adding the additivity attribute as a fix:

<logger name="myLogger" additivity="false">
    <level value="info"/>
    <appender-ref ref="myFile"/>
</logger>

Same idea for logger or for category. From the DTD (I'm using version 1.2.15):

<!ELEMENT category (param*,(priority|level)?,appender-ref*)>
<!ATTLIST category
  class         CDATA   #IMPLIED
  name  CDATA #REQUIRED
  additivity (true|false) "true"  
>

<!ELEMENT logger (level?,appender-ref*)>
<!ATTLIST logger
  name  CDATA #REQUIRED
  additivity (true|false) "true"  
>

This worked for me. As you can see, the additivity defaults to true. There may be other fixes, but I didn't dig any further after fixing my problem.

Determine Which Row is Selected in a JSF Table

2010-07-22T14:26:00.000-06:00

Let's say you have a JSF table with numerous rows, and that when one of them is "selected" you'd like to know which one it is. Let's assume a JSF event is fired off when you click on the row, resulting in a callback to your application. In this callback, you want to determine the identity of the row in a way that correlates to your application, such that you can proceed with whatever action is needed. Here's the sequence:

Table is constructed with rows representing application objects A, B and C
User clicks somewhere in row B, e.g. in an input text box
JSF event fires and your application is called back
The application determines that B has been selected and processes that application object as needed

Now the obvious way to "determine that B has been selected" would be to simply set the ID attribute for each row to an identifying value at table construction time. But, using such variables for the ID attribute is not allowed in JSF. Don't ask me why. Here's one way to get around this:

Add a parameter to the UIComponent with the identity as needed
Register an interest in the JSF event that will get fired when the UIComponent is selected
Examine the event received in your application callback to find the UIParameter child
Examine the UIParameter to extract the identity of the row

The code snippets, first for the JSF markup addressing step #1 and #2. Note that I'm using IceFaces, setting the per-row variable for use in each UIComponent comprising each row:

<ice:dataTable 
     var="thisRow"
     value="#{app.allRows}"
    ....
>
....
    <ice:inputText valueChangeListener="#{app.callback}">
        <f:param name="name" value="#{thisRow.id}"/>
    </ice:inputText>
....
</ice:dataTable>

Here are steps #3 and #4, the application callback:

public void callback(ValueChangeEvent event) { 

    for (UIComponent child : event.getComponent().getChildren()) {
        if (child instanceof UIParameter) {
            UIParameter param = (UIParameter) child;
            if ("name".equals(param.getName())) {
                String thisID = (String)param.getValue();
                if (thisID.equals(theIDofInterest)) {
                    // process as needed
                }
            }
        }
    }

Using JSF Conversion with Custom Objects

2010-07-20T17:25:00.000-06:00

My goal is to provide a user-facing input text box that accepts a comma-separated list of values, and converts these to an application-specific object. Since I'm using JSF, I'll take advantage of their built-in conversion facility - but I want to remain as decoupled from JSF as possible. This means I'll implement their Converter class, but I'll do so only with very minimal high-level business logic (as opposed to emulating their example usage in the JEE Tutorial example). That way, I can reuse my conversion logic in other frameworks, for other clients, etc.

My application object is a collection of name objects. The name objects look something like this:

public class MyName {   

    private String name;

    public MyName(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public boolean equals(Object obj) {
        if (null == obj) {
            return true;
        }

        if ((obj == null) || (getClass() != obj.getClass())) {
            return false;
        }

        MyName other = (MyName) obj;
        return new EqualsBuilder().append(this.name, other.name).isEquals();
    }

    public int hashCode() {
        return new HashCodeBuilder(3, 13).append(this.name).toHashCode();
    }

    public String toString() {
        return name;
    }
}

Note that I've implemented toString as well as the equals and hashCode methods. The builder classes are provided by Apache Commons Lang; I've blogged about them here.

The collection object uses the Java List to wrap the name objects:

public class MyNameList {

    private List<MyName> names = new ArrayList<MyName>();

    public MyNameList (List<MyName> names) {
        setNames(names);
    }

    public static MyNameList getInstance(String csv) {
        if (csv == null) {
            return new MyNameList();
        }
        String[] names = csv.split(",");
        List<MyName> nameList = new ArrayList();
        for (String name : names) {
            nameList.add(new MyName(name));
        }
        return new MyNameList(nameList );
    }

    public List<MyName> getNames() {
        return names;
    }

    public void setNames(List<MyName> names) {
        this.names = names;
    }

    public String toString() {
        StringBuffer sb = new StringBuffer();
        String comma = "";
        for (MyName name: names) {
            sb.append(comma).append(name);
            comma = ",";
        }
        return sb.toString();
    }
}

Here's what the JSF converter implementation might look like - there's not much there, as planned:

public class MyNameListConverter implements Converter {

    public Object getAsObject(FacesContext context,
                              UIComponent component, String newValue)
            throws ConverterException {

        return MyNameList.getInstance(newValue);
    }

    public String getAsString(FacesContext context,
                              UIComponent component, Object value)
            throws ConverterException {

        return value == null? "" : value.toString();
    }
}

I must register the converter with JSF:

<converter>
    <description>
        Converter for CSV list of name values
    </description>
    <converter-id>MyNameListConverter</converter-id>
    <converter-class>
        com.mybiz.MyNameListConverter
    </converter-class>
</converter>

Finally I reference the converter in my JSF page:

 <ice:inputText id="nameValues" partialSubmit="true"
     converter="MyNameListConverter"
     value="#{bean.nameList}"/>

Lots of moving parts are needed when working with JSF. But, the more I can encapsulate, the less it will cost to migrate to a different web framework down the road.

What Database Am I Connecting To?

2010-07-19T13:55:00.000-06:00

If you run your functional tests in different environments (your local box, the nightly build, etc.), in addition to running your components in a production setting, you might want to get feedback at startup time around just what database you're connecting to. Since I was recently bit by nightly tests that were wiping out my development-local database tables - since I'd hardwired the connection URL - I was motivated to put this kind of feedback in place. Here are some details around how I did this.

Our nightly build has certain environment variables set:


dbip=BuildHost
dbinstance=TestDatabase
dbuser=username
dbpass=password

To simulate this environment, I set these same variables in my shell (Unix, Cygwin, ...), changing the values as needed to facilitate connecting to my local database. Next I set up a property configuration in my Spring startup file that uses these variables, replacing things per the environment:

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="properties">
            <value>
                database.driver=oracle.jdbc.OracleDriver
                database.url=jdbc:oracle:thin:@//${dbip}:1521/${dbinstance}
                database.user=${dbuser}
                database.password=${dbpass}
            </value>
        </property>
    </bean>

These Spring-generated properties are used to create a data source, which is used to configure an iBatis SQLMapClient, which I then use in my DAO:

    <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
          destroy-method="close">
        <property name="driverClassName" value="${database.driver}"/>
        <property name="url" value="${database.url}"/>
        <property name="username" value="${database.user}"/>
        <property name="password" value="${database.password}"/>
    </bean>
....
    <bean id="sqlMapClient" class="org.springframework.orm.ibatis.SqlMapClientFactoryBean">
        <property name="configLocation" value="classpath:SqlMapConfig.xml"/>
        <property name="dataSource" ref="dataSource"/>
    </bean>
....
    <bean id="ibatisDao" class="com.mybiz.resource.db.ibatis.IbatisDao">
        <property name="sqlMapClient" ref="sqlMapClient"/>
    </bean>

The DAO implements ApplicationContextAware so that it can examine the Spring context, in particular the properties of interest in the data source:

public class MyDao implements ApplicationContextAware
{
    private ApplicationContext context;

    public void setApplicationContext(ApplicationContext ctx)
    {
        context = ctx;
        dumpDataSource(ctx);
    }

    public void dumpDataSource(ApplicationContext ctx)
    {
        BasicDataSource dataSource =  ctx.getBean(BasicDataSource.class);
        logger.info("============>>> Data Source: " + dataSource.getUrl()
                + " (user " + dataSource.getUsername() + ")");
    }
}

Finally, I get myself into the habit of looking for that log output at the start of my test runs to be sure I'm connecting to the expected database. Once burned, twice shy.

JEE Authentication: Login Errors, Roles, Access Denied, and Logout

2010-07-16T14:16:00.001-06:00

In a previous post around LDAP authentication using Jetty, I had some unfinished business. Here I'll deal with login errors, restrict the login to a given role, deal with subsequent access denied scenarios, display the currently logged-in user name, and provide log-out functionality.

The web.xml I'd started with already specified a login-error page, but I was simply pointing it to the same page as the login form. This results in that page simply refreshing without any indication to the user of why that happened. What we want is some kind of message displayed that indicates the given username or password was not valid.

Now, I could simply copy the login.jsp to another JSP, say login-error.jsp, add a message to that page and alter the web.xml to specify that new page on login error:

<login-config>
    <auth-method>FORM</auth-method>
    <realm-name>ldap</realm-name>
    <form-login-config>
        <form-login-page>/faces/login/login.jsp</form-login-page>
        <form-error-page>/faces/login/login-error.jsp</form-error-page>
    </form-login-config>
</login-config>

But now both login.jsp and login-error.jsp contain the same FORM snippet. Now if I next want to specify yet another login page to which the user is directed on an "access denied" error (which we'll deal with momentarily), and I'm still afflicted with copy-paste fever, I'll have the same login form in three places. Let's factor it out instead into a JSP snippet named loginform.jsp:

<form method=post action="j_security_check">
    <label for="j_username" style="font-weight:bold">Username</label>
    <input type="text" name="j_username" id="j_username" style="margin-left:10px"/>
    <label for="j_password" style="font-weight:bold">Password</label>
    <input type="password" name="j_password" id="j_password" style="margin-left:10px"/>
    <input type="submit" value="Log In"/>
</form>

The usual login page now looks like this:

<div style="margin-top:25px; margin-left:25%">
    <h2 style="text-decoration:underline; color:blue;margin-left:-10px">Management App</h2>
    <h3>Please Log In</h3>
    <%@include file="loginform.jsp"%>
</div>

And I'll provide an "access denied" page that looks like this:

<div style="margin-top:25px; margin-left:25%">
    <h2 style="text-decoration:underline; color:blue;margin-left:-10px">Management App</h2>
    <h3>Please Log In</h3>
    <div style="color:red;font-weight:bold;">Authentication failed. User is not in Required Role.</div>
    <%@include file="loginform.jsp"%>
</div>

Configuring HTTP-403 responses (i.e. access denied) to navigate to this page is done like so:

<error-page>
    <error-code>403</error-code>
    <location>/login/accessDenied.jsp</location>
</error-page>

Access denied problems will occur if a given user is not in the expected role. So far, the web.xml has granted authorization to all roles by virtue of the wild-card for the role-name. We can restrict that by naming a role instead:

<security-constraint>
    <web-resource-collection>
        <web-resource-name>Protected Resources</web-resource-name>
        <url-pattern>*.iface</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>admin</role-name>
    </auth-constraint>
    <user-data-constraint>
        <transport-guarantee>
            CONFIDENTIAL
        </transport-guarantee>
    </user-data-constraint>
</security-constraint>

Now, once a user provides his/her credentials at the login form, these are first checked by the LDAP module (configured as a Jetty realm, as per the previous post); if those are valid, the user is next confirmed to be assigned the admin role. If that is the case, all is well and navigation will proceed as configured by the JSF navigation rule (again, please see the previous post). If the credentials are not valid, the user will be redirected to the login-error page, this time with an informative error message about the login problem. If the credentials are good but the user is not assigned the admin role, the user will be redirected to the access-denied page, again with a informative message.

Displaying the current username is a simple matter of leveraging the built-in getRemoteUser(), provided by HttpServletRequest. Since, after logging in, I've transitioned into a JSF application - and because I'm adverse to using JSP scriptlets to accomplish use of that getter once I'm in JSF - I simply provide a getter in one of my JSF managed beans that fetches the HTTP request and returns the user name:

public String getUserName() {
    return getServletRequest().getRemoteUser();
}

...referencing it, as usual, with the JSF expression language:
User: #{svh.userName}
Finally, I'll provide log-out functionality. First, a command link (done with the IceFaces framework):

<ice:commandLink 
    action="logout" immediate="true" value="Logout" 
    style="margin-left:5px;color:blue;font-size:medium"/>

The logout action is mapped with a navigation rule:

<navigation-rule>
    <description>Logout</description>
    <from-view-id>/*</from-view-id>
    <navigation-case>
        <from-outcome>logout</from-outcome>
        <to-view-id>/login/logout.jsp</to-view-id>
        <redirect/>
    </navigation-case>
</navigation-rule>

...taking us to the logout.jsp page, which invalidates the session and invites the user to log back in:

<% session.invalidate(); %>

<div style="margin-top:5px; margin-left:25%">
    <h2 style="text-decoration:underline; color:blue;margin-left:-10px">Management App</h2>
    <h3>Logout Succeeded</h3>
    <p>
        You are now logged out of the Management UI.
    </p>
    <a href="/index.jsp" style="text-decoration:underline">Return to Login page.</a>
</div>

The index.jsp redirects to the application's JSF-based home page:

<html>
    <head>
        <title>Management UI</title>
    </head>
    <body>
    <%
        String redirectURL = "./index.iface";
        response.sendRedirect(redirectURL);
    %>
    </body>
</html>

And, as mentioned in the first post, all iface resources are protected by a security constraint, so this will redirect to the login page.

LDAP Authentication with Jetty

2010-07-14T15:20:00.000-06:00

If you noticed nothing else but the titles of my last two posts, you might suspect that I went down the Spring-Security road and back-pedaled to a standard JAAS approach. You would be correct.

As it turned out - to my surprise and with great disappointment - I found Spring Security to be impenetrable. Now, Spring makes a lot of things in my life easier, and in fact that's the only reason I use it. But when it becomes a tangled snarl of undocumented opaqueness, and especially when I read that even an expert in Acegi had trouble with it (see the Wrap-Up in that article), I decide to find another way.

Here are my basic building blocks for a JAAS approach with a JSF application in Jetty. First, I configure my web.xml with an authorization constraint:

    <security-constraint>
        <web-resource-collection>
            <web-resource-name>Protected Resources</web-resource-name>
            <url-pattern>*.iface</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>*</role-name>
        </auth-constraint>
        <user-data-constraint>
            <transport-guarantee>
                CONFIDENTIAL
            </transport-guarantee>
        </user-data-constraint>
    </security-constraint>

My web.xml has already mapped an IceFaces PersistentFacesServlet to *.iface, so that's what the URL pattern is about. My initial naive attempt was to use a URL pattern of /*, but that's much too broad - it will preclude e.g. loading image resources as part of your login page. That login page is also configured in the web.xml:

    <login-config>
        <auth-method>FORM</auth-method>
        <realm-name>ldap</realm-name>
        <form-login-config>
            <form-login-page>/faces/login/login.jsp</form-login-page>
            <form-error-page>/faces/login/login.jsp</form-error-page>
        </form-login-config>
    </login-config>

The login page (./login/login.jsp) is dirt-simple so far. I'm only working on basic functionality at this point, and there's nothing pretty about it:

    <form method=post action="j_security_check">
        <label for="j_username">Username</label>
        <input type="text" name="j_username" id="j_username"/>
        <br/>
        <label for="j_password">Password</label>
        <input type="password" name="j_password" id="j_password"/>
        <br/>
        <input type="submit" value="Login"/>
    </form>

The realm-name portion in the web.xml references the container environment; configuring the realm is delegated to the container in JEE. My container is (embedded) Jetty, and I configure the Jetty realm via a simple entry in my jetty.xml file:

    <Call name="addUserRealm">
        <Arg>
            <New class="org.mortbay.jetty.plus.jaas.JAASUserRealm">
                <Set name="name">ldap</Set>
                <Set name="LoginModuleName">ldapmodule</Set>
            </New>
        </Arg>
    </Call>

By the way, I followed the Jetty tutorial on JAAS to make all of this happen. My previous post mentioned a gotcha in that article around the LDAP Login Module package name. Depending on which version of Jetty you're using, you may need to make the change discussed there.

In either event, the Jetty realm configuration references a LoginModuleName of "ldapmodule", and as per standard JAAS, this configuration is captured in a file (in my case, a file named ./etc/ldap.conf) referenced by the JVM argument -Djava.security.auth.login.config=etc/ldap.conf. That file is basically a replica of the ldaploginmodule example in the Jetty tutorial (again, except for the package name of the LdapLoginModule class), configured of course with the proper schema references and credentials for my LDAP environment.

Finally, I configure a navigation rule so JSF will take me to my target page after successful login:

    <navigation-rule>
        <description>After Login</description>
        <from-view-id>/login/login.jsp</from-view-id>
        <navigation-case>
            <to-view-id>/index.jsp</to-view-id>
            <redirect/>
        </navigation-case>
    </navigation-rule>

Note that this a bare-bones scaffold for authentication. I have yet to deal with login failures, roles, JSF-messaging for the user, and the like.

Tip: Package Name Correction for Jetty LdapLoginModule

2010-07-14T14:00:00.003-06:00

If you're following the Jetty tutorial for establishing JAAS authentication, and you have copied the sample configuration file for the LDAP module, you may come across this error message after you enter your username and password on your login page:


javax.security.auth.login.LoginException: unable to find LoginModule class: org.mortbay.jetty.plus.jaas.spi.LdapLoginModule

The reason is that, at least for the version of Jetty I'm using (6.1.21), the package name has changed to org.mortbay.jetty.plus.jaas.ldap.LdapLoginModule. Make that change in the ldaploginmodule sample in that tutorial and you'll solve this problem (assuming of course you have all the needed dependencies).

For your reference, here's the complete dependency list I'm using for a JSF-based app running with embedded Jetty 6.1.21:

        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty</artifactId>
            <version>${jettyVersion}</version>
        </dependency>

        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty-util</artifactId>
            <version>${jettyVersion}</version>
        </dependency>

        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jsp-2.1-jetty</artifactId>
            <version>${jettyVersion}</version>
        </dependency>

        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty-plus</artifactId>
            <version>${jettyVersion}</version>
        </dependency>

        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty-ldap-jaas</artifactId>
            <version>${jettyVersion}</version>
        </dependency>

Tip: Provide the Spring Security NamespaceHandler Explicitly

2010-07-14T13:31:00.002-06:00

If you're working with Spring Security, you might start your pom with an entry like so:

<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-web</artifactId>
    <version>3.0.0.RELEASE</version>
</dependency>

This will bring in the spring-web, spring-security-core and commons-logging artifacts, and I would not fault you for thinking you're good to go. You're possibly following this tutorial or another among the many out there, and are providing a Spring config that references the http://www.springframework.org/schema/security namespace - and, at least for me, here's where things did not go as expected. At runtime, the error message I received was this:
Unable to locate Spring NamespaceHandler for XML schema namespace [http://www.springframework.org/schema/security]
After a fair amount of Google'ing and Stack-Overflow'ing, I found the problem - and I reproduce it here to save myself (and hopefully you) the headache next time around: there is a dependency missing that looks like this:

<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-config</artifactId>
    <version>3.0.0.RELEASE</version>
 </dependency>

Now you should be good to go.

Using JDK keytool To Generate Keys and Certs

2010-07-02T12:36:00.006-06:00

In a previous post, I stepped through use of IBM's KeyMan GUI to generate SSL keys and certificates, placing them in KeyMan's "token", or as more commonly known, a keystore. I then attempted to use this keystore in both a Windows and Linux deployment of my webapp, and met with mixed success - since I used the Windows version of the KeyMan tool, this worked out OK for the Windows webapp, but not so much for the Linux deployment.

My subsequent attempts to use KeyMan's Unix shell script (km) instead - under a Cygwin environment - were much the same: on deployment, an "Invalid keystore format" exception was issued. Clearly, this is about the difference between DOS and Unix file formats. The next obvious step was to simply execute the km script directly under Linux, foregoing any file encoding or translation issues that might be happening with Cygwin. Here, however, after appearing to generate the key pair, the KeyMan GUI went into some kind of blocking wait - or maybe an infinite loop? a deadlock? There was no way to tell; the GUI simply became unresponsive. Followup exercises to include setting KM_HOME in the environment, unpacking the native library support ZIP file and setting the LD_LIBRARY_PATH to point to them, and etc. all proved fruitless.

Finally, I reverted to using the JDK keytool utility, and - no surprise - this works out just fine in both Windows and Linux (i.e. in terms of generating a keystore that is recognized by the webserver). Here is the script I use to generate things, in both Linux and Cygwin:


######################################
#
#  generate-keystore.sh - Generate key and certificate
#
######################################

CN=MyKeystore
OU='Web - Development'
ORG='My Biz Inc.'
COUNTRY=US
ALIAS=MyBizKeystore
PASS=password
KEYSTORE=keystore
CERTFILE=cert
EXPIRY=730

# remove it if it's there
[ -f "$KEYSTORE" ] && /bin/rm $KEYSTORE

# generate the keystore with a self-signed cert and an RSA keypair
$JAVA_HOME/jre/bin/keytool -genkeypair -keyalg RSA \
-dname "cn=$CN, ou=$OU, o=$ORG, c=$COUNTRY" \
-alias $ALIAS -keypass $PASS -keystore $KEYSTORE \
-storepass $PASS -validity $EXPIRY

# export the certificate so we can look at it
$JAVA_HOME/jre/bin/keytool -exportcert -alias $ALIAS -file $CERTFILE -keystore $KEYSTORE -storepass $PASS

# print the certificate
$JAVA_HOME/jre/bin/keytool -printcert -file $CERTFILE

If you bump into this error message in the generate step:


Incorrect AVA format

...you'll want to make sure you didn't embed any commas or other special characters in the values you provide. For example, I started out with an Organizational Unit (OU) of 'My Biz, Inc.' - but that provoked the error message. Embedded dashes and periods are apparently OK, but note that I've enclosed any values with embedded spaces in single quotes. That's more a shell issue than a keytool problem.

If you bump into an error message something like this, in the print-certificate step:


lengthTag=109, too big

...you might be trying to pass in the entire keystore to the printcert command; that's why I export the certificate first in the script above, using just that piece as the argument to print it out.

Using KeyMan To Generate Keys and Certs

2010-06-30T16:58:00.004-06:00

This post continues my series around Jetty. Previous posts discuss setting up a basic embedded Jetty application, and then some adjustments to repeat that exercise in a Cygwin environment. My next step is to configure SSL...now, I recall earlier adventures using Java's keytool to manage this, and have hoped for something better. Jetty's documentation pointed me to KeyMan, which is by far a nicer way to go - it provides a decent intuitive GUI to help create, delete, and otherwise manage keys and certificates, among other things. Here's an outline of how to use KeyMan to create a PKCS#12 keystore with a self-signed certificate and public-private key pair:

Download, install (unpack zip, etc.), read the README.txt. I did nothing with the km.setup file, but did edit the km.bat as instructed. Turns out that, since I'm on cygwin, that wasn't needed; instead, I execute the km program. Click on the "New" icon to create a new "token" (i.e. repository for keys, certs, etc.):

Choose the PKCS#12 Token from the next dialog, and hit the checkmark ("Complete Dialog") to proceed:

Next, you need to store a key and a certificate in this token. Select "Actions -> Generate Key" from the token management window that appears:

The default algorithm is RSA-1024; that's strong enough for my needs. Click the Complete Dialog checkmark...this takes a second to complete, offering a cool little progress bar while you wait. The new key shows up in the All Certificate Items viewport of the token management window; now we need a certificate to go with it. Click "Actions -> Create Certificate...". Self-signed is good enough for my needs. Click checkmark and fill in the fields as needed (only "Your name" is required):

A verification appears when you check "Complete Dialog" here, with the option to label this certificate. Enter a label if you wish, and again move on with the checkmark:

Save the token to a file by selecting File -> Save. This first prompts you for a passphrase, then a file location.

Prove to yourself that the keystore (token, repository, whatever) is really there and that you can view it in human-friendly form by first exiting the program, restarting it and selecting the "Open existing..." icon, then "Local resource..." and "Open a file...". Browse to the file location you just saved to, enter the passphrase, and you should see your token listed in the Private Certificates category (in the dropdown). Click right on that item and you'll see all the informational details entered when you created the certificate.

Next, I'll see about using that keystore for my Jetty SSL setup. Meanwhile, here are some useful links around KeyMan, SSL and Jetty's SSL instructions:

Solaris Keytool: http://java.sun.com/j2se/1.4.2/docs/tooldocs/solaris/keytool.html
Windows Keytool: http://java.sun.com/j2se/1.4.2/docs/tooldocs/windows/keytool.html
KeyMan: http://www.alphaworks.ibm.com/tech/keyman
OpenSSL: http://www.openssl.org/docs/HOWTO/
OpenSSL FAQ: http://www.openssl.org/support/faq.html
Jetty SSL: http://docs.codehaus.org/display/JETTY/How+to+configure+SSL

Followup: Basic Embedded Jetty in Cygwin

2010-06-30T12:07:00.006-06:00

In a recent post, I described how to get a basic Jetty web application going, using the embedded approach. Since then, I've reproduced it in a Cygwin environment, and here I'll comment on that exercise.

As it turns out, specifying this type of Java startup in Cygwin (or in Windows XP per se) will not work:


java -server -Dbasedir=/usr/local/mywebapp/war -cp /usr/local/mywebapp/war/WEB-INF/lib/* com.mybiz.MyJettyWebServer

That's because the wild-card expression apparently is not supported in a DOS-based environment - even if I enclose the above classpath in quotes. Instead, I'd need to provide a semi-colon-delimited list (not colon-separated - I'm in XP) of all jars under ./WEB-INF/lib. This is not the kind of thing I'd like to do; maintaining that kind of list would be a headache as the webapp evolves. Additionally, keep in mind that I've told Jetty to start up with an exploded warfile location:

        WebAppContext appContext = new WebAppContext();
        File warPath = new File(System.getProperty("basedir"));
        appContext.setWar(warPath.getAbsolutePath());
        HandlerList handlers = new HandlerList();
        handlers.setHandlers(new Handler[]{ appContext, new DefaultHandler() });
        jetty.setHandler(handlers);
        jetty.start();

This will result in a web-level classloader to load all the jars under WEB-INF/lib, which is arguably redundant, since I'm explicitly setting my classpath to the same thing. That in turn will cause loader constraint violations when running the webapp in an IDE such as Intellij, if the run configuration you're using there points to the same classpath (since that application-level classloader loads the classes first, and then the webapp-level classloader tries to do the same thing). I'll defer solving the Intellij problem for now, and just address basic command line startup.

Given maintenance cost concerns, I'm motivated to load the minimal number of jars needed to get Jetty going, then allow it to load the rest of what it needs from WEB-INF/lib. In my particular setup, that minimal set includes my application jar and three Jetty jars:


MyApp-1.0.jar
jetty-6.1.21.jar
jetty-util-6.1.21.jar
servlet-api-2.5-20081211.jar

I figured out this minimal set by just trying to start up the WebServer class and seeing what classdef-not-found problems I had - then searching for the necessary jar by setting up a bash function that I can reuse:


findclass () { find . -name '*.jar' -o -type f |xargs -i bash -c "jar -tvf {}| tr / . | grep -i "$@" && echo {}"; }

...and subsequently invoking it like this:


findclass <dot-delimited-classname>

Once I have all my dependencies figured out, I can invoke my Jetty program with that minimal set, and rely on the webapp-level classloader to do the rest when the embedded Jetty webserver starts:


java -server -Dbasedir=/usr/local/mywebapp/war -cp "MyApp-1.0.jar;jetty-6.1.21.jar;jetty-util-6.1.21.jar;servlet-api-2.5-20081211.jar" com.mybiz.MyJettyWebServer

Note that I've wrapped the classpath in quotes, and, as mentioned, used semi-colons instead of colons.

Send/Receive Notification About an Exception

2010-06-23T17:07:00.002-06:00

From Java Message Service, Second Edition, by Mark Richards, Richard Monson-Haefel, and David A. Chappell - here's a lightweight mechanism for notifying interested consumers about an exception (in a JMS context):

try {
    ...
} catch (Exception up) {
    Message message = session.createMessage();
    message.setStringProperty("Exception", up.getMessage());
    publisher.publish(message);
    throw up;
}

Interested consumers can receive this notification like this:

public void onMessage(Message message) {
    System.out.println("Exception: " + message.getStringProperty());
}

This uses a simple Message object, which contains no payload - only JMS headers and properties. As noted in the book, when simple notification is all that is needed, use of the Message type is the most efficient means to do this.

Basic Embedded Jetty Setup: JSF 1.2 Webapp

2010-06-23T16:04:00.005-06:00

Here are some basic code snippets I've used (to-date ... subject to change) to get an IceFaces-based (JSF 1.2) webapp deployed as an embedded Jetty webapp - using Maven for building. The artifacts include a configuration file, a minimal bootstrap class, the pom and the file used as the maven assembly-plugin descriptor.

The configuration file is jetty.xml, and doesn't require much. It lives under the ./etc directory of my project:

<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">

<Configure id="Server" class="org.mortbay.jetty.Server">

    <Set name="ThreadPool">
        <New class="org.mortbay.thread.QueuedThreadPool">
            <!-- initial threads set to 10 -->
            <Set name="minThreads">10</Set>
            <!-- the thread pool will grow only up to 200 -->
            <Set name="maxThreads">200</Set>
            <!-- indicates that having 20 and below, the pool will be considered low on threads -->
            <Set name="lowThreads">20</Set>
            <!-- The number of queued jobs (or idle threads) needed before the thread pool is grown (or shrunk) -->
            <Set name="SpawnOrShrinkAt">2</Set>
        </New>
    </Set>

    <Call name="addConnector">
        <Arg>
            <New class="org.mortbay.jetty.nio.SelectChannelConnector">
                <!-- the ip address or domain to bind -->
                <Set name="host">
                    <SystemProperty name="jetty.host"/>
                </Set>
                <!-- the port to use/bind, defaults to 8080 if property not set -->
                <Set name="port">
                    <SystemProperty name="jetty.port" default="8080"/>
                </Set>
                <!-- the time in milliseconds when a connection is considered idle -->
                <Set name="maxIdleTime">300000</Set>
                <!-- the number of acceptors (their job is to accept the connection and dispatch to thread pool) -->
                <Set name="Acceptors">2</Set>
                <!-- should the connection statistics be turned on? (Not advisable in production) -->
                <Set name="statsOn">false</Set>
                <!-- the confidential port -->
                <Set name="confidentialPort">8443</Set>
                <!-- indicates the minimum number of connections when the server is considered low on resources -->
                <Set name="lowResourcesConnections">5000</Set>
                <!-- when low on resources, this indicates the maximum time a connection must be idle to not be closed -->
                <Set name="lowResourcesMaxIdleTime">5000</Set>
            </New>
        </Arg>
    </Call>

    <!-- Stops the server when ctrl+c is pressed (registers to Runtime.addShutdownHook) -->
    <Set name="stopAtShutdown">true</Set>
    <!-- send the server version in the response header? -->
    <Set name="sendServerVersion">true</Set>
    <!-- send the date header in the response header? -->
    <Set name="sendDateHeader">true</Set>
    <!-- allows requests(prior to shutdown) to finish gracefully -->
    <Set name="gracefulShutdown">1000</Set>

</Configure>

This configuration file is referenced by the bootstrap class, using it to configure Jetty. This class also sets the context for the webapp, points to the top-level directory of the exploded WAR content, sets a webapp context and a default context as handlers for Jetty, and starts up the webserver:

public class MyJettyWebServer {

    public static void main(String[] args) throws Exception {

        Server jetty = new Server();

        // configure Jetty by pointing to config file(s)
        String[] configFiles = { "etc/jetty.xml" };
        for (String configFile : configFiles) {
            XmlConfiguration configuration = new XmlConfiguration(new File(configFile).toURI().toURL());
            configuration.configure(jetty);
        }

        // set the context for the webapp
        WebAppContext appContext = new WebAppContext();
        appContext.setContextPath("/mycontext");

        // point to the top-level directory of the exploded WAR content
        File warPath = new File(System.getProperty("basedir"));
        appContext.setWar(warPath.getAbsolutePath());

        // set a webapp context and a default context as handlers for Jetty
        HandlerList handlers = new HandlerList();
        handlers.setHandlers(new Handler[]{ appContext, new DefaultHandler() });
        jetty.setHandler(handlers);

        // start up the webserver
        jetty.start();
    }
}

The pom specifies IceFaces 1.8.2, being careful to exclude the EL API jar wherever that's brought in transitively, as per http://www.icefaces.org/docs/v1_8_0/htmlguide/devguide/appendixA.html; and it specifies Jetty and Log4J artifacts. Note that JSP support is explicitly specified:

    ....
    <!-- use JAR packaging for embedded Jetty -->
    <packaging>jar</packaging>
    ....
    <dependencies>
        <dependency>
        <dependency>
            <groupId>org.icefaces</groupId>
            <artifactId>icefaces</artifactId>
            <version>1.8.2</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.el</groupId>
                    <artifactId>el-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.icefaces</groupId>
            <artifactId>icefaces-comps</artifactId>
            <version>1.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.icefaces</groupId>
            <artifactId>icefaces-facelets</artifactId>
            <version>1.8.2</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.el</groupId>
                    <artifactId>el-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>javax.faces</groupId>
            <artifactId>jsf-api</artifactId>
            <version>1.2_12</version>
        </dependency>
        <dependency>
            <groupId>javax.faces</groupId>
            <artifactId>jsf-impl</artifactId>
            <version>1.2_12</version>
        </dependency>
        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty</artifactId>
            <version>6.1.21</version>
        </dependency>
        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jetty-util</artifactId>
            <version>6.1.21</version>
        </dependency>
        <dependency>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>jsp-2.1-jetty</artifactId>
            <version>6.1.21</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4jVersion}</version>
        </dependency>
    </dependencies>
</project>

The assembly-plugin descriptor has a few things worth mentioning. In the dependency sets, I lay down my dependencies in the standard webapp location, ./WEB-INF/lib:

        <dependencySet>
            <unpack>false</unpack>
            <scope>runtime</scope>
            <outputDirectory>war/WEB-INF/lib</outputDirectory>
        </dependencySet>

I also specify several destinations for my dev-time files to deal with Jetty-related stuff:

        <!--
        Jetty deployment: configuration file location
        -->
        <fileSet>
            <directory>etc</directory>
            <outputDirectory>/usr/local/mywebapp/etc</outputDirectory>
            <fileMode>0644</fileMode>
        </fileSet>
        <!--
        Jetty deployment: webapp deployment location - hmmm, this one might not be needed...
        -->
        <fileSet>
            <directory>webapps</directory>
            <outputDirectory>/usr/local/mywebapp/webapps</outputDirectory>
            <fileMode>0644</fileMode>
        </fileSet>
        <!--
        Jetty deployment: production-time exploded warfile location
        -->
        <fileSet>
            <directory>src/main/webapp</directory>
            <outputDirectory>/usr/local/mywebapp/war</outputDirectory>
            <fileMode>0644</fileMode>
        </fileSet>

Finally, to start the program:


java -server -Dbasedir=/usr/local/mywebapp/war -cp /usr/local/mywebapp/war/WEB-INF/lib/* com.mybiz.MyJettyWebServer

Browse to localhost:8080 and your webapp should appear.

You'll notice I did not use the maven-jetty plugin (nor the jetty-maven plugin - yes, there are two different ones, each with different names, schemas and behaviors). The good news about the plugin is that it shields you from much of the configuration/deployment exercises you'll need; that's also the bad news. I needed to understand explicitly what dependencies, etc. I'd need for production, so I chose to do things manually. The good news around this is that I've done most of the heavy lifting to grease the skids for future embedded-Jetty exercises.

References

Jetty 6.x Wiki
Jetty 7x, 8.x Wiki

Ternary Expressions Problem with JSF 1.2

2010-06-22T12:29:00.001-06:00

This is a followup on my previous post that offered a workaround to this error message, as seen when working with JSF 1.2:

javax.el.ELException: Error Parsing: 
Caused by: com.sun.el.parser.ParseException: Encountered ":text"

In my first post, I failed to find the root cause, so I took the path of least resistance in the interest of the project schedule (using the heavy-handed approach of two separate panel groups, backing the condition out to the "rendered" attribute of each group). As it turns out, the root cause is lack of white space around the colon, as explained in this article from Oracle. That is, given an expression:

#{isThisTrue?doThis:doThat}

...and depending on what the deployment environment is, you'll either notice nothing or bump into this error message. When that first post was written, I was deploying to Glassfish 8.x - and I needed to use the workaround described above. Recently, I deployed to JBoss 4.x and later to 5.1, and here I did not notice anything - but then I redeployed to embedded Jetty 6.1.x, and the problem re-appeared. This is probably because JBoss shielded me from needing the magic combination of JSF, JSP, Facelets, JSTL and etc. to make things work, but vanilla Jetty does not.

In either event, you might have better luck than I with determining the dependency mix, including tweaking the web.xml to configure the correct expression factory, trying various permutations of Jetty's JSP libraries, and etc. (aka "time sink"); but, more simply, the fix is to change the problematic expression (well, actually it's a bug in JSF 1.2) to this:

#{isThisTrue?doThis : doThat}

This time, I googled for the right thing ("JSF 1.2 ternary parse exception") and, by now, the article from Oracle had finally been published (it wasn't there until after I originally bailed out with my workaround).

Blocking Sends When ActiveMQ Broker is Down

2010-06-14T17:15:00.012-06:00

As it turns out, even though you may have specified useAsyncSend on an ActiveMQ broker, your sends will block if that broker goes down (or if the broker was not running when the client first attempts a send). Don't waste any cycles (like I did) tweaking the delivery mode, whether or not the session is transacted, or other connection settings - the "problem" will occur with use of the failover transport (i.e., my broker URL is failover://(tcp://localhost:61616)) . This transport will automatically attempt reconnection until success, but during these attempts, your client will block. This may not be the behavior you want; I'm not able just yet to give you a satisfying solution, but I can give you some hooks to come up with something on your own (and if so, please let me know!).

One so-called solution is of course to back off using the failover transport - e.g., use TCP instead. But this is heavy-handed and arguably overkill - the failover transport serves an excellent purpose in the face of broker failures, and I'm reluctant to let it go just to solve an exceptional condition. In either event, you'll receive something like a java.net.ConnectException with the TCP protocol when the broker is down; you can try-catch that and do whatever makes sense to you. But that's a questionable workaround. Instead, here's some advice from the ActiveMQ website:

If you use failover, and a broker dies at some point, your sends will block by default. Using TransportListener can help with this regard. It is best to set the Listener directly on the ActiveMQConnectionFactory so that it is in place before any request that may require an network hop. Additionally you can use timeout option which will cause your current send to fail after specified timeout. The following URL, for example

failover:(tcp://primary:61616)?timeout=3000

will cause send to fail after 3 seconds if the connection isn't established. The connection will not be killed, so you can try sending messages later at some point using the same connection (presumably some of your brokers will be available again).

Now, I first tried the timeout parameter approach - as it turns out, your application will get an exception at the timeout expiration. But this introduces its own problems around performance - if you have thousands (or, even dozens) or messages, each one timing out after say 2-3 seconds, your throughput will be hurt badly. So, I bailed out on this approach and took a closer look at the TransportListener. Here's what I implemented in my publisher class:

    public void onCommand(Object command) { /* EMPTY */ }

    public void onException(IOException error)
    {    Logging.publisher.warn("Transport exception encountered", error);  }

    public void transportInterupted()
    {    Logging.publisher.warn("Transport interrupted! Is Broker down?");  }

    public void transportResumed()
    {    Logging.publisher.warn("Transport connectivity is restored");  }

My test program received callbacks for all of these methods, as expected, except for the onException. The ActiveMQ javadocs aren't much help here ("An unrecoverable exception has occured on the transport"). From this point, I tried combining the timeout parameter with the transport callback - e.g.

    public void transportInterupted()
    {    this.amqFactory.setSendTimeout(100); }

    public void transportResumed()
    {    this.amqFactory.setSendTimeout(-1); }

My strategy here is to minimize the timeout, and hence maximize the throughput in the face of dozens of sends, by reducing the timeout to a bare minimum (and then restoring it to the original timeout value when the broker comes back up). But, I had no success here; the timeout remained at the original value as set in the URL (which I consider to be too high in this context) even after transportInterupted (allegedly) changed it. I'm not willing to set my initial timeout to a low value either, since this may cause exceptions even under nominal operating conditions - not good.

From here, I leave it to you to find some clever use of the transport listener that can deal with a broker that goes down. Good luck, and let me know.

Preserve Unix Line Endings with Windows Intellij

2010-06-11T12:06:00.004-06:00

Just a quick note: If you're editing shell scripts and the like on a Windows platform with Intellij (my version is 9.0), here's how I succeeded preserving Unix-style line endings. First, to more accurately describe the problem, a given file has been saved in the Windows IDE, then checked in to a Perforce repository, and then sync'd onto a Unix server. When editing that file with vi, the settings [dos] [noeol] appear - and I'd like to get rid of these. I do not want a DOS format, and I do want an end-of-line. Here are the steps I took:

First, set the Intellij "Line separator" property to Unix - enter "Line separator" in the Settings search box, then choose the Code Style - General result. The Line separator property is towards the bottom of that panel. Yes, it does state "for new files", and not surprisingly I did not succeed simply editing an existing file - the DOS-style format remained after I saved and checked it in (Perforce). But, this does cover things going forward - at least this gets rid of the [dos] problem on new files - however, on the subsequent edit of the new file on Unix, the [noeol] is still there.
To get rid of the [noeol] problem, I set one more thing in Intellij: in IDE Settings - search for "Ensure blank", which will bring up the optional checkbox for "Ensure blank line before end of file on Save". Check that option, and from here it appears that newly created files in the IDE will no longer have the [noeol] affliction. To address this problem on existing files, simply edit/save/check-in the file with the IDE. Probably, you could also just add a blank line at the end with vi on Unix - I didn't try this, but it sounds reasonable.
To fix the [dos] problem on existing files, open the file in vi and type ":set ff=unix". Save, check in, and this problem should not reappear even after subsequent edits in the Windows IDE.

Don't forget, depending on what steps you take from above, that if you have set your CVS system to "revert unchanged" files (as I do with Perforce), that you may need to actually change something that the CVS will notice before it will check it in. Huh, maybe adding some comments...

Hope that helps. I'm sure I'll bump into this headache again, so I can also hope this helps me when that happens.

Get a Spring Aspect Working

2010-05-28T11:42:00.012-06:00

After some time away from AOP, I've found myself forgetting the bare essentials of getting off the ground - i.e., what's the least I have to do just to prove that my aspect is getting invoked?

Assuming a Spring 3.0 environment, here is the summary:

Create the aspect
Create a Spring context file that declares AOP
Use Spring to instantiate both the aspect and the application object
Run your test case to prove it's working

Here's a bare-bones aspect:

package com.mybiz;
import org.aspectj.lang.annotation.Aspect;

@Aspect
@Component
public class MyAspect
{
    // Flag indicating that a given pointcut was in fact invoked.
    public boolean madeIntercept = false;

    @Pointcut("execution(* com.mybiz.MyApp.myMethod(..))")
    public void myMethodPointcut() {}

    @After("myMethodPointcut()")
    public void doSomething() { madeIntercept = true; }
}

Here's a simple Spring file, named spring-test-aspect.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/aop
       http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context
       http://www.springframework.org/schema/context/spring-context.xsd">

    <context:annotation-config/>
    <context:component-scan base-package="com.mybiz"/>

    <aop:aspectj-autoproxy/>

</beans>

The aspect will automatically get Spring-managed since it uses the @Component annotation, and component scanning declared above includes its package. Use the same approach with your application object:

package com.mybiz;
@Component
public class MyApp
{
    public void myMethod() {}
}

Filling in the method is irrelevant; all we're trying to do is prove that if that method is invoked, that the aspect will intercept the control flow afterwards. Here's a test case that does this for us:

package com.mybiz;

@ContextConfiguration(locations ={"file:spring-test-aspect.xml"})
public class MyAspectTest extends AbstractTestNGSpringContextTests
{
    @Resource
    private MyAspect myAspect;

    @Resource
    private MyApp myApp;

    @PostConstruct
    public void init() { assert myAspect != null; assert myApp != null; }

    @Test
    public void confirmInterception() 
    { 
        assert !myAspect.intercepted();
        myApp.myMethod(); 
        assert myAspect.intercepted();
    }
}

There's the proof-of-concept; from here, you'd add real functionality to your application class and decorate that functionality with the aspect. I should offer a fair warning: the above snippets have not been actually compiled and run, rather simply copy-pasted with various names changed to be more general purpose (as such, it might not be totally correct). So, if you bump into any problems - you have my apologies, but I'm sure the problems will be trivial.

Unit Test with a Reusable Spring-JMS Configuration

2010-04-01T15:42:00.012-06:00

In my previous post, I introduced a general-purpose, allegedly reusable Spring configuration for an ActiveMQ-based JMS application. In this post, I'll expand on that by writing a unit test around a basic configuration that establishes an evolving prototype.

Writing tests that fail, as it turns out, is pretty simple. Let's pick that low-hanging fruit:

package com.mybiz.jms;

import org.springframework.jms.core.JmsTemplate;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.testng.AbstractTestNGSpringContextTests;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import javax.annotation.Resource;
import javax.jms.DeliveryMode;

@ContextConfiguration(locations =
{
    "classpath:com/mybiz/jms/spring-jms-demo.xml"}
)
public class BasicConfigTest extends AbstractTestNGSpringContextTests
{
    @Resource
    private JmsTemplate myTemplate;

    @BeforeClass
    public void setUp() throws Exception
    {
        System.out.println("Run BasicConfigTest...");
        assert applicationContext != null;
    }

    @Test
    public void testConfig() throws Exception
    {
        System.out.println("testConfig...");
        assert myTemplate.getDeliveryMode() == DeliveryMode.PERSISTENT;
        assert myTemplate.getTimeToLive() == 10000;
        assert myTemplate.isExplicitQosEnabled();
        assert myTemplate.isPubSubDomain();
        assert myTemplate.isSessionTransacted();
        assert myTemplate.isPubSubNoLocal();
    }
}

I don't need to tell you that, without any other work, this test will fail; let's take on something a bit less trivial. The test assumes a Spring configuration which resides in the same logical directory tree as the class. Here's that spring-jms-demo.xml configuration:

<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"     
       xmlns="http://www.springframework.org/schema/beans" 
       xsi:schemalocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
    
    <import resource="spring-context.xml"/>
    <import resource="spring-jms.xml"/>

</beans>

The first import establishes context-based annotation, scanning for components and property placeholder
resolution, which will combine to help us reuse the generic Spring-JMS configuration (the second import). This spring-context.xml file is the beginning of my application-specific configuration:

<beans xmlns:context="http://www.springframework.org/schema/context"             
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
       xmlns="http://www.springframework.org/schema/beans"  
       xsi:schemalocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.springframework.org/schema/context
       http://www.springframework.org/schema/context/spring-context-3.0.xsd">
       
        <context:annotation-config>
            <context:component-scan base-package="com.mybiz.jms"/>
            <context:property-placeholder 
                location="com/mybiz/jms/spring-jms.properties" 
                system-properties-mode="OVERRIDE"/>
        </context:annotation-config>
</beans>

That file uses a properties file (spring-jms.properties) that backfills the generic Spring-JMS configuration with more of our application specifics:

# URL for connection to broker - use testing URL:
brokerURL=vm://localhost?async=false&broker.persistent=false
#
# messages will remain in broker for 10 seconds
timeToLive=10000
#
# 1 - non-persistent / 2 - persistent
deliveryMode=2
#
# Specify concurrency of 40 for starters
sessionCacheSize=40
#
# big perf boost, minimal risk (see comments in spring-jms.xml)
useAsyncSend=true
#
# support for flexibility on per-send basis
explicitQosEnabled=true
#
# concern is around a blocking consumer (see comments in spring-jms.xml)
dispatchAsync=true
#
# txn around sends
sessionTransacted=true
#
# just in case
pubSubNoLocal=true
#
# must specify this if it's a topic; default is false
pubSubDomain=true
#
# txn around receives
acknowledge=transacted
#
# destination for topic
destination=theTopic

Finally, I need two listeners that are referenced by the Spring-JMS configuration:

package com.mybiz.jms;

import org.springframework.stereotype.Service;

@Service
public class MyListener {}  // works for now; but this will need some work

package com.mybiz.jms;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import javax.jms.ExceptionListener;
import javax.jms.JMSException;

@Component
public class MyExceptionListener implements ExceptionListener
{
    private final Logger log = LoggerFactory.getLogger(getClass());
    public void onException(JMSException e)
    {
        log.error("JMS error", e);
    }
}

Now the unit test will run successfully; there's no (meaningful) output to show you, and for that matter the test and listeners are along the lines of "what's the simplest thing that would work" - it's really only done to demonstrate that all the moving parts are in place and that I've used them correctly so far. In my next, I'll start adding on some target functionality and look for ways to test it.

A General Template for Spring-JMS Applications

2010-04-01T14:52:00.006-06:00

Here I'll post a simple Spring setup that can be reused for specific JMS-based applications. It assumes the use of ActiveMQ 5.3.0, opting to use Spring's caching connection factory in lieu of ActiveMQ's pooling version. It uses various Spring-JMS components, including one JmsTemplate and one MessageListenerContainer, respectively intended for sending and receiving messages to-and-from a topic. In upcoming posts, I'll use this to prototype a JMS application.

The connection factory and JMS template have most of their specific configuration parameters externalized to facilitate better reuse.

For the ActiveMQ connection factory, the externalized properties include the broker URL, use of asynchronous sends and choice around dispatching asynchronously. The example XML template that follows includes some commentary around the pros/cons of configuring the last two. This factory is used by Spring's caching connection factory, externalizes configuration of session cache size, and assumes the presence of an exception listener.

The JMS template has the following properties that are configurable: delivery mode, time-to-live, whether or not explicit quality of service is enabled, whether or not the session is transacted, whether or not a given connection should listen for messages sent by that same connection, and whether or not the domain is pub-sub or point-to-point.

I would have externalized configuration for the listener container but, as it turns out, apparently the listener-container parent tag does not support this, so I've hardwired this to use the default listener container, listening on a topic within a transaction.

Here's that template:

<?xml version="1.0" encoding="UTF-8"?>
<!--
A reusable template for Spring-JMS applications. The following must be configured externally, presumably
via Spring's property-placeholder mechanism:

brokerURL
useAsyncSend
dispatchAsync
sessionCacheSize
deliveryMode
explicitQosEnabled
sessionTransacted
pubSubNoLocal
pubSubDomain
destination

As well, the following is assumed:

An exception listener that can be dereference by the name 'myExceptionListener' is assumed present 
in classpath.
A message listener that can be dereference by the name 'myListener' is assumed present in classpath.
-->

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:jms="http://www.springframework.org/schema/jms"
       xmlns:amq="http://activemq.apache.org/schema/core"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
           http://www.springframework.org/schema/context
           http://www.springframework.org/schema/context/spring-context-3.0.xsd
           http://activemq.apache.org/schema/core
           http://activemq.apache.org/schema/core/activemq-core-5.3.0.xsd
           http://www.springframework.org/schema/jms
           http://www.springframework.org/schema/jms/spring-jms-3.0.xsd">

    <!--
    Configure connection factory
    -->
    <bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
        <!--
        for testing, we want to use vm://localhost?async=false&broker.persistent=false.
        This does intra-JVM messaging using synchronous sends and doesn't bother with persisting
        messages to cover provider failures. See http://activemq.apache.org/uri-protocols.html for
        protocol options.
         -->
        <property name="brokerURL" value="${testUrl}"/>
        <!--
         See http://activemq.apache.org/async-sends.html - use async sends for better throughput but
         only if you're able to tolerate some message loss. 

         From the FUSE Tuning Guide (http://fusesource.com/wiki/display/ProdInfo/FUSE%20Message%20Broker%20Performance%20Tuning%20Guide):

         If you are using persistent messaging and you don't want to use Async Sends (see below) then you should
         use JMS transactions to batch up many operations inside a single transaction. If you enable Async Sends
         then you will reduce the blocking in senders which increases throughput.

         The only downside of asynchronous sending is if the send fails for whatever reason (security exception
         typically or some transport failure), then you don't get an exception thrown in the sender thread,
         since all the work is done asynchronously, though your ErrorListener will get notified.

         From the activemq.xsd:

         Forces the use of Async Sends which adds a massive performance boost; but means that the send() method will
         return immediately whether the message has been sent or not which could lead to message loss.
         -->
        <property name="useAsyncSend" value="${useAsyncSend}"/>
        <!--
        From the activemq.xsd:

        Enables or disables the default setting of whether or not consumers have their messages dispatched
        synchronously or asynchronously by the broker. For non-durable topics for example we typically dispatch
        synchronously by default to minimize context switches which boost performance. However sometimes its better
        to go slower to ensure that a single blocked consumer socket does not block delivery to other consumers.
        See http://activemq.apache.org/consumer-dispatch-async.html.
        -->
        <property name="dispatchAsync" value="${dispatchAsync}"/>
    </bean>

    <!--
    Use the vanilla connection factory to configure a caching connection factory, using the Spring
    version instead of ActiveMQ. Default session cache size for Spring factory is 1. An exception listener
    that can be dereference by the name 'myExceptionListener' is assumed present in classpath.
    -->
    <bean id="cachingConnectionFactory"
          class="org.springframework.jms.connection.CachingConnectionFactory"
          destroy-method="destroy">

        <property name="targetConnectionFactory" ref="connectionFactory"/>
        <property name="sessionCacheSize" value="${sessionCacheSize}"/>
        <property name="exceptionListener" ref="myExceptionListener"/>

    </bean>

    <!--
    The JMS template intended for sending of messages.
    -->
    <bean id="myTemplate" class="org.springframework.jms.core.JmsTemplate">

        <constructor-arg ref="cachingConnectionFactory"/>
        <property name="deliveryMode" value="${deliveryMode}"/>
        <property name="timeToLive" value="${timeToLive}"/>
        <!--
        if explicit QoS is enabled, you can control deliveryMode, priority and TTL on a per-send basis
        -->
        <property name="explicitQosEnabled" value="${explicitQosEnabled}"/>
        <property name="sessionTransacted" value="${sessionTransacted}"/>
        <property name="pubSubNoLocal" value="${pubSubNoLocal}"/>
        <!--
        override JMS template default domain type of PTP if you want pub-sub
        -->
        <property name="pubSubDomain" value="${pubSubDomain}"/>
    </bean>

    <!--
    concurrency should be 1 for topic listeners. This is one attribute that apparently canNOT be
    externalized into the properties file. Use transacted acknowledgment (another attribute that
    cannot be externalized). A message listener that can be dereference by the name 'myListener' 
    is assumed present in classpath.
    -->
    <jms:listener-container container-type="default"
                            destination-type="topic"
                            acknowledge="transacted"
                            connection-factory="cachingConnectionFactory"
                            concurrency="1">
        <jms:listener
                destination="${myTopic}"
                ref="myListener"/>
    </jms:listener-container>

</beans>

That's a starting point; from here, I'll add configuration, listeners and a test case as the next step.

References

Apache ActiveMQ
Spring 3.0.0 Javadoc
ActiveMQ Javadoc
FUSE Tuning Guide
Spring 3.0.0 Reference Documentation
Efficient Lightweight JMS with Spring and ActiveMQ
Creating Robust JMS Applications (Java EE 5 Tutorial)

Asynchronous JMS Request/Reply

2010-03-25T13:50:00.008-06:00

My recent previous posts have described a JMS-based request/reply client that uses blocking request semantics, and has no way of knowing if the service has thrown an exception and as such will never reply; and another that is marginally better, which listens on an invalid channel for messages about exceptions. That slight improvement still leaves a fair amount to be desired.

To clarify the goal of this exercise, I'd like the following:

JMS request-reply arrangement, i.e. issue a request to some service for some information or for a task to be performed, and receive a reply (either with the information or a status/confirmation about the task)
Receive information about an invalid request, or an exception that happened on the service side, in its attempt to fulfill the request
No blocking request is left hanging in the face of a service-side exception

So, to address #3, we already know we could use block-without-wait or block-with-timeout; or we can use an asynchronous listener. Using the different flavors of blocking request means using one's own MessageConsumer instead of the built-in QueueRequestor (which is what I used in the first two parts of this series). From here, however, I'd prefer to just move on to an asynchronous idiom - but before leaving, check this article for some additional insights around synchronous request-reply.

Now if we use our own asynchronous consumer, we no longer have the automatic temporary queue construction done via the QueueRequestor. We'll construct our own invalid queue, but we do not want one for each potential requestor - construction of these is expensive. To be more precise, I'll quote the ActiveMQ recommendation:

The best way to implement request-response over JMS is to create a temporary queue and consumer per client on startup, set JMSReplyTo property on each message to the temporary queue and then use a correlationID on each message to correlate request messages to response messages. This avoids the overhead of creating and closing a consumer for each request (which is expensive). It also means you can share the same producer & consumer across many threads if you want (or pool them maybe).

This example will not do exactly that; instead, for simplicity (and as an excuse to demonstrate message selection), I'll construct one shared invalid queue - but this presents the problem of how to return invalid information only to the requestor that should hear about it (i.e., if there are multiple queue-based clients, then only the first one to receive from a given queue will get that message). To address this problem, we can use message selection - but keep in mind, this relies on all requestors in a given system to be using the same filter. If for example one rogue client did no filtering with a message selector, it would be competing with the intended receiver for a message about exceptions - and if it got there first, the intended receiver would never know about it.

We've already seen a remote service that sends messages about exceptions to an invalid queue; now we want to extend that to additionally set a property in the reply message that facilitates the intended message selection. Here's an example of that, where the service agrees to set a unique ID as a property in the reply message (see the previous posts for details around the RemoteService, InvalidQueueRemoteService, etc.):

package com.mybiz.jms.activemq.server.requestreply.replier;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.TextMessage;

public class MessageSelectorRemoteService extends InvalidQueueRemoteService {

    public static final String OID_PROPERTY = "oid";

    public MessageSelectorRemoteService(String theUrl, String serviceQueueName, String invalidQueueName)
            throws JMSException {

        super(theUrl, serviceQueueName, invalidQueueName);
        System.out.println("   ...service will now add OID property to reply...");
    }

    protected TextMessage decorateMessage(Message requestMessage, TextMessage replyMsg)
            throws JMSException {

        // include the OID_PROPERTY property
        String oid = requestMessage.getStringProperty(OID_PROPERTY);
        replyMsg.setStringProperty(OID_PROPERTY, oid);
        return replyMsg;
    }

    public static void main(String[] args) throws Exception {

        RemoteService service = 
                new MessageSelectorRemoteService(url, serviceQueueName, invalidQueueName);
        RemoteService.run(service);

    }
}

One consumer using this service could set up a single MessageListener that receives both nominal and exceptional messages:

package com.mybiz.jms.activemq.server.requestreply.requestor.async;

import com.mybiz.jms.activemq.server.requestreply.connection.AsyncConsumerConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.connection.ConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.connection.ProducerConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.replier.InvalidQueueRemoteService;
import com.mybiz.jms.activemq.server.requestreply.replier.MessageSelectorRemoteService;
import com.mybiz.jms.activemq.server.requestreply.replier.RemoteService;
import com.mybiz.jms.activemq.server.requestreply.util.MessageUtil;

import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;

// you'll see lots of protected access in fields and methods here - that's because I just happen
// to know that I'll be extending this class, and protected qualifiers are "good enough" for
// demo purposes:
public class InvalidListener implements MessageListener {

    // add message selector so queue listeners only gets their own messages,
    // and so that no other clients receive those messages instead. This particular
    // OID can fail if two clients are started at exactly the same millisecond:
    protected static final String OID = MessageUtil.getDateTime();
    protected static final String MSG_SELECTOR =
            MessageSelectorRemoteService.OID_PROPERTY + "='" + OID + "'";

    // keep track of number of message sent and received; if the gap between these grows
    // too much, log a message
    protected static int numSent = 0, numReceived = 0;
    protected static final int maxGap = 6;

    // protect numSent and numReceived during concurrent access
    protected static final Object mutex = new Object();

    // this client uses message selectors so it receives only those messages intended
    // for it on the invalid queue, ftm no other clients will receive those messages. With
    // message selectors, the filtering is done by the provider so it's more efficient than
    // asking each client to inspect each message for those that apply to it alone.

    public void onMessage(Message msg) {

        processReceived(msg);
    }

    protected ConnectionStuff setupInvalidConnection() throws Exception {

        // miracle occurs here: this sets the message listener to this object, so both
        // nominal and invalid messages will now arrive in the onMessage method:
        return new AsyncConsumerConnectionStuff(
                RemoteService.url, InvalidQueueRemoteService.invalidQueueName,
                this, MSG_SELECTOR, true);
    }

    protected static void run(InvalidListener requestor) throws Exception {

        boolean shutdown = false;

        // set up producer that sends requests to the service queue. Messages to the queue should
        // have set the replyTo value to a temporary queue created manually.
        ProducerConnectionStuff requestorConnection =
                new ProducerConnectionStuff(RemoteService.url, RemoteService.serviceQueueName, true);

        // set up consumer that receives asynchronously at the temporary queue and start the connection
        ConnectionStuff consumerConnection =
                new AsyncConsumerConnectionStuff(
                        RemoteService.url, "MyListener", requestor,
                        requestor.MSG_SELECTOR, true);

        // factored out so it can be overridden, which will happen in next example requestor:
        ConnectionStuff invalidConnection = requestor.setupInvalidConnection();

        // send a message - async handler should get the reply
        int msgNum = 0;
        while (!shutdown) {
            TextMessage requestMsg = requestorConnection.getSession().createTextMessage();
            if ((++msgNum % 3) == 0) {
                // every 3rd message is NULL, which we know will provoke an invalid queue message
                requestMsg.setText(null);
            } else {
                requestMsg.setText("Async data request #" + msgNum);
            }

            // remote service agrees to set property in invalid replies matching any given
            // with RemoteService.OID_PROPERTY name
            requestMsg.setStringProperty(MessageSelectorRemoteService.OID_PROPERTY, requestor.OID);
            requestorConnection.send(requestMsg, consumerConnection.getDestination());

            requestor.processSent(requestMsg);
        }

        requestorConnection.getConnection().close();
        consumerConnection.getConnection().close();
        invalidConnection.getConnection().close();
    }

    public static void main(String[] args) throws Exception {

        run(new InvalidListener());
    }

    private void processSent(Message requestMsg) throws Exception {

        synchronized (mutex) {
            numSent++;
        }

        MessageUtil.examineSentRequest(requestMsg);
        Thread.currentThread().sleep(3000);

        // confirms the number of messages received vs sent is within a specified threshold. If
        // the remote service goes down, this message will log continuously; but within a finite
        // amount of time (depending on how long the service was down), the gap will be made up
        // (from empirical observations)
        if (numSent - numReceived > maxGap) {
            System.err.println("WARNING: # sent = " + numSent
                    + ", # received = " + numReceived);
        }
    }

    protected void processReceived(Message msg) {

        synchronized (mutex) {
            ++numReceived;
        }
        // confirms the message received has an OID property value as expected
        MessageUtil.process(msg, MessageSelectorRemoteService.OID_PROPERTY, OID);
    }
}

This requestor also monitors the gap between the number of messages sent vs the number received - if that starts getting too large, it will notice; this could indicate some kind of trouble around receiving the expected replies. As hoped, it now receives messages about exceptions, etc. - here's the startup of the service, and then the requestor:

Mar 25, 2010 1:27:25 PM org.apache.activemq.transport.failover.FailoverTransport doReconnect
Service is waiting for a request...
INFO: Successfully connected to tcp://localhost:61616
   ...service will now send replies to an INVALID queue...
   ...service will now add OID property to reply...

Mar 25, 2010 1:28:42 PM org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://localhost:61616
13:28:42.779 Request for service 'Async data request #1' has been sent...
    Message ID: ID:localhost-34234-1269545322429-0:0:1:1:1
    Correlation ID: null
    Reply to:   queue://MyListener
    OID property:   13:28:42.217
----------------------------------------------
13:28:42.796 Received reply 'Normal Service Reply'
    Message ID: ID:localhost-34222-1269545245163-0:0:1:2:1
    Correlation ID: ID:localhost-34234-1269545322429-0:0:1:1:1
    Reply to:   null
    OID property:   13:28:42.217
----------------------------------------------
13:28:45.792 Request for service 'Async data request #2' has been sent...
    Message ID: ID:localhost-34234-1269545322429-0:0:1:1:2
    Correlation ID: null
    Reply to:   queue://MyListener
    OID property:   13:28:42.217
----------------------------------------------
13:28:45.819 Received reply 'Normal Service Reply'
    Message ID: ID:localhost-34222-1269545245163-0:0:1:3:1
    Correlation ID: ID:localhost-34234-1269545322429-0:0:1:1:2
    Reply to:   null
    OID property:   13:28:42.217
----------------------------------------------
13:28:48.798 Request for service 'null' has been sent...
    Message ID: ID:localhost-34234-1269545322429-0:0:1:1:3
    Correlation ID: null
    Reply to:   queue://MyListener
    OID property:   13:28:42.217
----------------------------------------------
13:28:48.815 Received reply 'Exception occurred: java.lang.IllegalStateException: java.lang.IllegalStateException: NULL message received'
    Message ID: ID:localhost-34222-1269545245163-0:0:2:2:1
    Correlation ID: ID:localhost-34234-1269545322429-0:0:1:1:3
    Reply to:   null
    OID property:   13:28:42.217
----------------------------------------------

As expected from the requestor logic, every 3rd message results in an exception, and the requestor is notified about this. So we've solved the problem around a request blocking, potentially "forever", if the service throws an exception; however, this particular flavor of consumer must examine each message received to determine whether or not it indicates such a problem. If you'd like to avoid putting that burden on every requestor (since most of the time, the message will not be about an exception - that's why they're called "exceptions"), here's an extension to this consumer that provides two separate callbacks, one for nominal messages and another for invalid ones:

package com.mybiz.jms.activemq.server.requestreply.requestor.async;

import com.mybiz.jms.activemq.server.requestreply.connection.AsyncConsumerConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.connection.ConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.replier.InvalidQueueRemoteService;
import com.mybiz.jms.activemq.server.requestreply.replier.RemoteService;
import com.mybiz.jms.activemq.server.requestreply.util.MessageUtil;

import javax.jms.Message;
import javax.jms.MessageListener;

public class DualListener extends InvalidListener implements AbstractInvalidNotifier {

    public void notifyInvalidRequest(Message notification) {

        processReceived(notification, true);
    }

    protected ConnectionStuff setupInvalidConnection() throws Exception {

        MessageListener invalidQueueListener = new MessageListener() {
            public void onMessage(Message msg) {
                notifyInvalidRequest(msg);
            }
        };

        return new AsyncConsumerConnectionStuff(
                        RemoteService.url, InvalidQueueRemoteService.invalidQueueName,
                        invalidQueueListener, MSG_SELECTOR, true);
    }

    public static void main(String[] args) throws Exception {

        run(new DualListener());
    }

    private void processReceived(Message msg, boolean invalid) {

        if (invalid) {
            System.out.println(
                    "---->>> " + MessageUtil.getDateTime() + " Notified about INVALID response <<<----");
        }
        processReceived(msg);
    }
}

Running this modified requestor confirms that invalid messages are received in a separate method, precluding the need to parse replies just to catch the occasional exception:

Mar 25, 2010 1:33:29 PM org.apache.activemq.transport.failover.FailoverTransport doReconnect       
INFO: Successfully connected to tcp://localhost:61616       
13:33:29.939 Request for service 'Async data request #1' has been sent...       
   Message ID: ID:localhost-34245-1269545609591-0:0:1:1:1       
   Correlation ID: null       
   Reply to:   queue://MyListener       
   OID property:   13:33:29.348       
----------------------------------------------       
13:33:29.959 Received reply 'Normal Service Reply'       
   Message ID: ID:localhost-34222-1269545245163-0:0:1:4:1       
   Correlation ID: ID:localhost-34245-1269545609591-0:0:1:1:1       
   Reply to:   null       
   OID property:   13:33:29.348       
----------------------------------------------       
13:33:32.951 Request for service 'Async data request #2' has been sent...       
   Message ID: ID:localhost-34245-1269545609591-0:0:1:1:2       
   Correlation ID: null       
   Reply to:   queue://MyListener       
   OID property:   13:33:29.348       
----------------------------------------------       
13:33:32.954 Received reply 'Normal Service Reply'       
   Message ID: ID:localhost-34222-1269545245163-0:0:1:5:1       
   Correlation ID: ID:localhost-34245-1269545609591-0:0:1:1:2       
   Reply to:   null       
   OID property:   13:33:29.348       
----------------------------------------------       
13:33:35.954 Request for service 'null' has been sent...       
   Message ID: ID:localhost-34245-1269545609591-0:0:1:1:3       
   Correlation ID: null       
   Reply to:   queue://MyListener       
   OID property:   13:33:29.348       
----------------------------------------------       
---->>> 13:33:35.957 Notified about INVALID response <<<----       
13:33:35.957 Received reply 'Exception occurred: java.lang.IllegalStateException: java.lang.IllegalStateException: NULL message received'       
   Message ID: ID:localhost-34222-1269545245163-0:0:2:3:1       
   Correlation ID: ID:localhost-34245-1269545609591-0:0:1:1:3       
   Reply to:   null       
   OID property:   13:33:29.348       
----------------------------------------------

The caveat mentioned above remains, however: since this is a shared queue destination, any consumers that are not using the same message selector as this one can compete to consume the invalid messages, subverting the intended behavior. The "correct" remedy here is to simply follow the ActiveMQ recommendation, noted above.

Sync Request w/Async Invalid Channel

2010-03-23T15:18:00.011-06:00

The first part of this series of writeups introduced a request-reply client that used the JMS 1.1 QueueRequestor to send the request, and a service that would throw an exception if the message was null or was not a TextMessage. The client will not know that the exception occurred, and since it is using the QueueRequestor, it can block "forever". Even without exceptions, blocking clients consume resources and can compromise the responsiveness of a system; while a trivial remedy to this would be to simply use an asynchronous listener, or to block-without-wait or block-with-timeout, in this case, again due to use of the QueueRequestor, a block-until-reply request is the only option. Before moving on to alternative client approaches, I'll first introduce a strategy that at least partially mitigates the problem with block-until-reply, and that can also be used with other request approaches as a mechanism for communicating exception information across the messaging middleware.

Here's an example class that blocks "forever":

package com.mybiz.jms.activemq.server.requestreply.requestor.sync.broken;

import com.mybiz.jms.activemq.server.requestreply.connection.SyncRequestorConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.replier.RemoteService;
import com.mybiz.jms.activemq.server.requestreply.util.MessageUtil;
import javax.jms.JMSException;
import javax.jms.Message;

public class BlockForeverRequestor {

    public void runBlocking(SyncRequestorConnectionStuff stuff) throws JMSException {

        // create a null text message so as to provoke an exception. While the
        // exception will indeed get thrown, the client using this approach to request
        // something will never know it - it will block forever
        Message oopsieMyBad = stuff.getSession().createTextMessage();

        // this method calls use the QueueRequestor - a block-until-reply approach:
        MessageUtil.processRequestAndReply(stuff.getRequestor(), oopsieMyBad);

        // at this point, this client is going to block forever.
    }

    public static void main(String[] args) throws JMSException {

        BlockForeverRequestor requestor = new BlockForeverRequestor();
        SyncRequestorConnectionStuff serviceQueueStuff =
                new SyncRequestorConnectionStuff(RemoteService.url, RemoteService.serviceQueueName);
        requestor.runBlocking(serviceQueueStuff);
        serviceQueueStuff.getConnection().close();
    }
}

Continuing to use the plain-vanilla RemoteServer (introduced in Part 1) when running this client results in the following output:

Mar 23, 2010 10:26:11 AM org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://localhost:61616
10:26:11.514 Sending request for service 'null'...

And meanwhile, the service has issued this complaint, but the client will not be aware of it:

SEVERE: ID:localhost-52606-1269361388958-0:0:1:1 Exception while processing message: java.lang.IllegalStateException: java.lang.IllegalStateException: NULL message received
java.lang.IllegalStateException: java.lang.IllegalStateException: NULL message received
   .....

One technique to remedy the client's ignorance of the exception is to listen on a separate queue where messages about exceptions, invalid messages, and etc. can be posted. A service that posts such messages might look like this:

package com.mybiz.jms.activemq.server.requestreply.replier;

import com.mybiz.jms.activemq.server.requestreply.connection.ConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.connection.ProducerConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.util.MessageUtil;
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.TextMessage;

// see Part 1 to examine the RemoteService class
public class InvalidQueueRemoteService extends RemoteService {

    public static final String invalidQueueName = "InvalidQueue";
    private ConnectionStuff invalidStuff;

    public InvalidQueueRemoteService() {}

    public InvalidQueueRemoteService(String theUrl, String serviceQueueName, String invalidQueueName)
            throws JMSException {

        super(theUrl, serviceQueueName);
        // use the same connection for the invalid queue as is used for the normal service replies.
        // Connections are expensive, but you can get multiple sessions out of them.
        invalidStuff = new ProducerConnectionStuff(getServiceQueueConnection(), invalidQueueName);
        System.out.println("   ...service will now send replies to an INVALID queue...");
    }

    private Connection getInvalidQueueConnection() {
        return invalidStuff.getConnection();
    }

    public void onMessage(Message requestMsg) {

        try {
            if (requestMsg instanceof TextMessage) {
                super.onMessage(requestMsg);
            } else {
                MessageUtil.examineReceivedRequest(requestMsg);
                // send a message back to an "invalid" queue. Any blocking requests
                // would still block forever. At least in this case, that requestor gets some indication
                // that something went wrong.
                sendReply(invalidStuff, requestMsg, "Replied to Invalid Queue",
                        invalidStuff.getDestination());
            }
        } catch (Exception e) {
            try {
                sendReply(invalidStuff, requestMsg, "Exception occurred: " + e,
                        invalidStuff.getDestination());
            } catch (JMSException e1) {
                throw new IllegalStateException("Multiple exceptions occurred; first " + e
                        + " -- and then " + e1);
            }
        }
    }

    protected void shutdown() throws Exception {
        super.shutdown();
        getInvalidQueueConnection().close();
    }

    public static void main(String[] args) throws Exception {
        RemoteService service =
                new InvalidQueueRemoteService(url, serviceQueueName, invalidQueueName);
        RemoteService.run(service);
    }
}

A base class that listens (asychronously) on this queue could look like this:

package com.mybiz.jms.activemq.server.requestreply.requestor.sync;

import com.mybiz.jms.activemq.server.requestreply.util.MessageUtil;
import com.mybiz.jms.activemq.server.requestreply.connection.AsyncConsumerConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.connection.ConnectionStuff;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.TextMessage;

public abstract class AbstractInvalidListener implements MessageListener {

    public void onMessage(Message msg) {

        try {
            System.out.println("Received message on INVALID queue!");
            MessageUtil.examineReceivedReply(msg);
            msg.acknowledge();
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }

    protected ConnectionStuff setUpInvalidQueue(ConnectionStuff stuff, String theQueueName) throws JMSException {

        System.out.println("Setting up 'invalid queue' to listen for exceptions, etc...");
        ConnectionStuff invalidStuff = new AsyncConsumerConnectionStuff(stuff.getConnection(), theQueueName, this);
        // need to return a different "stuff" object here that reuses existing connection but encapsulates
        // new session and destination; client should use that stuff object for subsequent invalid-queue
        // messaging
        return invalidStuff;
    }

    public void testInvalidQueue(ConnectionStuff stuff) throws JMSException {

        TextMessage testMsg =
                stuff.getSession().createTextMessage("Testing invalid queue");
        testMsg.setText("Testing Invalid Queue");
        testMsg.setJMSReplyTo(stuff.getDestination());
        MessageUtil.examineRequestBeforeSend(testMsg);
       
        MessageProducer producer = stuff.getSession().createProducer(stuff.getDestination());
        producer.send(testMsg);
        MessageUtil.examineSentRequest(testMsg);
    }
}

A concrete subclass that uses this could be something like this:

package com.mybiz.jms.activemq.server.requestreply.requestor.sync.broken;

import com.mybiz.jms.activemq.server.requestreply.connection.ConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.connection.SyncRequestorConnectionStuff;
import com.mybiz.jms.activemq.server.requestreply.replier.InvalidQueueRemoteService;
import com.mybiz.jms.activemq.server.requestreply.replier.RemoteService;
import com.mybiz.jms.activemq.server.requestreply.requestor.sync.AbstractInvalidListener;
import com.mybiz.jms.activemq.server.requestreply.util.MessageUtil;
import javax.jms.JMSException;
import javax.jms.Message;

public class InvalidListener extends AbstractInvalidListener {

    public void runBlocking(SyncRequestorConnectionStuff stuff) throws JMSException {

        // create a null text message so as to provoke an exception.
        Message mistake = stuff.getSession().createTextMessage();
        MessageUtil.processRequestAndReply(stuff.getRequestor(), mistake);

        // Though this client now listens on the invalid queue, it is STILL going to block 
        // forever since the request still gets no reply - but at least this client gets 
        // notified since it's listening on the invalid queue. At that point, the client might 
        // be able to e.g. kill a separately-launched thread that is blocking, or otherwise 
        // deal with state as needed.

        // Can we provoke a reply to the blocked service request from that point?
        // That would be ideal, since that guarantees that the service reply comes AFTER the 
        // invalid reply. But that would involve knowing the name of the temporary queue that 
        // was set as the reply-to, but that isn't set until after the message is sent - which 
        // is too late - since it's a blocking request. Once the send is done, the requestor 
        // blocks for a reply. We need a better strategy.
    }

    public static void main(String[] args) throws JMSException {

        InvalidListener requestor = new InvalidListener();
        SyncRequestorConnectionStuff serviceQueueStuff =
                new SyncRequestorConnectionStuff(RemoteService.url, 
                RemoteService.serviceQueueName);
        ConnectionStuff invalidStuff =
                requestor.setUpInvalidQueue(serviceQueueStuff, 
                     InvalidQueueRemoteService.invalidQueueName);
        requestor.testInvalidQueue(invalidStuff);
        requestor.runBlocking(serviceQueueStuff);
        serviceQueueStuff.getConnection().close();
    }
}

Here's the console output when starting this new service:

Mar 23, 2010 10:59:57 AM org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://localhost:61616
Service is waiting for a request...
   ...service will now send replies to an INVALID queue...

...and when running the requestor client:

Mar 23, 2010 11:00:28 AM org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://localhost:61616
Setting up 'invalid queue' to listen for exceptions, etc...
11:00:29.104 Sending request for service 'Testing Invalid Queue'...
Received message on INVALID queue!
11:00:29.242 Request for service 'Testing Invalid Queue' has been sent...
    Message ID: ID:localhost-52894-1269363628763-0:0:2:1:1
    Correlation ID: null
    Reply to:   queue://InvalidQueue
----------------------------------------------
11:00:29.236 Received reply 'Testing Invalid Queue'
    Message ID: ID:localhost-52894-1269363628763-0:0:2:1:1
    Correlation ID: null
    Reply to:   queue://InvalidQueue
----------------------------------------------
11:00:29.243 Sending request for service 'null'...
Received message on INVALID queue!
11:00:29.275 Received reply 'Exception occurred: java.lang.IllegalStateException: java.lang.IllegalStateException: NULL message received'
    Message ID: ID:localhost-52884-1269363597038-0:0:2:2:1
    Correlation ID: ID:localhost-52894-1269363628763-0:0:1:1:1
    Reply to:   null
----------------------------------------------

So, we've succeeded in at least notifying the client that its service request resulted in an exception; this is marginally helpful in the context of using a block-until-reply requestor. Note that the TopicRequestor is another JMS 1.1 concrete helper class, used again to facilitate a simple request-reply arrangement, but the API is the same - the request is blocking. Bottom line, we'd have better resilience - and be less coupled in either event, which is arguably one of the major benefits of a messaging approach - to use asynchronous messaging.

The next article will continue with use of the separate queue for invalid messaging, using asynchronous messaging for both that and for service requests, and evolve to ensure that invalid messages are delivered only to the appropriate requestor (since the invalid queue is a system-wide resource, shared by all requestor clients).

References

JMS Request/Reply, Part 1
JMS Home Page
JMS Downloads
JMS FAQ
JEE 5 JMS Tutorial
JEE 5 JMS Tutorial - Example Code
JEE 5 Online Javadoc
JMS 1.1 Online Javadoc
Apache ActiveMQ
Enterprise Integration Patterns - Messaging