Thursday, November 20, 2008

Nice way to tell what all you know with wordle





Tuesday, November 11, 2008

Do we really need a web service api for serving static data when we already have a good query and mining language? (SQL, XML)

Google, Amazon & eBay provide the static data by publishing through their docs.
- Location information (City, Country...)
- Categories
- Currency data
- Hierarchy information

These are the advantages that I see from the above approach. 
- Uses local CPU power, saves unnecessary traffic to the web service
- Supports online/offline scenarios
- Supports programming logic flow, since we will have the information at the compile time it's easy to code
- Solves the performance problem, what ever be the improvement network calls are always costly compared to local calls
- data model is essentially a flat data model, easy to import,save & navigate

But I guess they should be better exposed as XML rather than as HTML

The problems I see with this approach
- We need to have stringent way of updating the local cache of static code (usually stored in SQL or XML or as text file with some delimiters)
- static information that are easy to represent in code (say number is less than 10 & unlikely to change) are represented as Enums , so we will be 2 rules, some will be represented with static enums & some will be represented with static code
- There will be different rules for representing the data & we will be sacrificing the strong typing which I think is not a issue as benefits to the both parties (client & server) exceeds the pain with String based APIs.

As from the mature real SOAP based web services, it's clear that we don't need to expose API StaticCodeService() to get these static information. The demand for quality, correct designs far exceeds our capacity to create them. Web services should emerge from the needs of real applications, and that they should make common tasks ridiculously easy.

In Summary,
It's not worth wasting SOAP web service API calls for getting static information.

Thursday, November 06, 2008

Why Fluent Interface? Here are some thoughts on using fluent interface while designing APIs & their impact
Definition : In software engineering, a fluent interface (as first coined by Eric Evans and Martin Fowler) is an object oriented construct that defines a behavior capable of relaying the instruction 'context' of a subsequent call.

First philosophy,

The success of Conversation/Communication/Integration depends on the quality of signals between two entities involved & the signal is clean & is understood well. Communication is always based on context, shared context. Context also influences interpretation. For example , "use a fork" differently depending on context (Unix, dining table, Java or Ant build file). Well understood shared context improves the signal-to-noise ratio in communication it makes communication effective, expressive, easy to understand and easy to work with.
Domain Specific languages (DSLs) have implicit context, context is never mentioned, once the context is established repeating it again & again is a noise,. The end goal for involved parties in communication should be to reduce the noise to zero.
We have 2 types of DSL, Internal - based on exiisting language & external - A new language with parser & full fledged grammar, evidently external DSL is tough to implement & very effective (SQL,HTML) on the other hand Internal DSL is easy to implement especially with dynamic languages.

That's all about the boring philosophy...

The benefits,

No need to document the APIs.
- They are all self explanatory
- Examples should do all the explanation if any
- Testing/adaptability becomes easier

Less effort required to use, resulting in better economics
- There is very less chance to make mistakes
- All the extra noise is hidden which does not add domain concepts
- User has to write less & less number of lines of code, LOC most
of the time translate to number of bugs. Less the the code less the bugs

Correctly written fluent API gives the satisfaction of well written novel to
the author & the same reading experience to the consumer.

Market is always right, now all the new libraries are coming up using fluent APIs
Big software houses are putting more efforts to make code less noisy. The trend from Microsoft (C#3, IronRuby, F#...) Sun (JRuby & other dynamic language support) points towards that.

Experts think Fluent APIs are cool.
- Martin Fowler is writing a book on DSL
- Well proven frameworks like JUnit are coming up with fluent API alternatives
- Google collections, JMock, Fest, Guice... countless popular APIs are based
on fluent APIs
- Ruby on Rails Active Record is best example how a DSL can simplify the job & has successfully forced to think differently
- Market Signals shows that, best brains are talking more about DSL, functional
langauges & is definately are the way go about to develop software.
- Joshua Bloch (Google Java Architect) in his new Effective Java book talks about Builder pattern for building immutable fluent java objects.
buying more memory can be easier and cheaper than to pay someone to understand/debug code. Well written code (read fluent interface) results in good economics.

Now the real world code using Fluent APIs,

No marketing is better than showcasing real working code for technique or technology.

New Java Date APIs JSR 130:
Period thePeriod = Periods.periodBuilder().years(8).months(3).build();
This is what experts feel how the code should be written & this code is definitely looks better than java.util.Calendar,java.util.Date APIs.

Fluent way of handling XML marshalling un-marshalling:

Here we have sample example showcasing the XML usage with fluent APIs

<contacts>
<contact>
<name>praveenm</name>
<phone type="mobile">98862342333</phone>
<phone type="office">080-2344234233</phone>
<email>pm@aol.com</email>
</contact>
</contacts>

Groovy Sample:
def mkp = new MarkupBuilder()
mkp.contacts {
contact {
name("praveenm")
phone(type: "mobile", "98862342333")
phone(type:"office", "080-2344234233")
email("pm@aol.com")
}
}
Ruby Sample
require 'builder'
x = Builder::XmlMarkup.new(:target => $stdout, :indent => 2)
x.contacts {
x.contact {
x.name('praveenm')
x.phone '98862342333', :type => 'mobile'
x.phone '080-2344234233', :type => 'office'
x.email 'pm@aol.com'
}
}

Google Collections:
A excelellent example how JDK Collections can be simplified with fluent APIs
public static final ImmutableSet FAVORITE_NUMBERS
= ImmutableSet.of(2,9, 8, 15, 16, 50);

FEST Example: DSL-oriented API for functional Swing GUI testing

dialog.comboBox("domain").select("Users");
dialog.textBox("username").enterText("alex.ruiz");
dialog.button("ok").click();
dialog.optionPane().requireErrorMessage()
.requireMessage("Please enter your password");

JaxB common:

USAddress address = new USAddress()
.setName(name)
.setStreet(street)
.setCity(city)
.setState(state)
.setZip(new BigDecimal(zip));

I guess this is not a good example for fluent.

Guice:
@Override
protected void configure() {
binder().bind(IUserService.class).to(UserServiceMockImpl.class);
binder().bind(AuditInfo.class).to(DummyAuditInfo.class);
}

Hibernate:
List cats = session.createCriteria(Cat.class).setMaxResults(50).list()

DesignGridLayout:
A Fluent Layout manager

layout.row().label(label("Last Name")).add(lastNameField, 2).add(label("First Name")).add(firstNameField, 2);
layout.row().label(label("Phone"))
.add(phoneField, 2).add(label("Email")).add(emailField, 2);
layout.row().label(label("Address 1")).add(address1Field);
layout.row().label(label("Address 2")).add(address2Field);

Frameworks like Grails, JMock, JPA utilizes the fluent APIs. We also have some samples in standard JDK itself like StringBuffer, StringBuilder, ProcessBuilder etc...

These are the commonly seen techniques in Fluent APIs with Java.
- Method Chaining
- Nested Interfaces
- Builder Pattern
- static imports

And finally the problems with fluent APIs,

It's definitely not the 'wow' technique. People have been using this for a quite long time & now we have just fancy name 'Fluent API' that's it. So let's not think this is like OOP or OOAD or MDA. It's a simple programming technique for making code look better.

- DSL is about writing good essay. Programmers are not essayists
- Java is not suitable for DSL- It doesn't have closures, open classes , quite verbose & is filled badly written APIs.
- It's very difficult to get correct in developing DSLs.Fluent API can be useful for highly used API,otherwise the investment of developing may not be as effective as intended. Thus, not all API can be made fluent.
- Difficult to track down null return value issues that occur somewhere in the chain
- Difficult to handle the exceptions, especially while dealing with existing APIs having checked exceptions.
- One of my 'java friend' was not ready to believe some of the fluent samples that I showed him were actaully a java code, :-), so there is also -ve impact on readability
- Need to write more code to make code fluent.

Summary:
I strongly believe that DSL, both internal & external helps in developing better software that is easy to learn,extend, use & hard to misuse.

That's it, Hope that I was able sell fluent APIs to the new guys.

References:
Domain Specific Language Book by Martin Fowler
Domain Specific Language by Martin Fowler
DSL Boundary by Martin Fowler

Friday, October 17, 2008

I am currently working on building a fluent interface over soap toolkits. I did discussed the possibilities of rich APIs value addition. One of the problem with fluent APIs is that we have to get rid of checked exception to make it easier, but there are cases where fault containment is necessary. I am just thinking the better appraoach would be to generate the unchecked Exception counterparts for each of the fault code & provide the easier error handling.

For Example: (I have modified sample shown by the Martin Fowler), Let us assume that newOrder process will talk to 2 other web services & there is possibilities of 2 soapfault errors.

import static test.Customer.newOrder;

private void orderNew(){
newOrder() .
with(6, "TAL") .with(5, "HPK").skippable()
.withDollar(35.d) .withAccountId(23423l).withLicense("AS900980") .
.with(3, "LGV")
.priorityRush();
}
@Test
public void test(){
try{
orderNew();
}catch(GenricException exp){
switch(exp.getErrorCode()){
case SOAPFaultsCodes.E1001:
// handle the exception possibly giving more useful message or some other recovery action
case SOAPFaultsCodes.7003:
}
throw new RuntimeException("Unhandled Error");
}
}

Here again GenericExcption is abstract Exception extedning RuntimeException that is implemented by all the soap fault exception correspding to the errorcode & will be thrown by the rich APIs.

I guess with JDK7 I guess we can have String based switch() & upgraded catch block.

Well, for unit testing exception we can laverege the annotations support to assert with JUnit4
@Test(expected = E7069Exception.class)

I wanted to generate all these Exceptions automatically from soap faults document, For that I wrote this groovy script. (show casing the usage of multiline string, closure & file I/O)

constantsFile=""
converFileLineIntoException = {
sarray = it.split(" ")
exception = sarray[0]+"Exception"
errorCode = exception.substring(1,sarray[0].length())
constantsFile+="\n public static final int E$errorCode = $errorCode;"
className =
"""package com.yahoo.sm.ws.builders.exception;
// Generated code
public class $exception extends GenericException {

private String description;
private String shortDescription;

public $exception(String description, String shortDescription){
this.description=description;
this.shortDescription=shortDescription;
}

public int errorCode(){
return $errorCode;
}

public String getDescription(){
return this.description;
}

public String shortDescription(){
return this.shortDescription;
}
}
"""
new File(sarray[0]+"Exception.java").write(className)
}

def lines = new File("soapFaultList.txt").eachLine(converFileLineIntoException)
soapFaults = """
package com.yahoo.sm.ws.builders.exception;
public interface SOAPFaultsCodes {
$constantsFile
}"""
new File("SOAPFaultsCodes.java").write(soapFaults)
println "--- Gr8 I am done "

Groovy, JUnit4 & static imports just rocks :-)

Sunday, October 12, 2008

Dependency Injection with Guice, JUnit 4 & Get-Set problem:
I was evaluating Guice,JUnit-4,while trying out with these same I wrote a sample application using the same. & also tried out a possible solution for the problem with data conversion.
In a typical web application we also use heavily with get/set to convert from UI representation of object to back end implementation object. Many a times (In my experience most of the time) they are all heavy parallel structures required by framework (Like classic struts forces UI object to extend & ActionFormBean, limited data type support) or data type representation in the different UI models. With the advent of domain driven design we have more rich domain objects which usually contains many other data/behaviour which should not be or need not be exposed to the UI layer.
The problem with this conversion is that they are not only look dumb consuming lot of source code lines but they are error-prone & since there is no contract with back end we are forced to unit test the code as we cannot rely on compile time checks. I also tried a possible solution for this, 3 years back I tried out implementing this in a classic struts based web application & was successful in reducing the number of bugs.

The sample application is user management system. (Please note that the code has been written in such a way to show the usage Guice, JUnit & type safe data conversion & shouldn't be mistaken as real time design,there are no exceptions, validation etc...). It follows a typical MVC pattern followed in web applications.

Model Layer
User.java Represents a User domain object. As we notice it also contains the information that is populated through context (Logged in user, context, date, etc... usually through HttpSession or EJBContext) in the form of AuditInfo object. Here the fullName field (which doesn't make sense to UI) representing the firstName & fullName as 'firstName,lastName' in a single field. This is done to show how easily back end object can be refactored to accomadate client requirements. This can be even applied to data types. Since extracting of interface from any class is supported by all the IDEs, there is no coding effort required here.


IUserService - Exposing the functionality of user management service

UserServiceMockImpl.java - Implements the service by saving the content in a file using object serialization.



UI Layer
IUser.java - Now this is the trick we have IUser that's required by UI layer (which is always a subset of back end object) that's being implemented by the UI model object.

UserForm.java - UI bean honouring java bean spec & can extend the classes like ActionFormBean representing the HTML form in the screen.



controller
UserController.java - Controls the logic flow b/w UI & the back end. Here is the place we will introduce the dependency injection. In a typical web application audit information is captured in HttpSession object, since in test envioronment we will not be having references to HttpRequest, HttpResponse etc... we will inject those data with our Guice & also we will inject service implementation which could be changed without disturbing the other layers.



Unit test layer
MockBinder.java - As I am obsessed with fluent interface, generics & type safety, it's really enjoyable to wire up dependencies using Guice APIs rather than through XML.

UserTest.java - & now finally we have unit testing code with JUnit4 test case show-casing the usage of the applications.

------- Final Summary-------

  • Guice simplifies the testing a lot doing everything in java, If you just want dependency Injection, Guice beats all other frameworks (Spring...) hands down in ease of use

  • JUnit4 is much easy to use with annotations & now we don;t have to extend any class & don't have to write methds with test...(). I really liked the way we can test the Exceptions

  • Adding interface to both backend & UI objects we can get rid of get/set noise.
    Although it makes(or forces) back end to be aware of UI & other form of clients, in some cases (like we we use ORMs instead of JDBCTemplate) one time conversion logic shifts from UI to backend & it might put more development load on persistent layer team. My guess is these objections can be ignored because of the benefit of to cleaner & less error prone code. In case of distrbuted enviornment all the extra fields can be set to null to reduce the payload, but anyway number of rows are much more important than the number of columns in deciding the data size.

    Hope that this sample application help new comers to understand/appreciate the value of Guice & JUnit4 libraries.

    References;
    Guice Dependency Injection Framework from Google
  • JUnit 4 - with Java5 features & is quite different from older versions.


    Thursday, August 14, 2008

    What value could custom client SDK can provide over SOAP toolkit(s)?

    1. Single library to deal.
    Developers need not have to spend time in knowing or generating stubs & which framework to choose for for the same (XML marshaling/un-marshaling).This single library client can hide this implementation methodology by exposing easy to use APIs & hiding these techniques behind the scene using some kind of dependency injection APIs. (Jax-ws, Axis, XFire, Xstream so on...). The single library can be collection of core + optional service libraries in the same fashion provided by the Spring Like spring-jdbc.jar, spring-jndi.jar so on... with spring-core.jar as kernal providing infrastructure for all other services.

    2. Handling backward compatibility issues
    Carefully designed client SDK can solve backward/upward compatibility issues without compromising on the design elegance, If we don't want to use new SOAP version, it should be OK to use existing version most of the time. Client SDK can be designed in such a way user is shielded from SOAP schema changes that does not require recompilation or rebuild what so ever.For using new APIs user just has to drop in new library(jar) that's it. Well designed Mock objects can even save the need to goto sandbox during development.

    3. Easy to use fluent APIs.
    Domain specific language (DSL) design can bring lot of changes & provide better user experience with fail safe APIs. Fluent APIs can bring lot of +ve changes. For me this is the single most important feature.
    For example:

    MailBuilder.mail().
    .from(praveen.manvi@yahoo.com)
    .to(pmanvi@aol.com)
    .withSubject("Fluent Mail API")
    .withBody("Fluent API & DSL can make developer life simpler")
    .attach(new File("c:/test.txt")).
    .send();

    is definitely much easier to use. This is ok for 90% of users to start, if we have hooks to get into Transport & other Java Mail APIs,it can simplify the way we code & importantly understand without compromising the power for advanced users.

    4. Meaningful Exception hierarchy stack for better recoverable actions
    Users don't have to deal with SoapFault Error code. Just like Spring DAOException hierarchy translating the SQLException & SQL error codes giving meaningful RuntimeException (even exteding it to hibernate, ibatis,jpa) API providers can make developer's life easier. Like SQLException & SQL error code APIFaultException & fault code is difficult use & introduces lot of boiler plate code (because of checked exception) without providing the meaningful recovery options. Hibernate, Spring & C# have proved that checked exception is like communism, works best in theory only.

    http://static.springframework.org/spring/docs/2.5.x/reference/dao.html


    5. Automatic logging soap XML handshake (Making this switch on & off on need basis) between client & server. This comes as very handy & we can even make them asynchronous & log this information to to different data sources like file & database by providing proper hooks. This automation improves the performance as well as avoids the need to develop auditing APIs.

    6. Most of error handling/validation can done by the SDK framework, there is very less chance for error requests coming to server improving server performance. 99% of the time server will deal with valid requests

    7. Meaningful defaults for all the APIs & actions, can reduce initial learning curve & improve the perception. It can help user to get into the the details step by step instead of throwing him everything upfront.

    8. Can enforce the best practices. & develop new interfaces that are very difficult to achieve with simple plane SOAP.

    For example a utility can be developed to export the data in chunks from large files like Excel sheet, a csv file or a PDF., as the sending large files & batch processing with SOAP is not mature (mtom...)

    9. Encourage more developer community participation. Easy to use APIs improves the possibility of
    developer to develop new tools & application using the SDK especially desktop tools.

    10. Get rid technical limitation of programming technique. SOAP inherintly not object orieneted & it's procedural style of programming is not what a modren developer would like to work.

    I just evaluated different SDK provided by various well known API providers & most of them don't really provide above facilities & there is very good opportunity of 3'rd party developers to fill the gap over here.

    Same concepts can be carried out for developing C# APIs. As for as dynamic langauges(Groovy,Ruby, PHP) concerned I guess they don't need any SDK, they are smart enough to deal with XML directly without any code generation.

    Many a times developers need to design wrappers over SOAP apis -- that involves substantial effort. Client SDK can save the time & money here. Like Spring removed the necessity of having in-house framework for standard java enterprise project by not tying the implementation any specific technology but to the interface & also DSL developed with business in mind can make writing code like writing an essay.

    References:
    http://developer.ebay.com/DevZone/XML/docs/HowTo/BestPractices/JavaToJsdkToSoap.html - eBay SOAP SDK sample
    https://www.paypal.com/IntegrationCenter/ic_sdk-resource.html - Paypal API
    http://code.google.com/p/fluentmailapi/ - Fluent Mail APIs
    http://www.javaworld.com/javaworld/jw-08-2008/jw-08-dsls-in-java-3.html - DSL in java
    http://www.toolsforteams.com/code/facebook-api/ - Facebook APis
    http://code.google.com/p/java-twitter/ - Twitter APIs
    http://www.oracle.com/technology/tech/soa/mastering-soa-series/part4.html - Rich UI for web services
    http://www.javabeat.net/articles/29-introduction-to-google-guice-5.html - Guice Samples

    Thursday, July 24, 2008

    Some wise words on design - I recently was listening to a presentation in InfoQ on design principles of successful large scale enterprise financial application based on Spring & Hibernate. I noted some points while listening to that. Although these are well known established principles I thought it's easier to refer & document when ever we need to come up new design document.
    • Minimize data movement
    • Task Parallel execution where ever possible
    • Physically partition of data
    • Optimized reads & writes of volatile data
    • Minimize contention -- Different connection pooling
    • Asynchronous Decoupling
    • Complete Business Logic performed by data
    • Caching frequently accessed data
    Like all popular scalable architecture (eBay, Yahoo) the transaction (distrubuted) & state management were simple (stateless & transactionless for 90% of operations)
    JDBCTemplate - fantastic example of Dependency injection
    Spring promotes good design - modularity, loose coupling separation of concerns
    Testability provides better confidence for refactoring
    Hibernate promotes domain driven design - 2'nd level caching is gr8

    Contents of design document can be divided into following categories:
    • Architectural review
    • Jumpstart programs
    • Deployment strategy
    • Platform change
    • Proof of concept
    • Performance tuning
    • Development
    Capacity Planning questions:
    • How much memory footprint the application requires
    • How many threads the application components need
    • How many connections each application has to handle
    • How many system resources like file handles are required
    • How much parallelism is attainable in a single process
    • How many CPUs can be leveraged at the same time from within the same hardware
    • What are the synchronization primitives available in case of multiple processes and nodes
    • How we coordinate shared accesses and leases
    • The allowable number of nodes that a unit of application job can cross, taking into consideration the performance criterion
    I also listened to Joshua Bloch's presentation on design guide lines, Joshua Bloch is currently Java Architect at Google & has great influence on Java's future. He also influenced (JDK5) & wrote many JDK classes as distinguished Engineer at Sun. His assertions carries lot of weight.
    Importance of API design for internal and external use.
    • Public APIs are forever - one chance to get it right
    • Good code is modular–each module has an API
    • Thinking in terms of APIs improves code quality
    • Easy to use, even without documentation
    • Don’t let implementation details “leak” into API
    • Make classes and members as private as possible
    I guess the most important one was "when in doubt, don't provide the API, leave it".

    "There's a natural law in programming language and API design : as backwards compatibility increases, elegance decreases." - Bill Venners
    So be minimalist,
    because of the compatibility requirement, it's much easier to put things in than to take them out. So don't add anything to the API that you're not sure you need

    Sunday, July 13, 2008

    Bio-Rhythmic Cycles - I got curious about bio-rhythmic cycles & wanted to try out the the results on my life.

    For those who don't know what bio rhythmic cycles are, here is explanation:

    The theory of biorhythms claims that one's life is affected by rhythmic biological cycles, and seeks to make predictions regarding these cycles and the personal ease of carrying out tasks related to the cycles. These inherent rhythms are said to control or initiate various biological processes and are classically composed of three cyclic rhythms that are said to govern human behavior and demonstrate innate periodicity in natural physiological change: the physical, the emotional, and the intellectual (or mental) cycles. Others claim there are additional rhythms, some of which may be combinations of the three primary cycles. Some proponents think that biorhythms may be potentially related to bioelectricity and its interactions in the body


    I wrote a sample application in swing, which basically takes the date of birth as input & generate the charts.


    Here is the sample screenshot, I used JFreecharts & some Date manipulating APIs to create above application. BTW is there any way I can show applet in this blog? Please let me know.

    The results were interesting, Most of the events in my life had some kind of correlation with the above graph & explanation:

    Physical cycle - 23 days; coordination strength well-being

    Emotional cycle - 28 days; creativity sensitivity mood perception awareness

    Intellectual cycle - 33 days; alertness analytical functioning logical analysis memory or recall communication

    I also tried correlate our god Sachin's best innings with these cycles & surprisingly, cycle behaviour used to match with his best time :-)

    Anyway I will try out with some other important figures before coming to conclusion these cycles have any impact or it's just a humbug. BTW, I am little skeptical about all astrology stuff.

    So I am planning to develop a RESTful sample as well sample mobile application accepting date of birth & generating the charts possibly using Google charts instead of Jfreecharts

    Wednesday, July 09, 2008

    GeoNames.org - provides geocoding services. GeoNames provide a huge databases of place names (with coordinates) organized in a hierarchy (eg. country/state/region/province/etc...). Also there is a query that allows to retrieve the geo tags within a range from a specific point. Geonames is integrating geographical data such as names of places in various languages,elevation, population, alternate names, Administrator division type,continent, Address,timezone & weather observation.

    Geonames provides RESTful based webservices with both XML & JSON rendering. These seems to defined by seasoned developer & have Style enum (Short,Medium,Long, Full) for defining result verbosity & many such developer friendly criteria APIs.

    For off-line applications, geographical database is available for download free of charge under a creative commons distrubution license & can leverage the service by downloading all the data. Geonames.org claims that it has already serving up to over 3 million web service requests per day & has the ability scale.

    I guess applications (especially free & open source ones) have excellent opportunity to integrate with this service. Web applications can be relieved from maintaining the huge database of geo services. They can just store the geo-ids & can rely geonames to fetch the details. Hierarchical data is always to difficult store & fetch in standard RDBMS & now there is no need to maintain this data. GeoNames includes the reference implementation in all popular languages. I just checked the java implementation & it was not that good. It could be have been made more easier (or fluent).

    The important point to note is that high-profile users of GeoNames include Slide.com & LinkedIn.

    Reference
    http://www.geonames.org/export/ws-overview.html
    http://www.geonames.org/export/place-hierarchy.html
    http://en.wikipedia.org/wiki/GeoNames
    Bookmark and Share