Design Methodologies - where do you want to spend your money?

I recently revisited articles on design methodologies, specifically TOGAF (The Open Group Architecture Framework) and Agile methodologies. Initially, I thought that both approaches contradict each other and so could not be used together. However, like writing software you combine best practices from different methodologies that meet your project requirements, improving the chances of a successful project.

I'm not going to try and pretend to be a TOGAF professional, I have an awareness of the framework and have never used it end to end. I believe that if you are going to design software then you should at least be aware of frameworks such as TOGAF and alternatives. A good start on agile is Martin Fowler "The New Methodology" and "Can SOA be done with an agile approach".

TOGAF combined with Zachman framework is a top down design methodology and provides an excellent mechanism for eliciting requirements and providing a framework for communication in a common language, but as Fowler states big up front design can become very expensive. Also, it can lead a project to not deliver what is required by the business when the system goes live, rather it delivers what was required initially. The difference may well be dramatic.

A good reason for top down design is to do with procurement and the need to identify appropriate commercial products to use, a focus of TOGAF. Large organisations want to outsource at fixed price and the vendor organisation doing the work wants all the details up front to have the best chance of delivering the project at a profit. Again, Fowler makes a good point about the "Adaptive Customer" and how customers need to be adaptive in the way they engage with vendors. Although from my experience of being both the vendor and using vendors to do work there is never an ideal approach. A vendor wants to make as much money as possible while the customer wants to control the scope and spend as little as possible.

So where do you want to spend your money?

Up front with lots of design, then on lots of change requests and the end result will be lots of documentation that in theory should reduce the maintenance overheads, keeping the future costs down.

The alternative is relying upon highly skilled and talented  software engineers who may or may not use a silver bullet, document the code or write any design material thus making maintenance harder for someone un-familiar with the code.

I have seen both top down and bottom up methodologies succeed and fail to varying degrees. It is my opinion that success is based upon good judgement and adaptability at the beginning and during the project on the part of everyone involved.

Combining the requirement analysis and control features of TOGAF with an Agile methodology, and the use of PRINCE project management looks like a good combination as long as you are adaptable in selecting the parts that make most sense for the project you are working on.

Altio support for composite applications

In recent blog entries I have used the term mashup to describe building applications from reusable components, reading recent blogs and press articles I should have used the term composite application. Back in June 2007 I read the specification for Service Component Architecture and discussed this with John Young who was at the time the CTO at Mantas Inc. John had ideas for enhancing the Altio architecture by making it more modular, beyond the use of linked applications in AltioLive 4.x. Reusability is something that I strongly support, but at the time could not see a fully working solution being implemented in AltioLive 4.x. Re-engineering of the AltioLive user interface to use Java Swing has made it possible to enable functionality to create reusable components.

Composite Applications

SOA is effectively a composition of services supporting business functionality. The missing functionality from most descriptions of composite application architecture is the user interface.

It is my belief that the only way to achieve true reuse of functionality within the enterprise is to consider vertical slices from the user interface to the data, identifying the reusable components within different verticals. This will enable business focused,  reusable composite components that should be easy to test and implement. This in turn will mean the enterprise can achieve rapid turn around of business solutions to a high quality standard.

Attributes of composite application framework

In a number of recent discussions and news articles it appears that there is a resurgence of the term composite applications, Microsoft appears to be particularly active in this area with some very good articles on what a composite application framework should deliver, a summary of which is:

Alignment

Composition must promote alignment among key stakeholders. This could be internal alignment (align different groups within the enterprise) or external alignment (align with suppliers, customers, and partners).

• Solution-provider perspective: Solutions should align existing IT assets with the needs of business owners.
• Solution-consumer perspective: It should be easy to assemble composite applications that align groups within a company, and also across organizational boundaries. Frequently, this will require applications that support cross-functional processes and enable collaboration.

Adaptability

Composition must reduce time-to-response for the business when markets change.

• Solution-provider perspective: The platform for composition should provide adaptive range. This means that it should be easy to externalize points of variability within the solution, so that those artifacts can be changed easily without ripple effects across the rest of the solution. For example, this might mean changing business rules, user interfaces, and business processes.
• Solution-consumer perspective: It should be easy to reconfigure composite applications when the business must respond to changing market conditions.

Agility

Composition should reduce the time-to-benefit of enterprise applications by reducing the time-to-deployment of end-to-end solutions, reducing development and deployment costs, and leveraging best practices. Also, when the business must respond to changes quickly or handle external disruptions smoothly, it should be easy to modify composite applications to accomplish this.

• Solution-provider perspective: Assets that make up a composite application should leverage established industry standards that make the solution quick to build and easy to assemble together.
• Solution-consumer perspective: A deployed enterprise solution should enable quick decision making by business decision makers through real-time data flows and business-intelligence tools in the context of the automated process

The full Microsoft article can be found at http://msdn.microsoft.com/en-us/library/bb220803.aspx.

For me, the grand vision of composite applications in the enterprise would be for development teams to create products that aligns with business requirements. This would be achieved using a framework that enables the attributes of adaptability and agility described above. The products would contain artifacts for servicing data requests and user interface artifacts that understand how to interact with the data requests.

In business terms there would be products or composite components for CRM, Billing, HR etc. Each of these composite components would be loosely coupled to the underlying technology supporting the business process to enable rapid change in technology requirements to support changing business demands. Development teams would be responsible for creating the components, but solution teams would be responsible for putting together the final application using the appropriate composite components.

Altio architecture and AltioLive as a composite application framework.

The introduction of Composite Service Requests to the Altio architecture in AltioLive 5 and the ability to create modules for related groups of services was the first intentional step towards making the Altio architecture a framework for composite applications. AltioLive 5.2 will see the introduction of composite components that will allow user interface functionality to be packaged as a widget for re-use across many applications.

It was never an intentional path for the Altio architecture to deliver a support framework for composite applications, the roadmap has arisen from business demand. This makes the Altio architecture and AltioLive 5.2  product extremely powerful tools for implementing agile solutions which are adaptable and aligned with business requirements.

I have yet to decide if there is a demand for Altio to provide the ability to interact with OSOA definitions, the first steps will be to provide better integration with XSD's.

Conclusion

It is my opinion that the days of implementing tightly coupled user interfaces is nearing an end. Composite components make the following quality attributes - portability, maintainability and usability far easier to achieve, whereas a user interface that is tightly bound to a specific solution will normally take longer to implement and typically requires far more work to modify. Although, it often takes the creation of a specific user interface solution to identify the creation of reusable components. It is then important that a tool is used that allows components to be created from the different parts of the specific user interface.

Bibliography

OSOA, Service Component Architecture http://www.osoa.org/display/Main/Service+Component+Architecture+Specifications

Micorosoft, Composite Applications, http://msdn.microsoft.com/en-us/library/bb220803.aspx

Wikipedia, Service Component Architecture http://en.wikipedia.org/wiki/Service_component_architecture

EDS, Composite Applications and Portals http://www.eds.com/services/compositeappsandportal/

Bill Burnham Blogs, Software's Top 10 2005 Trends: #8 Composite Applications http://billburnham.blogs.com/burnhamsbeat/2005/03/softwares_top_1_1.html

 

(This article was originally posted to http://www.altio.com )

AltioLive enabling the enterprise mashup

The AltioLive product provides both server side and client side architecture. This enables a secure load balanced mechanism to deliver data to and from a client. The client can be a user interface or another system.

What does this have to do with AltioLive? It has been the belief at Altio for a long time that a mashup does not have to be visual, but can be associated with data as well. For effective delivery of business solutions a product needs to be able to deliver data from multiple originating sources and the user should not need to know or understand the source of the data. AltioLive provides composite components to create visual business solutions and on the server side aggregated data feeds, this provides an effective mechanism to deliver browser based applications that interact with data from multiple sources.

A recent article in SD Times by David Linthicum provides a good overview of this, with the visual mashup described as

"the ability to change the manner in which a visual interface behaves by mashing it up with other content or services"
(Software Development Times, June 15 2008, page 37)

The article describes a non-visual mashup as 

"the mashing up of two or more services to create a combined application, or integration point to service a business process".  (Software Development Times, June 15 2008, page 37)

AltioLive composite controls enable a designer to take several existing components and package them together as a new business component or enterprise mashup widget. This means a library of re-usable components can soon grow and result in reduced development time for future solutions.

The AltioLive aggregated data feed provides a simple yet powerful workflow for retrieving data from disparate data sources. It is possible to use data attributes from one source as a parameter for the retrieval of data from another data source, execute multiple request in parallel or sequentially and stop execution if a failure occurs. This is the non-visual mashup.

Whatever you want to call this approach the ultimate aim is to deliver an application that makes the end user more effective in delivering business benefit as quickly as possible, without losing sight of maintainability.

The key factors of SOA/enterprise mashups as described in the article are:

• the ability to place volatility into a single domain, thus allowing for changes and for agility
• The ability to leverage services, both for information and behavior.
• The ability to bind together many back-end systems, making new and innovative uses of the systems.

This is something Altio continues to deliver through the AltioLive product as innovative ideas.

 (This article was originally posted to http://www.altio.com )

RIA + SOA – The AltioLive Solution

I finally managed to get around to reading magazines I picked up at JavaOne and one article stands out for me. SOA World Magazine, June 2008 (http://soa.sys-con.com/read/513263_2.htm) contains an article called "RIA + SOA : The Next Episode, Building next-generation web platform" which describes how RIA and SOA are shifting web applications architecture back to Client Server from Model View Controller (MVC). So I've decided to pick up on this article and analyse it in the context of AltioLive which uses a far simpler implementation methodology – basically all the hard work is done for you.

Complimenting Technologies

I believe Rich Internet Application (RIA) frameworks provide a complimentary technology solution to enabling the user interface for Service Oriented Architecture (SOA). To quote the SOA World Magazine Article

in the standards-based world of HTML, CSS, and JavaScript, RIA developers have to assemble multiple third-party libraries

to build rich user interfaces. AltioLive provides a single framework to enable the rapid development of rich user interfaces and provides a loosely coupled server side framework to make use of SOA and legacy data sources. This enables an Agile development approach to be used to deliver a user front end that can be adapted quickly to meet user requirements.

Poaching the headers from SOA World Magazine

This section takes the headers from the article and replaces the list of possible HTML, CSS and JavaScript options with the simple AltioLive framework implementation. In my opinion having a choice of different frameworks and custom javascript is a maintenance nightmare and is something AltioLive architecture tries to avoid.

1. Design the "look" of the application.
   For a complex solution requiring branding then a graphic designer can create and define an AltioLive skin. Skins provide the look and feel of borders, scroll bars etc. no need to understand HTML, CSS. For business developers who just want to get on and provide end users with a working system this saves a lot of effort without the need to think about colour codes and styles etc. The design of the user interface is a simple case of dragging and dropping widgets, setting properties and binding them to data.
   
2. Integrate Widgets.
   If a component is not defined as a standard control or business widget (AltioLive Composite Control) then a Java developer can create a new Swing or JavaFX widget by extending the base AltioLive custom control or extending existing controls.
   
3. Add dynamic behaviour to the user interface
   see next section
   
4. Consume Services
   No need to do anything in AltioLive, dynamic user interface interaction with the server comes for free. Plus with AltioLive you get context based routing, ability to poll database tables or HTML service to simulate real time updates to the client. If the infrastructure is available then simply subscribe to a message queue using JMS service requests.
   Not to forget the ultimate aim of connecting to a SOA web service – then just use the Web Service Wizard to create a SOAP service request. No complicated API's to understand and no code to implement, all the business logic is either in the web service or the SQL statement. Validation and data manipulation can all be done in the AltioLive user interface if necessary.
   
5. Create Services
   If the server side business logic does not exist then some hard work will be required. Web Services or alternative business services will need to be written, if a short term solution is needed then it may be possible to use AltioLive JDBC Service Requests to retrieve and insert data while putting logic into the user interface.
   

The article goes on to discuss the need to "Provide an Open Widget Framework", I feel that this is possibly the most important point. The industry needs to have standards that enable Java Applets, Microsoft Silverlight, Macromedia Flash and AJAX widgets to communicate effectively so that a widget that meets the user's needs can be selected regardless of the underlying technology. The key to enabling this standard will probably be JavaScript as it is not tightly coupled to any single vendor solution.

Provide an Integrated RIA Programming Model. This integrated model exists if you use AltioLive. The primary focus for AltioLive is on creating a user interface as simply, quickly and cheaply as possible – through drag and drop and setting properties. The source of the user interface is stored in a plain text file as XML and so can be manually edited if there is a real desire to do so. The drawback of this is that developers can't get their hands as dirty as they may like to. If, as a developer, you really want to get into the depths of the code then a developer can write their own widgets, or focus upon the business logic at the server side. I believe Todd Fast is right when he made the statement "Applications for the masses by the masses: why engineers are an endangered species" (http://www.regdeveloper.co.uk/2008/05/09/developers_endangered_species/ ), the user interface should be about making it easy for the user. Although, I believe there will remain the need for software engineers but their focus should be on providing frameworks and the enablers for the masses to easily create applications.

Provide an Integrated Services Platform. Again AltioLive meets the need to have the following attributes:

• Support for creating services in any programming language
  
• Seamless interoperability between the RIA and SOA tiers
  
• Ability to consume local mock services
  

The AltioLive Presentation Server provides the mechanism to interact with server side components and allows for JMS, JDBC, SOAP, HTTP connections. This means it is possible to make use of direct SQL connections or to write a web service using .Net and call it through the SOAP or HTTP request. While AltioLive is primarily a Java technology it's loosely coupled framework doesn't make Java based services mandatory.

As mentioned through this document AltioLive provides a simple approach to create a service request using one technology and allows migration to another with little or no impact upon the overall application architecture, thus providing seamless interoperability between the RIA and SOA tiers.

Mock services have been a part of AltioLive since its inception. The AltioLive IDE allows a service request to be created with example response data built in, more complex solutions can be created using Composite Service Requests so that different responses can be provided based upon input values. When testing the user interface it may be preferable to replace heavy weight services with a XML file that contains the expected response and then use a HTTP service request to retrieve the data from the XML file. The HTTP service request can simply be replaced with the request to the fully implemented business logic later in the implementation.

Conclusion

The business challenge of moving to SOA

I have yet to work with an organisation that has implemented a SOA architecture from scratch as a green field project, in my experience there has always been legacy integration required to deliver the final solution.

AltioLive overcomes this by allowing the implementation team to expose AltioLive Service Requests. An AltioLive Service Request exposes the same interface regardless of the underlying technology that the data is being retrieved from or written to. This means that an implementation team can initially create a JDBC connection direct to a database and then when a full SOA architecture exposing web-services is in place then the service request can be changed to a Web Service request without any changes to the user interface. This approach enables the rapid development of the user interface required to meet business needs and then a more formal approach can be used to deliver a well defined and structured SOA.

And what about the Enterprise Service Bus (ESB)?

Well using AltioLive the move to a ESB is achieved by modifying the AltioLive server side request to use message queues. If the solution implemented using AltioLive has been well designed then there will be no changes to the XML structure passed to or from the user interface, and now a system will be able to push updates from the message queue to the user interface without any major system changes to the presentation layer.

New acronyms for the IT industry

I'm beginning to think that everyone who works in the software industry applies Object Oriented (OO) methodologies to everything. For example in 2001/2002 we had Service Oriented Architecture (SOA) and since 2006 there has been talk of Web Oriented Architecture (WOA), when these acronyms first appear there is normally little substance to how technology supports the theory, Web 2.0 being a prime example as it seems to be mentioned for anything associated with the Web (IMHO).


I first saw WOA mentioned in 2006 on a zdnet blog titled called "The SOA with reach: Web-Oriented Architecture" it was posted on the 1st April so I had to take a deep breath and make sure it wasn't a joke. My point is that WOA is just an extension of SOA, and isn't something newly invented, in my opinion it is just the natural progression - and shows how the IT industry can adapt to business or user demands, or just to prove that something new can be invented. Unlike technology specific trends like Java, Groovy, JRuby etc which are created to improve the original technology they either replace or are based upon SOA and WOA acronyms are labels for theories and practices. The benefit of these labels is that they provide a focus for identifying the key requirements that make up the theory, and let people in the industry categorize and know what is being discussed.


So why has it been so difficult to identify the requirements for labelling a Web 2.0 product/solution or component (WIKI entries seem to debate what Web 2.0 is, rather than provide a specific definition) . I believe it is because Web 2.0 is a social phenomenon and has been driven by business or user needs to have information in a easily accessible format and to be able to configure that data to meet their own needs. To enable this requires rich user interfaces provided through Rich Internet Applications (RIA).



More acronyms evolved from SOA:

• User Oriented Architecture (UOA), not sure this exists yet. If it did there would be a close alignment with Business Oriented Architecture. The basics are that users would have maximum flexibility to work with UI widgets and data objects to generate screens how they want them. This could be viewed as an extension to Business Intelligence (BI).
• Business Oriented Architecture (BOA) makes use of BPM and SOA to provide flexible and scalable systems which will enable an organisation to adapt quickly to a changing market place. I believe the driver here is senior executives and the IT department trying to meet business requirements.

Of the two I believe User Oriented Architecture is the most powerful. It can provide great benefits through empowering end users to access data on demand. It could also cause a lot of damage to a business by users manipulating data in an incorrect manner either intentionally or unintentionally resulting in incorrect decisions being made.

Is Web 2.0 just User Oriented Architecture or will Web 3.0 provide this? I believe the answer is no, if User Oriented Architecture were ever to become an adopted term I feel there would be a demand for highly configurable and yet very simple user tools (NOTE - I'm not talking developer IDE's here). These tools would initially be aimed at power users but would eventually reach into the mass market, and would not require in-depth code or development experience, the first steps are mashups or Web 2.0 components and tools such as Yahoo Pipes.

In my opinion the Enterprise is lagging behind in adopting technologies for end users, and in some cases for good reasons. There needs to be some control over how data is accessed and manipulated otherwise there is the risk of not knowing if data is "a fact" and what data has evolved from a mashup of facts to deliver what people want to hear.

Anyway back to where I started Object Oriented Design and Programming are very powerful but if you keep extending the original concept(object), things just end up becoming overcomplicated and over used so diluting the original purpose. Do we need to re-factor some of the terms used in the IT industry to make things simple and more maintainable?

References

• http://www.webware.com/8301-1_109-9923360-2.html?tag=blogFeed
• http://www.gartner.com/DisplayDocument?doc_cd=114358
• http://blogs.zdnet.com/Hinchcliffe/
• http://www.bptrends.com/publicationfiles/TWO%2003-07-ART-ABusinessOrientedArchitecture-Gilbert-Final.pdf
• http://pipes.yahoo.com/pipes/
• http://en.wikipedia.org/wiki/Web_2.0
• http://en.wikipedia.org/wiki/Web_3

Using complex web services in Altio

Since 2003 AltioLive has had the ability to work with SOAP web services but we only have simple examples that use primitive variables. Recent professional service projects have required the use of Web Services which use complex input and output data types so I thought I would make some notes here before a more formal document is produced.

AltioLive was developed to make integration with SOA as simple as possible, but because of the broad set of functionality available in AltioLive it is not always obvious how a certain implement a specific solution. This note will highlight how to use AltioLive to work with SOAP messages and WSDL's.

What is a complex web service?

When I mention a complex web service I mean one that takes complex XML as input and returns complex XML. This is predominant now in the Enterprise where .Net and JAX-WS object serialization makes it easy to transform objects to and from XML, resulting in deeply nested XML hierarchies.

Listed below is one approach to using a complex web service in AltioLive, the default approach is to use controls mapped to parameters in the service request - this is the default for AltioLive.

1. Generate a template of the XML structure for use by Altio Screens. The template XML can be generated using Static Data, AltioDB, or a request to a template object.
2. Map screen objects to the XML template
3. Submit the XML to a SOAP service request.
4. Map the response data to the required location

The Template

There are several options for providing the template of the complex XML. The general principal is that the application requires the XML structure that is to be passed to the Web Service, and the controls will be mapped to this XML structure.

Probably the simplest solution is to create a XML file containing the template and use this in AltioDB or as a file on the server which can be called from a HTTP request. Then create a HTTP service request to retrieve the template XML.

NOTE: Make sure the data keys are correct, as this will be a common reason why the data does not display correctly. This will be especially important for the response data.

The SOAP request

Using the WSDL wizard is probably the simplest way of creating the required SOAP request:

1. Open AltioLiveApplication Manager
2. Select WSDL from the service request types on the menu
3. Enter the URL of the WSDL that declares the SOAP service you which to use
4. Click "Retrieve Operations"
5. Select the "Operation" you want to use
6. Click "Use Selected Operation"

These steps will create a new SOAP Service Request.

Because the service request will be using complex XML structure to send and receive data then the "Parts" section of the Service Request will need to be modified. By default AltioLive will create a placeholder for the data that will be passed from the client to the request, as shown below:

<tt:create xmlns:tt='http://www.bbb.com/reports/ReportInstanceAdmin'>  
<tt:ParamData xmlns:tt='http://www.bbb.com/reports'>
    <tt:Schedule>
      <Recurrence>
        <DailyRecurrence>${client.DAILYRECURRENCE}</DailyRecurrence>
        <WeeklyRecurrence>${client.WEEKLYRECURRENCE}</WeeklyRecurrence>
        <YearlyRecurrence>${client.YEARLYRECURRENCE}</YearlyRecurrence>
        <MonthlyRecurrence>${client.MONTHLYRECURRENCE}</MonthlyRecurrence>
      </Recurrence>
    </tt:Schedule>
    <Params>
      <tt:InventoryParams>
        <ReportTitle>${client.REPORTTITLE}</ReportTitle>
        <TitleOpt>${client.TITLEOPT}</TitleOpt>
        <ReportCode>${client.REPORTCODE}</ReportCode>
        <Uid>${client.UID}</Uid>
            </Params>
    <ReportInstance>
${client.REPORTINSTANCE}</ReportInstance>
  </tt:ParamData&gt
;</tt:create>

As the XML structure will be provided through a template the XML shown above can be replaced with:

<tt:create xmlns:tt='http://www.BBB.com/reports/ReportInstanceAdmin'>${client.PARAMS}:xml</tt:create>

The parameter reference ${client.PARAMS}:xml informs AltioLive to process the parameter as a encoded XML block and will de-encode the XML when the WebService operation is called.

Processing the response

By default Altio will generate the response template using ${response.result}, this works fine for simple WebServices but as the scenario is using complex XML then the response body is the important part of the SOAP envelope and so the syntax to retrieve the message content needs to be ${response.body}, otherwise Altio will report that no XML was provided in the message.

To implement this change edit the SOAP Service Request. The "Response" tab contains a field called "Literal XML string", this defines the structure of the XML that the results of the SOAP request should be placed into. For this example use the default value but replace the ${response.result} value with ${response.body}. So that the "Literal XML string" looks like the following:

<DATA><SOAP><execute>${response.body}</execute></SOAP></DATA>

Client Action

The final part to using a complex web service is to implement the client logic. The first step is to retrieve the template XML, this is done through a simple request to the Service Request that will return the template XML. The second step is to implement the logic to pass the data to the SOAP Service Request that will execute the SOAP operation, the syntax of the parameter to be passed to the SOAP Service Request is shown below:

PARAMS='eval(escape(xml-string(/ReportParams/*)))'

• xml-string() – serializes the XML element into a XML string.
• escape() – performs a HTML escape of the XML string to ensure correct transmission from the client to the server.
• eval() – informs AltioLive that this is a function execution block and that the content needs processing.

The "Parameter source" property of the "Server request" action block needs to be set to STRING, otherwise AltioLive will expect to find a control called PARAMS and use the content of the control for the XML structure.

Aside from the service request execution you will need to map controls to the XML.

Conclusion

The use of a Template for working with a complex XML structure is an alternative approach to using controls to pass the required data as individual fields. The benefits of using a template is that it allows the XML block to be manipulated by many screens and allows the XML to be built up from multiple sources if necessary.

The negative aspect of using template XML might be in migrating to a different technology, and example would be moving from a SOAP message to a REST format where parameters would need to be passed rather than a complex XML structure. In my opinion if the request is very complex then a SOAP message is probably far more effective and manageable than a equivalent REST message.

The AltioLive online help provides further details of using WSDL's and SOAP Service Requests, and if you are working in a SOA environment it is worth familiarising yourself with different techniques for interacting with SOAP message.