Friday, May 30, 2014

Transforming Domino into a Social Collaboration Platform, Part 2B - Experimentation

Transforming Domino into a Social Collaboration Platform, Part 1 - Where Do We Begin?
Transforming Domino into a Social Collaboration Platform, Part 2A - Experimentation

This is the second part of our experimentation portion of the series. In Part 2A, I talked about the different experiments that we did for creating web pages for our social collaboration applications and the design of the back-end database.  In part 2B, I will be talking about the application structure and business process and workflow management that you need in order to have a successful social collaboration tool. 

As I mentioned before, there were four areas of technical challenges that we had to overcome.
  • Web Interface Generation
  • Database Infrastructure
  • Application Structure
  • Business Process and Workflow Management
In this blog we will cover the last two.

Application Structure
Traditionally with Domino applications, an application consist of one or more Domino databases.  We build the forms, the views, and then the application logic.  With XPages, we can create the application using a combination of many languages and methods of implementation.  This is one of the greatest features of Domino, but also is one of its biggest drawbacks because it allows developers to chose whatever direction they want and it will work.   Each time you want to create a new application you create more databases.  So the more applications the more Domino databases you needed to maintain and manage leading up to an unyielding amount of maintenance and inconsistency. 

We wanted to minimize the number of databases that we needed and utilize the JSON database structure that we created.  Since we did not use the traditional XPages development method, we were free to develop our own approach to application development in Domino.  Having the front-end approach and the database infrastructure, we needed the controller to tie everything together. 

We could create a tightly coupled solution that allowed the server to generate the web page which has better performance or we could use RESTful services to provide the data and have the client JavaScript to generate the information on the web page.  After many experiments, we concluded that the best solution was to have the server generate a web page framework and use RESTful services using our Dojo data binding to populate the web page.  This approach was more flexible and totally decoupled the data from the form.  However, the danger of this approach is that for very RESTful service call a HTTP request is required.  Therefore, we had to be very careful on how we used the RESTful requests.

We created a standardized framework that allowed us to create RESTful services using whatever tools we were comfortable with including Java, SSJS, or LotusScript.  We started off with LotusScript since that was our existing toolset. Since RESTful services are independent of the Web page generation, we can migrate from LotusScript to whatever toolset when appropriate.  At this time, most of the RESTful services are still written in LotusScript though we are beginning to migrate over to Java.  We decided for now that only services that are call often will be migrated to Java.

Since we are generating our own RESTful JSON services, we standardized on a simple extremely lightweight JSON data format that minimizes the amount data that the server needed to transfer.  The format is far different compared to the Domino Data REST format which is very heavy. With a lightweight format, the data binding process is faster, requires far less coding and allowed us to create as a reusable Dojo class that can be applied to all binding objects.

RESTful versus REST
As I mentioned, we are using RESTful JSON services and NOT REST JSON services. There is a major distinction here.  For complete REST services, you have READ, WRITE, EDIT and DELETE. which involves GET, POST, PUT, and DELETE.  For us this is a security problem.  Yes, a person maybe authenticated but in environment where there are multiple ACL and roles defined, you can easily open a security hole where the user has undesired access.  In our RESTful JSON services we only have GET and POST.  In each process, the controller determines from its security analysis what information you can get and also whether you have rights to read, save, edit, or delete down to the individual field level.  By using the controller to determine user access and data security, users NEVER has direct access to the data.  Only through the RESTful APIs can the user request the data. By centralizing data access you reduce your data exposure.

Business Process and Workflow Management
For organizations social collaboration tools aren't very useful unless there is a compelling reason for using it.  It needs to be an integral part of an organization's business process and workflow to be effective.  If not, the social collaboration technology becomes no more that a paperweight and what I call a feel good tool.  From our experience of  developing business process and workflow solutions, we developed an entire framework over the years for creating and managing business processes and workflow applications which became our iPhora Foundation technology. So the next step was how do we integrate social collaboration to become part of an organization's business processes.

The solution was to create bidirectional event trigger points within the social collaboration tool API and utilize the iPhora Foundation API to handle the process.  An incoming event received by the social collaboration tool would trigger a process within the social collaboration tool and it itself can then trigger addition events within or externally to others processes.  Since the trigger event maybe activated by web services or something as simple as a email event we wanted to create a standardized communication process that provided an abstraction layer.  We accomplished this by creating a connector framework.

We are still expanding on this approach.

Next time, part 3
Transforming Domino into a Social Collaboration Platform, Part 3 - Teaching an Old Dog To Do New Tricks

Wednesday, May 14, 2014

MWLUG 2014 Abstract Submission Extended until Next Friday, May 23, at 5:00 PM

With the sad news of Tim passing away, many of us have not had the time to absorb what has happened.  Given that, the MWLUG committee has decided to extend the abstract submission period for MWLUG 2014 until the end of next week, May 23.  MWLUG was the first time that I met Tim in person.  He was one of the first to volunteer as a speaker when we first started the MWLUG conferences.  He will be sorely missed. 

Monday, May 12, 2014

Transforming Domino into a Social Collaboration Platform Part 2A - Experimentation

In my last blog, I described our desire to a create a community-based social collaboration platform designed to meet the needs of small and medium-sized organization.  This post is the geeky part of this series. Only being na├»ve would one have taken on this journey. So the great experimentation began.

So we wanted to transform Domino into a social collaboration platform.  But what did that mean in the technical sense. Since data is fluid and mostly unstructured, we needed the entire infrastructure to be fluid and that was the challenge.  The challenges create many forks in the road of which many were dead ends.

There was four areas of technical challenges that we had to overcome.
  • Web Interface Generation
  • Database Infrastructure
  • Application Structure
  • Business Process and Workflow Management

Web Page and Interface
Though XPages technically separates the data from the form, it is intimately tied together and requires you to use the same infrastructure used in Notes client development.  This is fine if you want to continue working in that environment.  This locked structure does not allow for a fluid development environment. It required you to create views upon views for the front-end which we did not want to do.  Instead, we want a loosely coupled structure so that the interface was totally independent of the back-end. A true MVC approach.  If we decided at one point to replace Domino, we would not have to redo the entire front-end.

However, we needed a way to generate the Web pages and take advantage of the capabilities of XPages.  We were experimenting with a development environment that we created called UXPages a number of years ago.  It had the fluidity, but it was used primary for static web pages. Through the UXPages compiler, it took XML like code and converted it to HTML all written in LotusScript might I add.  It was modeled after Adobe Flex.

We decided to expand on the UXPages compiler which eventually became the iPhora Application Designer to handle serious web application development. Rather than an application being a number of XPages and databases, our approach uses a single XPage as a Web container from which all web pages are generated.

Since we did not want the interface to be intimately tied to the back-end structure we decided that we would use JSON Restful services to pull/push the data from/to the server and generate the web pages using JavaScript.  The advantages is that we can use any type of toolset to generate the JSON Restful services, we could use SSJS, Java, LotusScript, or and anything else. 

A number of different JavaScript frameworks were consider including jQuery.  Dojo won out because it incorporated a structured methodology that is especially important for creating widgets. The JSON data that comes from the server had to automatically bind and generate the information on the Web page.  So we had to create our own data binding approach with the Dojo widgets similar to Angular.js and Backbone.js which created an entire development fork in itself.

Since we wanted the interface to be responsive, so naively started creating our own responsive CSS framework.  However, after hearing about Bootstrap from a blog by Jake Howlett, we decided that Bootstrap was the way to go.  We did consider Foundations, but there was so much open source stuff for Bootstrap. 

We started creating the web interface using standard Dojo widgets, but discovered Dojo widgets to be slow and bloated and really does not work in the new responsive design paradigm. We experimented with different approaches, but after many tries we decided to build our own Dojo Bootstrap widgets.  This gave us the simplicity of Bootstrap widgets while allowing us to have the automatic data binding.  But one problem, almost 100% of the stuff was written in jQuery!!! So that started another venture of learning how to create Bootstrap widgets using Dojo.

After many, many attempts, we eventually were able to create an entire framework and toolset to create the desired development environment.  Now that we had a fluid front end we needed a fluid back-end.

Database Infrastructure
NoSQL databases are now a big buzz in the IT world.  However, Domino is one of the first NoSQL database system. Many newer databases like CouchDB are modeled after Notes and Domino.  One difference between the newer NoSQL database and Domino NSF is the integrate of form and data in an NSF database which is unnecessary in the Web paradigm.

From our previous work with Integrated Business Framework (IBF), we knew that the Domino concept of a Notes form would not cut it.  The main problem was that for every application you built you needed many forms and many views even if you are using XPages.  So the more applications, the more views. And what slows down a Domino server, view indexes. So if we have 10 applications each with many forms and many views, you do the math. So one of the biggest challenges for us was how to create a database structure that would minimize the number views and forms.  We struggled with many different designs and approaches until we finally developed a database structure that had only one form which served as the database container and a fixed number of views no matter how many applications were running.  The form was a mixture of fields and MIME objects that stored everything as JSON allowing us to handle very large records similar to other NoSQL databases like CouchDB. 

One important note.  The newer NoSQL databases are fast and cool, but they lack one critical feature, security, especially when it comes to record access.  As a domino developer we have one critical tool that no one else has, reader fields.  If you are not taking advantage of readers fields to control access, then you need to learn how to effectively use them. With reader fields in place, records in the our database had the added security that we needed.

So now that we had a fluid web interface and a database infrastructure, we needed a controller to manage the security and data.  That for next time.  The blog is getting too long.

Transforming Domino into a Social Collaboration Platform Part 2B - Experimentation.

Monday, May 5, 2014

Transforming Domino into a Social Collaboration Platform, Part 1 - Where Do We Begin?

It has been almost 20 years, since we have been involved the IBM collaboration space.  We have built everything from training simulators, inventory management system, workflow and business process management tools, web site to CRM systems for customers using a variety of tools.  We even built interactive kiosks for educating kids in a museum.

Many years ago when Phora Group was still named Taishan Works, we starting developing an application framework on top of Domino to provide a uniform application environment called Integrated Business Framework. We wanted a framework that our customers could run their business and share information and content with their customers and partners in a team environment like projects and communities. The collaborative environment needed to be an integral part of their business process.  We developed the solution as a Notes client application with a series of plugin modules.  Though it worked, we found the interface hard to develop and maintain. The structure of how Domino used views and forms required too many tweaks and jerry rigging and became a mess especially when IBM killed off a feature we were using.  At that point, we decided that the our future did not reside with the Notes client applications, but Web applications.  Now XPages just started to appear, but was unstable and bad.

We had a number of expectations for the collaborative solution. We wanted our platform to work not only on a desktop environment but also on mobile devices.  So the interface had to be responsive. It need to work on all major browsers including IE, Chrome, Safari and Firefox. The interface needed to be simple and easy to use.

We looked at creating our apps as plugins for Quickplace/Quickr and make the hold thing a Web solution.  But we were shocked after reviewing the internal structure or lack of .  So we abandoned that concept.  And good that we did since IBM ended Quickr. Meanwhile, IBM introduced their series of Websphere-based solution which eventually evolute to IBM Connections.  IBM Connections is a very nice product.

Though IBM Connections is a great product, all of our clients are in the SMB space.  IBM Connections is geared for big enterprises none of which were our customers nor were we interested in competing in the enterprise space.  We looked around for an alternative collaborative solutions outside of IBM, but none really met our expectations.

Our requirements included:
  • A multi-tenant solution that we can have hosted as a SaaS solution and at the same time it can be installed as an on-premises solution. 
  • It needed to be easy to install and manage whether it was hosted or on premises.
  • It needed to be fully secured and allow full control on who, how and when a user can have access to any data
  • It had to work within an internal directory of users or external directory of users.
  • It need to be a scalable solution 
  • It needed to be simple and not bloated with features that users did not want.
  • It needed to be a truly secure platform that customers could store their sensitive data 
  • Have built-in business process and workflow management
So we looked at alternative Web platforms to develop our social collaborative solution for the SMB space.  Thought we have been developing primary in the IBM Domino space for the past 20 years, we wanted to look at other web application platforms.  Developing Domino Web applications has difficult and not natural.  Also, the IBM Domino licensing model was prohibitive expensive.  We looked at PHP, Rails, and other open source solution.  With the advent of NoSQL, MongoDB and CouchDB was a major consideration as the database back-end since collaboration data is liquid and somewhat unstructured.

We struggled to find a solution that met our requirements.  All of these application environments where able to do the job.  However, one nagging issue was data security.  Coming from the Domino world, the data security looked disheartening.  We had to build everything ourselves and we did not want to take the risk.  So there we were, stuck.  Can't use these open source platform and Domino too expensive and difficult as an Web application environment. So just as we were planning to abandon the whole concept, the new XWork licensing model was being discussed. 

Though we still thought XPages was unstable, we made a leap of faith and decide to stick with the Domino platform as our web application platform of choice.  It had the security and with XPages the web engine that we needed.  As least that's what we thought.

Next time: Transforming Domino into a Social Collaboration Platform, Part 2 - The Experimentation

Friday, May 2, 2014

Registration for MWLUG 2014 is Now Open

Registration for MWLUG 2014 is now open.  To register for MWLUG 2014 go to:

As always almost all the cost is covered by our incredible sponsors. However, the MWLUG donation for MWLUG 2014 is slight higher than in the past due to the increased cost that we have been incurring over the past few years. We did not want to cut back on the MWLUG experience that many have expected. Therefore we have increase the donation from $35 to $50.

So for a $50 donation, you be getting:
  • Over 40 technical and business sessions and workshops
  • Breakfast and lunch for Thursday and Friday
  • Wednesday Evening Reception
  • Thursday Evening Social Event
Each year MWLUG is made possible by all the volunteers within our ICS community including IBM Customers, IBM Partners, and IBM Staff.  So if you are interested in helping out send an email to

As a reminder abstract submission for MWLUG 2014 closes on May 16, 2014 so get your abstract in.

CollabSphere 2022 Presentation: COL103 Advanced Automation and Cloud Service Integration for your Notes/Nomad Applications

Our presentation for CollabSphere 2022. Learn how to quickly add workflow automation into Notes/Nomad applications using No-code tools that ...