I wrote this blog post about 9 months ago and never got around to finishing it. I felt this would be a good time to post this since MWLUG 2017 is coming up and with the Domino Application Development Roundtables at MWLUG 2017 both customers and partners have the opportunity to voice their desires on how they want IBM to move forward with IBM Domino.
As an long time IBM Business Partner and ISV focused on the IBM Domino platform, we have been quite disappointed with the progress and improvements made to IBM Domino over the past 7 years. As other platforms improved and grew there was not much progress with our favorite platform. However, we were committed in staying with the IBM Domino platform for NOW because it provided a number of unique features that was important to us. So we ventured out to see if we could modernize IBM Domino to match what we were seeing in other platforms and create a modern application development platform but at the same time maintain and exceed the secure environment we have come to know in IBM Domino. Data security is the up-most of importance for us.
Please Note: The blog entry is not about taking a Notes application and making it into a Web application.
--------------------------------------------------------------------------------------------------------------------
Created on: 10/24/2016
Recently, Sean Cull wrote a blog entitled "ICONUK and the State of Domino Web Development" discussing his concerns and his future with Domino as a platform. I agree with many of the things that he said. He also mentioned what new features that he wants to see in Domino for him to continue supporting Domino as platform for his business.
According to the latest information, IBM will be announcing a new application modernization strategy in the near future. I have no idea what that is and when this will happen. Only time to tell on whether IBM addresses Sean's requests.
This blog entry is not about what IBM is doing or not doing. I hope IBM finally starts adding the support and features that they promised and will seriously improve and support Domino. It is not just the platform at stake, but for many ICS community members their livelihood.
Rather than laboring on this point, I wanted to address the mindset of developing a Domino application and how that needs to change regardless of whether you are planning to continue working with Domino or move on to a different platform.
Many will disagree with what I am going to say, but please hear me out. One of the things that Sean mention was moving the NSF to a Cloudant/CouchDB style database.
A Cloudant / CouchDB style database in the XPages Core product ( on premises ) that leverages the existing Java APIs ( @dblookup etc ) and has new ones so that we could develop performance applications with larger data sets, faceted filtering and dynamic charts ( instead of cached ones ).
In my opinion, even if IBM does switch the XPages core to use a NoSQL database like Cloudant or CouchDB, there will still be issues. It is not the NSF that is the issue. The heart of one of my biggest issues with Domino is the tightly integration between form and data in processing and storing data. This integration was a huge advantage when Notes started. It allowed developers and power users to quickly create applications. However, the integration of form and data creates a real problem with a significant number of drawbacks.
This fundamental design approach is wrong for modern applications forcing fit into the format, you need to take a pragmatic view and take a look on how other platforms are delivering Web applications.
For example, sortable view columns. This drastically increases view indices and slows down searches and the operation of Domino. It totally unnecessary for Web applications, but yet in XPages it is there. If you followed how IBM wanted you to use XPages, you will have the same issues and you are limiting yourself. Therefore, you have a hammer and you are force fitting it as a catch all tool.
This leads to the title of this blog. I am a big fan of bucking the trend and thinking out of the box.
More than 5 years ago we stepped back and looked at how things were progressing in XPages, and we decided that it was not for us. It bounded the form with the data and then embedded Dojo in a way that made it difficult to update and utilize the newer tools that Dojo had to offer. Thus in eyes of Domino developers, Dojo was crappy and unmanageable. Rather than a simple replacement of Dojo with a newer version, the update process is extremely difficult. This tightly coupled approach does not allow easy updates or improvements. We had already experimented with separating the data with the form when we were developing Notes applications using our Integrated Business Framework.
Regardless if you are dedicated to a full-stack approach or a RESTful API/Javascript MVC approach you need to remove yourself away from the form/data paradigm. It is important to totally separate the data and form to make the components of your application as flexible as possible. We decided upon a RESTful JSON API framework with a client-based MVC framework. This allowed the back-end code to be client independent. The client could be a Web, mobile, or even headless. This front-end and back-end separation allows each developer to focus on their specialty and skills with front-end issues not affecting the back-end code and vice versus.
Using Domino with a JavaScript MVC framework is nothing new. Many are currently creating applications using JavaScript frameworks like Angular and React with Domino. However, the data structure of a Domino document though extremely powerful and secure was designed for the Notes client not for a web client.
Most newer NoSQL databases utilize JSON as the data structure for storage. By storing your data as JSON, the data can be delivered to the JavaScript MVC framework that it easily understand. We decide that a JSON-based data store was the way to go.
We could have abandoned Domino and go with newer JSON document NoSQL databases like CouchDB or MongoDB, but then we would lose many features and functionalities that are not found in other technologies.
So how can we transform the NSF database from a form/data document container to become a JSON-based document container similar to CouchDB / MongoDB while taking advantage of NSF features including replication, clustering, record and field security, and many other things.
First, we had to review the advantages and disadvantages of Domino/NSF and determine how we would utilize the advantages while mitigating the disadvantages. Yes, there are limitations to the NSF format like the 32K field, but that can easily be resolved. Tim Tripcony addressed this many years ago, NSF field size limits no longer matter. Yes, the indexing is not the fastest. and yes there are a host of other issues, but we were able to get around the issues by taking a stepping back and identifying the strengths and weakness of Domino and determine the best approach in utilizing NSF as a JSON document NoSQL data store.
So we started with the following design criteria for our new database structure:
General Requirements
Domino Specific Requirements
So based on these requirements, we decided to transform Domino into a JSON-based document database which forms the basis of our iPhora platform.
Next time, Part 2.
As an long time IBM Business Partner and ISV focused on the IBM Domino platform, we have been quite disappointed with the progress and improvements made to IBM Domino over the past 7 years. As other platforms improved and grew there was not much progress with our favorite platform. However, we were committed in staying with the IBM Domino platform for NOW because it provided a number of unique features that was important to us. So we ventured out to see if we could modernize IBM Domino to match what we were seeing in other platforms and create a modern application development platform but at the same time maintain and exceed the secure environment we have come to know in IBM Domino. Data security is the up-most of importance for us.
Please Note: The blog entry is not about taking a Notes application and making it into a Web application.
--------------------------------------------------------------------------------------------------------------------
Created on: 10/24/2016
Recently, Sean Cull wrote a blog entitled "ICONUK and the State of Domino Web Development" discussing his concerns and his future with Domino as a platform. I agree with many of the things that he said. He also mentioned what new features that he wants to see in Domino for him to continue supporting Domino as platform for his business.
According to the latest information, IBM will be announcing a new application modernization strategy in the near future. I have no idea what that is and when this will happen. Only time to tell on whether IBM addresses Sean's requests.
This blog entry is not about what IBM is doing or not doing. I hope IBM finally starts adding the support and features that they promised and will seriously improve and support Domino. It is not just the platform at stake, but for many ICS community members their livelihood.
Rather than laboring on this point, I wanted to address the mindset of developing a Domino application and how that needs to change regardless of whether you are planning to continue working with Domino or move on to a different platform.
Many will disagree with what I am going to say, but please hear me out. One of the things that Sean mention was moving the NSF to a Cloudant/CouchDB style database.
A Cloudant / CouchDB style database in the XPages Core product ( on premises ) that leverages the existing Java APIs ( @dblookup etc ) and has new ones so that we could develop performance applications with larger data sets, faceted filtering and dynamic charts ( instead of cached ones ).
In my opinion, even if IBM does switch the XPages core to use a NoSQL database like Cloudant or CouchDB, there will still be issues. It is not the NSF that is the issue. The heart of one of my biggest issues with Domino is the tightly integration between form and data in processing and storing data. This integration was a huge advantage when Notes started. It allowed developers and power users to quickly create applications. However, the integration of form and data creates a real problem with a significant number of drawbacks.
This fundamental design approach is wrong for modern applications forcing fit into the format, you need to take a pragmatic view and take a look on how other platforms are delivering Web applications.
For example, sortable view columns. This drastically increases view indices and slows down searches and the operation of Domino. It totally unnecessary for Web applications, but yet in XPages it is there. If you followed how IBM wanted you to use XPages, you will have the same issues and you are limiting yourself. Therefore, you have a hammer and you are force fitting it as a catch all tool.
This leads to the title of this blog. I am a big fan of bucking the trend and thinking out of the box.
More than 5 years ago we stepped back and looked at how things were progressing in XPages, and we decided that it was not for us. It bounded the form with the data and then embedded Dojo in a way that made it difficult to update and utilize the newer tools that Dojo had to offer. Thus in eyes of Domino developers, Dojo was crappy and unmanageable. Rather than a simple replacement of Dojo with a newer version, the update process is extremely difficult. This tightly coupled approach does not allow easy updates or improvements. We had already experimented with separating the data with the form when we were developing Notes applications using our Integrated Business Framework.
Regardless if you are dedicated to a full-stack approach or a RESTful API/Javascript MVC approach you need to remove yourself away from the form/data paradigm. It is important to totally separate the data and form to make the components of your application as flexible as possible. We decided upon a RESTful JSON API framework with a client-based MVC framework. This allowed the back-end code to be client independent. The client could be a Web, mobile, or even headless. This front-end and back-end separation allows each developer to focus on their specialty and skills with front-end issues not affecting the back-end code and vice versus.
Using Domino with a JavaScript MVC framework is nothing new. Many are currently creating applications using JavaScript frameworks like Angular and React with Domino. However, the data structure of a Domino document though extremely powerful and secure was designed for the Notes client not for a web client.
Most newer NoSQL databases utilize JSON as the data structure for storage. By storing your data as JSON, the data can be delivered to the JavaScript MVC framework that it easily understand. We decide that a JSON-based data store was the way to go.
We could have abandoned Domino and go with newer JSON document NoSQL databases like CouchDB or MongoDB, but then we would lose many features and functionalities that are not found in other technologies.
So how can we transform the NSF database from a form/data document container to become a JSON-based document container similar to CouchDB / MongoDB while taking advantage of NSF features including replication, clustering, record and field security, and many other things.
First, we had to review the advantages and disadvantages of Domino/NSF and determine how we would utilize the advantages while mitigating the disadvantages. Yes, there are limitations to the NSF format like the 32K field, but that can easily be resolved. Tim Tripcony addressed this many years ago, NSF field size limits no longer matter. Yes, the indexing is not the fastest. and yes there are a host of other issues, but we were able to get around the issues by taking a stepping back and identifying the strengths and weakness of Domino and determine the best approach in utilizing NSF as a JSON document NoSQL data store.
So we started with the following design criteria for our new database structure:
General Requirements
- JSON key/value document data storage
- Master-Master Replication
- Clustering
- High performance JSON parse.
- ACL and permission management all the way from document records to sub-field level control
- Total separation between form and data, datastore (NSF Database) only stores data
- Like CouchDB and other modern NoSQL database there should only be one standard data container type that holds the record of any document regardless of its functionality or purpose.
- All data stored for any record needs to be stored as JSON objects and portable to other platforms like CouchDB and MongoDB
- Absolutely no user regardless of their role have any direct access to any data record
- All data request is controlled by a gatekeeper that determines dynamically your access rights and what information you have access to
Domino Specific Requirements
- Automatic management of reader/author fields to control data access to individual records and individual key/value
- From the developers POV, a NSF database must be a blackbox.
- A datastore contains a fixed number of views used to access data, developers can not add additional views without a significant reason to do so
- Absolutely, no sorted views are allowed. (Sorted view kills performance by significantly increase view indexes and serves no purpose)
So based on these requirements, we decided to transform Domino into a JSON-based document database which forms the basis of our iPhora platform.
Next time, Part 2.
Comments