Skip to main content

The iPhora Journey - Part 4 - JSON is King - The How

 The iPhora Journey - Part 1 - Reimagining Domino

The iPhora Journey - Part 2 - Domino, the Little Engine that Could

The iPhora Journey - Part 3 - Creating an Integrated UI Framework

The iPhora Journey - Part 4 - JSON is King - The Why

The iPhora Journey - Part 4 - JSON is King - The How


As we mentioned yesterday, in reimagining Domino, we wanted Domino to be a modern web application server, one that utilized a JSON-based NoSQL database and be more secure compared to other JSON-based NoSQL platforms.

A Domino document existing within a Domino database is the foundational data record used in iPhora, just as it is with traditional Domino applications. But instead of just storing data into individual fields, we wanted to store and process the JSON in a Domino document.  However, text fields (AKA summary fields) in Domino documents are limited to only 64 KBytes, and that is a serious limitation. 64 KBytes of JSON data does not even touch what the real world typically transfers back and forward between browser and server. We looked into the possibility of using rich text fields to store JSON data, but such fields are messy to work with and the rich text storage format contaminates the data with carriage returns and other unnecessary artifacts. Instead, we decided to make extensive use of MIME fields. We discovered that reading/writing data into MIME fields is just as fast as accessing summary fields, but they can store vastly more data -- up to 1 GBytes in a single field without corrupting the data. Realistically, the average size of data that we store in a MIME field is roughly one MByte.




Since our UI framework was a JavaScript SPA framework, it only seemed natural to use a RESTful API approach to communicate between browser and server, with JSON as the data exchange format. However, LotusScript web agents have an inherent limitation of being able to send and receive only 64 KBytes at a time, a limitation that is incredibly -- well -- limiting to designers. Unfortunately, this limitation does not seem to be going away any time soon, as it even affected the Notes HTTP Request class and the JSON Navigator class that were so recently added to LotusScript. It has since been improved. In order to resolve these two issues, we had to experiment over and over again. To resolve the output of JSON, we ended up chunking the JSON output into 64 KByte chunks and streaming them out to the browser. In order to receive JSON data over 64 Kbytes, we had to resort to receiving the JSON data as form-post and treating the form submission document as merely a container for JSON. After working this out, we can easily send and receive megabytes of JSON data without a problem.

However, Domino had another nasty surprise in store for us. We were happy campers with the ability to send and receive megabytes of JSON. However, iPhora is used globally, including Asia and Europe. Therefore, support for the UTF-8 character set is extremely important. Imagine the look of undisguised horror on our faces when we discovered that incoming POST data to LotusScript web agents utterly corrupted all UTF-8 characters. This is because LotusScript Web agents receive the incoming POST data using the Lotus Multi-Byte Character Set (LMBCS) format, with no support for UTF-8. We notified HCL regarding this issue, but the complexity of the existing LMBCS implementation made it difficult to change without causing other problems. At the same time, the pressure we were receiving from Asia and Europe to provide support for UTF-8 resulted in some serious depression and near-despair at Phora Group. So it was back to the drawing board, and with the aid of Java we were able to create our own scheme to smoothly and quickly handle UTF-8 characters within LotusScript agents.

As we moved forward in the development of iPhora, JSON became the standard, not just for data exchange between servers, but also for defining security, APIs, data structures, data processing, data exchange between software components and modules, and even between functions and methods. JSON allowed us to decouple components, thus extending the concept of reusability. This is something that JavaScript supports intrinsically since JSON is inherent to JavaScript, but in LotusScript, no.

So how does one read, write and process JSON using LotusScript? That is a good question. Over the years, there have been a number of LotusScript JSON parsers created by individuals, such as the SNAPPs JSON Writer from Rob Novak and his company and later Lars Berntrop-Bos created his Turbo JSON Writer which was much faster. Eventually, support for JSON was added to LotusScript with a new class in Domino 10. However, that JSON support was severely limited and quite buggy. It has since been improved to handle more than 64K bytes of data; however, that class is still cumbersome to use when you have to manipulate massive amounts of multi-level JSON data, such as data containing a mix of objects and arrays sometimes embedded 7 to 10 levels down. Processing JSON data is not just about reading it or simply creating it one line at a time. You often need to insert a large JSON object into an existing object, or remove an object or element from an array, or inject data into an existing array element. From a JavaScript programmer's point of view, those are all very simple tasks. However, they are not so easy within LotusScript, nor even with Java.

As JSON became more and more important to us, we had to find a way to quickly handle and process large amounts of JSON. One alternative was to switch from LotusScript to JavaScript/Node and the Domino AppDev Pack. Unfortunately, that does not support MIME fields, which were now essential to iPhora. Over the years, our team created several different types of JSON parser to help process JSON. The first versions were slow and awkward to use, but over time, they got better and better. After many generations of development, we created our Flex JSON Parser, a true JSON parser that is fast and flexible. It is much more in par to how JSON is processed using JavaScript.

Next time, we will talk about the use of this JSON parser and why having such a parser was a game changer. The next edition is titled, "The iPhora Journey - Part 5 - Dammit, Jim, I'm a LotusScript Developer not a JavaScript Developer (AKA Processing JSON Data with the LotusScript Flex JSON Parser)" We will do a quick comparison with JSON processing using JavaScript and the LotusScript NotesJSONNavigator classes.


The iPhora Journey - Part 4 - JSON is King - The How

The iPhora Journey - Part 4 - JSON is King - The Why

The iPhora Journey - Part 3 - Creating an Integrated UI Framework

The iPhora Journey - Part 2 - Domino, the Little Engine that Could

The iPhora Journey - Part 1 - Reimagining Domino

Labels:

Comments

Post a Comment

Popular posts from this blog

The iPhora Journey - Part 8 - Flow-based Programming

After my last post in this series -- way back in September 2022, several things happened that prevented any further installments. First came CollabSphere 2022 and then CollabSphere 2023, and organizing international conferences can easily consume all of one's spare time. Throughout this same time period, our product development efforts continued at full speed and are just now coming to fruition, which means it is finally time to continue our blog series. So let's get started... As developers, most of us create applications through the conscious act of programming, either procedural, as many of us old-timers grew up with, or object-oriented, which we grudgingly had to admit was better. This is true whether we are using Java, LotusScript, C++ or Rust on Domino. (By the way, does anyone remember Pascal? When I was in school, I remember being told it was the language of the future, but for some reason it didn't seem to survive past the MTV era).  But in the last decade, there a...
Post a Comment
Read more

Creating Twitter Bootstrap Widgets - Part II - Let's Assemble

Creating Twitter Bootstrap Widgets - Part I - Anatomy of a Widget Creating Twitter Bootstrap Widgets - Part II - Let's Assemble Creating Twitter Bootstrap Widgets - Part IIIA - Using Dojo To Bring It Together This is two part of my five part series "Creating Twitter Bootstrap Widgets".   As I mentioned in part one of this series, Twitter Bootstrap widgets are built from a collection standard HTML elements, styled, and programmed to function as a single unit. The goal of this series is to teach you how to create a Bootstrap widget that utilizes the Bootstrap CSS and Dojo. The use of Dojo with Bootstrap is very limited with the exception of Kevin Armstrong who did an incredible job with his Dojo Bootstrap, http://dojobootstrap.com. Our example is a combo box that we are building to replace the standard Bootstrap combo box. In part one, we built a widget that looks like a combo box but did not have a drop down menu associated with it to allow the user to make a select...
Post a Comment
Read more

The iPhora Journey - Part 3 - Creating an Integrated UI Framework

The iPhora Journey - Part 1 - Reimagining Domino The iPhora Journey - Part 2 - Domino, the Little Engine that Could The iPhora Journey - Part 3 - Creating an Integrated UI Framework There are many ways to create the user interface (UI) for a web application. The HTML page could be created on the server and then pushed out. It could be static with the data generated on the page by the server with JavaScript, providing a more dynamic experience, or the server could generate new HTML content to update portions of the web page. XPages or PHP are good examples of this. Another method is to have the web page partially generated by the server and have JavaScript build the rest of the HTML by pulling data from the server via an API. This is the approach used in the Single Page Application (SPA) model. In all cases, it is still dependent on the web server technology being using.  As mentioned previously in this blog, XPages is dependent on complete integration between form and document, whi...
Post a Comment
Read more