The iPhora Journey - Part 4 - JSON is King - The How

 The iPhora Journey - Part 1 - Reimagining Domino

The iPhora Journey - Part 2 - Domino, the Little Engine that Could

The iPhora Journey - Part 3 - Creating an Integrated UI Framework

The iPhora Journey - Part 4 - JSON is King - The Why

The iPhora Journey - Part 4 - JSON is King - The How

As we mentioned yesterday, in reimagining Domino, we wanted Domino to be a modern web application server, one that utilized a JSON-based NoSQL database and be more secure compared to other JSON-based NoSQL platforms.

A Domino document existing within a Domino database is the foundational data record used in iPhora, just as it is with traditional Domino applications. But instead of just storing data into individual fields, we wanted to store and process the JSON in a Domino document.  However, text fields (AKA summary fields) in Domino documents are limited to only 64 KBytes, and that is a serious limitation. 64 KBytes of JSON data does not even touch what the real world typically transfers back and forward between browser and server. We looked into the possibility of using rich text fields to store JSON data, but such fields are messy to work with and the rich text storage format contaminates the data with carriage returns and other unnecessary artifacts. Instead, we decided to make extensive use of MIME fields. We discovered that reading/writing data into MIME fields is just as fast as accessing summary fields, but they can store vastly more data -- up to 1 GBytes in a single field without corrupting the data. Realistically, the average size of data that we store in a MIME field is roughly one MByte.

Since our UI framework was a JavaScript SPA framework, it only seemed natural to use a RESTful API approach to communicate between browser and server, with JSON as the data exchange format. However, LotusScript web agents have an inherent limitation of being able to send and receive only 64 KBytes at a time, a limitation that is incredibly -- well -- limiting to designers. Unfortunately, this limitation does not seem to be going away any time soon, as it even affected the Notes HTTP Request class and the JSON Navigator class that were so recently added to LotusScript. It has since been improved. In order to resolve these two issues, we had to experiment over and over again. To resolve the output of JSON, we ended up chunking the JSON output into 64 KByte chunks and streaming them out to the browser. In order to receive JSON data over 64 Kbytes, we had to resort to receiving the JSON data as form-post and treating the form submission document as merely a container for JSON. After working this out, we can easily send and receive megabytes of JSON data without a problem.

However, Domino had another nasty surprise in store for us. We were happy campers with the ability to send and receive megabytes of JSON. However, iPhora is used globally, including Asia and Europe. Therefore, support for the UTF-8 character set is extremely important. Imagine the look of undisguised horror on our faces when we discovered that incoming POST data to LotusScript web agents utterly corrupted all UTF-8 characters. This is because LotusScript Web agents receive the incoming POST data using the Lotus Multi-Byte Character Set (LMBCS) format, with no support for UTF-8. We notified HCL regarding this issue, but the complexity of the existing LMBCS implementation made it difficult to change without causing other problems. At the same time, the pressure we were receiving from Asia and Europe to provide support for UTF-8 resulted in some serious depression and near-despair at Phora Group. So it was back to the drawing board, and with the aid of Java we were able to create our own scheme to smoothly and quickly handle UTF-8 characters within LotusScript agents.

As we moved forward in the development of iPhora, JSON became the standard, not just for data exchange between servers, but also for defining security, APIs, data structures, data processing, data exchange between software components and modules, and even between functions and methods. JSON allowed us to decouple components, thus extending the concept of reusability. This is something that JavaScript supports intrinsically since JSON is inherent to JavaScript, but in LotusScript, no.

So how does one read, write and process JSON using LotusScript? That is a good question. Over the years, there have been a number of LotusScript JSON parsers created by individuals, such as the SNAPPs JSON Writer from Rob Novak and his company and later Lars Berntrop-Bos created his Turbo JSON Writer which was much faster. Eventually, support for JSON was added to LotusScript with a new class in Domino 10. However, that JSON support was severely limited and quite buggy. It has since been improved to handle more than 64K bytes of data; however, that class is still cumbersome to use when you have to manipulate massive amounts of multi-level JSON data, such as data containing a mix of objects and arrays sometimes embedded 7 to 10 levels down. Processing JSON data is not just about reading it or simply creating it one line at a time. You often need to insert a large JSON object into an existing object, or remove an object or element from an array, or inject data into an existing array element. From a JavaScript programmer's point of view, those are all very simple tasks. However, they are not so easy within LotusScript, nor even with Java.

As JSON became more and more important to us, we had to find a way to quickly handle and process large amounts of JSON. One alternative was to switch from LotusScript to JavaScript/Node and the Domino AppDev Pack. Unfortunately, that does not support MIME fields, which were now essential to iPhora. Over the years, our team created several different types of JSON parser to help process JSON. The first versions were slow and awkward to use, but over time, they got better and better. After many generations of development, we created our Flex JSON Parser, a true JSON parser that is fast and flexible. It is much more in par to how JSON is processed using JavaScript.

Next time, we will talk about the use of this JSON parser and why having such a parser was a game changer. The next edition is titled, "The iPhora Journey - Part 5 - Dammit, Jim, I'm a LotusScript Developer not a JavaScript Developer (AKA Processing JSON Data with the LotusScript Flex JSON Parser)" We will do a quick comparison with JSON processing using JavaScript and the LotusScript NotesJSONNavigator classes.

The iPhora Journey - Part 4 - JSON is King - The How

The iPhora Journey - Part 4 - JSON is King - The Why

The iPhora Journey - Part 3 - Creating an Integrated UI Framework

The iPhora Journey - Part 2 - Domino, the Little Engine that Could

The iPhora Journey - Part 1 - Reimagining Domino