Getting started
The campaign you ran on Black Friday was a huge success, and customers started flocking to your website. Your mixpanel setup that normally has 1000 customer events per hour ends up having millions of customer events within an hour. That way, your data pipeline is now tasked with processing large amounts of JSON data and storing it in your database. You realize that your standard json library json chases can't handle sudden data growth, and your real-time analytics reports fall behind. This is where you see the importance of an efficient JSON parsing library. In addition to handling large payloads, JSON parsing libraries must be able to parse and decode well-written JSON payloads.
In this article, we explore Python passing libraries for large workloads. We specifically look at the capabilities of Ujson, Orjson, and Ijson. We then looked closely at Bradical Json Library (STDLIB / JSON), UJONSO, and Orjson for robustness and efficiency. Since we use strong words and derialization throughout the article, here is a refresher. Serialization involves converting your python objects to a json string, while serialization involves reconstructing a JSON string from your Python data structures.
As we progress through the article, you'll find a decision flow diagram to help decide which parser to use based on your workflow and unique parsing needs. In addition to this, we also test NDJSON and libraries for NDJsons. Let's get started.
Stdlib json
Stylib JSON supports validation of all basic Python data types, including dicts, arrays, and tuples. When the function JONN.OLDS() is called, it loads all the json into memory at once. This is fine for small workloads, but on large computers, JSON.OLDS() can cause critical performance issues such as out-of-memory errors and rushing Downstream operations.
import json
with open("large_payload.json", "r") as f:
json_data = json.loads(f) #loads entire file into memory, all tokens at oncejson
For structured uploads of hundreds of MBS, it's best to use ijson. IJSON, short for 'Iterative JSON', reads files one symbol at a time without memory overhead. In the code below, we compare Json with Ijson.
#The ijson library reads records one token at a time
import ijson
with open("json_data.json", "r") as f:
for record in ijson.items(f, "items.item"): #fetch one dict from the array
process(record) As you can see, ijson fetches one object at a time from json and loads it into python dict object. This is then fed into the driving function, in this case, the process (record) function. The complete functionality of Ijson is given in the figure below.
json
json has been a widely used library for many applications involving large amounts of json, as it was designed to be a fast alternative to STDLIB JSON in Python. The paring speed is good as the underlying code of UJson is written in C, it has Python bindings that connect to the Python interface. Areas that needed improvement in the standard JSON library have been optimized in Ujson for speed and performance. But, Ujson is no longer used in new projects, as the developers themselves explained in PPI that the library is only included in the mode. Below is an illustration of Ujson's high-level processes.
import ujson
taxonomy_data = '{"id":1, "genus":"Thylacinus", "species":"cynocephalus", "extinct": true}'
data_dict = ujson.loads(taxonomy_data) #Deserialize
with open("taxonomy_data.json", "w") as fh: #Serialize
ujson.dump(data_dict, fh)
with open("taxonomy_data.json", "r") as fh: #Deserialize
data = ujson.load(fh)
print(data)We move to the next library named 'Orjson'.
orjson
Since Orjson is written in Rust, it is designed not only for speed but also has memory-safe mechanisms to prevent buffer overflows that developers face when using UJSON. In addition, Orjson supports the creation of additional datatypes beyond the standard python datatypes, including data and datastate objects. Another key difference between Orjson and other libraries is that Orjson's garbage() function returns a byte object, while others return a string. Retrieving data as a Bytes Object is one of the main reasons for Orjson's Fast Footput.
import orjson
book_payload = '{"id":1,"name":"The Great Gatsby","author":"F. Scott Fitzgerald","Publishing House":"Charles Scribner's Sons"}'
data_dict = orjson.loads(book_payload) #Deserialize
print(data_dict)
with open("book_data.json", "wb") as f: #Serialize
f.write(orjson.dumps(data_dict)) #Returns bytes object
with open("book_data.json", "rb") as f:#Deserialize
book_data = orjson.loads(f.read())
print(book_data)Now that we've explored some JSON parsing libraries, let's explore their quiet capabilities.
Testing the serialization capabilities of JSON, UJSON and ORJSON
We create a sample data object with a number, string and datette variable.
from dataclasses import dataclass
from datetime import datetime
@dataclass
class User:
id: int
name: str
created: datetime
u = User(id=1, name="Thomas", created=datetime.now())After that we pass it on to each library to see what happens. We start with STDLIB JSON.
import json
try:
print("json:", json.dumps(u))
except TypeError as e:
print("json error:", e)As expected, we get the following error. (The json library does not support serialization of “daclass” objects and datetime objects.)
Next, we test the same with the UJSON library.
import ujson
try:
print("json:", ujson.dumps(u))
except TypeError as e:
print("json error:", e)
As we can see above, UJSON cannot separate data class object and datatype datatype. Finally, we use the Orjson library for robustness.
import orjson
try:
print("orjson:", orjson.dumps(u))
except TypeError as e:
print("orjson error:", e)We see that Orjson was able to separate dackass and datetime datatypes.
Working with NDJson (special mention)
We've seen JSON paring libraries, but what about NDJson? NDJSON (newly shortened to JSSO), as you may know, is a format where each line is a JSON object. In other words, the compiler is not a comma but a new character. As an example, this is what Ndjson looks like.
{"id": "A13434", "name": "Ella"}
{"id": "A13455", "name": "Charmont"}
{"id": "B32434", "name": "Areida"}NDJSON is widely used for logs and data streams, and as such, NDJson Uploads are excellent choices for manipulation using the IJSON library. For NDJSON BAHOLDS PROPER, it is recommended to use STDLIB JSON. Besides Ijson and Stdlib Json, there is a dedicated NDJSON library. Below are code snippets that demonstrate each method.
Ndjson uses Stdlib Json and Ijson
Since ndjson can be delimited by commas, it is not suitable for heap load, because Stdlib JSON expects to see a list of dicts. In other words, stdlib Json's parser looks for one valid json object, but is instead given multiple json objects in the file being uploaded. Therefore, the file must be split iteratively, line by line, and sent to the calling function for processing.
import json
ndjson_payload = """{"id": "A13434", "name": "Ella"}
{"id": "A13455", "name": "Charmont"}
{"id": "B32434", "name": "Areida"}"""
#Writing NDJSON file
with open("json_lib.ndjson", "w", encoding="utf-8") as fh:
for line in ndjson_payload.splitlines(): #Split string into JSON obj
fh.write(line.strip() + "n") #Write each JSON object as its line
#Reading NDJSON file using json.loads
with open("json_lib.ndjson", "r", encoding="utf-8") as fh:
for line in fh:
if line.strip(): #Remove new lines
item= json.loads(line) #Deserialize
print(item) #or send it to the caller functionWith Ijson, paring is done as shown below. With standard json, we have just one root object, which is a dictionary if it's a single json or an array if it's a list of dicts. But with NDJson, each row is its own root object. The “” argument in Ijson.items() tells IJSON PARSER to look for each root item. Arguments “” and Multiple_Values = True Let the IJSON parser know that there are multiple root json objects in the file, and download one line (each json) at a time.
import ijson
ndjson_payload = """{"id": "A13434", "name": "Ella"}
{"id": "A13455", "name": "Charmont"}
{"id": "B32434", "name": "Areida"}"""
#Writing the payload to a file to be processed by ijson
with open("ijson_lib.ndjson", "w", encoding="utf-8") as fh:
fh.write(ndjson_payload)
with open("ijson_lib.ndjson", "r", encoding="utf-8") as fh:
for item in ijson.items(fh, "", multiple_values=True):
print(item)Finally, we have the experience of dedicating Ndjson. It basically converts NDJson format to standard json.
import ndjson
ndjson_payload = """{"id": "A13434", "name": "Ella"}
{"id": "A13455", "name": "Charmont"}
{"id": "B32434", "name": "Areida"}"""
#writing the payload to a file to be processed by ijson
with open("ndjson_lib.ndjson", "w", encoding="utf-8") as fh:
fh.write(ndjson_payload)
with open("ndjson_lib.ndjson", "r", encoding="utf-8") as fh:
ndjson_data = ndjson.load(fh) #returns a list of dictsAs you have seen, the NDJson file formats are usually classified using Strdlib Json and Ijson. For very large uploads, ijson is the best choice as it works efficiently. But if you want to generate NDJson payloads from other Python objects, the NDJSON library is a good choice. This is because the NDJson.Dumps() function automatically converts Python objects to NDJson format without accessing the data structures.
Now that we've explored NDJson, let's go back to writing the corn symbols, Strylib Json, Ujson and Orjson.
Unchecked json reason
'IJSON' being a distributed parser is very different from most of the parsers we've looked at. If we banked on Ijson and many of these people, we would be comparing apples to oranges. Even if we looked closely at Ijson alongside other parsers, we would get the false impression that Ijson is slow, when it serves a completely different purpose. ijson is optimized for memory efficiency and therefore has a lower type than bulk parsers.
A raw json is created for the paid json for benchmarking purposes
We generate a large synthetic json field that is paid to have a million records, which uses the 'library. This data will be used to document the libraries. The code below can be used to create the Payload for this benchmarking, if you wish to replicate this. The generated file will be between 100 MB and 150 MB in size, which I believe, is large enough to perform tests on the logo.
from mimesis import Person, Address
import json
person_name = Person("en")
complete_address = Address("en")
#streaming to a file
with open("large_payload.json", "w") as fh:
fh.write("[") #JSON array
for i in range(1_000_000):
payload = {
"id": person_name.identifier(),
"name": person_name.full_name(),
"email": person_name.email(),
"address": {
"street": complete_address.street_name(),
"city": complete_address.city(),
"postal_code": complete_address.postal_code()
}
}
json.dump(payload, fh)
if i < 999_999: #To prevent a comma at the last entry
fh.write(",")
fh.write("]") #end JSON arrayBelow is a sample of what the generated data looks like. As you can see, the address fields have been verified that JSON is not just a dimension but also represents real world jsons.
[
{
"id": "8177",
"name": "Willia Hays",
"email": "[email protected]",
"address": {
"street": "Emerald Cove",
"city": "Crown Point",
"postal_code": "58293"
}
},
{
"id": "5931",
"name": "Quinn Greer",
"email": "[email protected]",
"address": {
"street": "Ohlone",
"city": "Bridgeport",
"postal_code": "92982"
}
}
]Let's start by writing a logo.
A closer look at the demands
We use the read() function to store the json file as a string. Then we use the load() function in the library (JSON, Json, and Orjson) to use the JSON string in the jython object. First, we create a Payload_Str payload object from the JOW JSON document.
with open("large_payload1.json", "r") as fh:
payload_str = fh.read() #raw JSON textAfter that we create an approximation function with two arguments. The first argument is the test function. In this case, it's the load() function. The second argument is the Payload_Str constructed from the above file.
def benchmark_load(func, payload_str):
start = time.perf_counter()
for _ in range(3):
func(payload_str)
end = time.perf_counter()
return end - startWe use the above function to test both serialization and mock speed.
Benchmarking Deserialization Speed
We load three libraries tested. We then ran the Benchmark_load() function against the load() function for each of these libraries.
import json, ujson, orjson, time
results = {
"json.loads": benchmark_load(json.loads, payload_str),
"ujson.loads": benchmark_load(ujson.loads, payload_str),
"orjson.loads": benchmark_load(orjson.loads, payload_str),
}
for lib, t in results.items():
print(f"{lib}: {t:.4f} seconds")As we can see, Orjson took little time to weave.
Benchmarking sealialization speed
Next, we test the sealing speed of these libraries.
import json
import ujson
import orjson
import time
results = {
"json.dumps": benchmark("json", json.dumps, payload_str),
"ujson.dumps": benchmark("ujson", ujson.dumps, payload_str),
"orjson.dumps": benchmark("orjson", orjson.dumps, payload_str),
}
for lib, t in results.items():
print(f"{lib}: {t:.4f} seconds")When comparing the runtimes, we can see that Orjson takes the least amount of idle time for Python objects to JSON objects.
Choosing the best JSON library
Clipboard & Workflow Hacks for JSON
Let's say you'd like to view your json in a text editor like notepad++ or share a snippet (from a large payload) in slack with a colleague. You will immediately run into the clipboard or text editor / crash. In such cases, one can use piperclip or tkinter. PyperClip works well for payloads under 50 MB, while TKIRTER works well for medium payloads. For larger loads, you can write JSON to a file to view the data.
Lasting
Jason may seem useless, but with paid play and more nest, this is where this writing can quickly turn into a performance bottleneck. This article aims to highlight how each Python library covers this challenge. When choosing a JSON data paring, speed and accuracy are not always the main criteria. It is the function that determines whether turning, memory efficiency, or long-term attention is required for the payment. In short, json parsing shouldn't be one-way – every way.
