JSON Serialization

Extreme Optimization Numerical Libraries for .NET Professional

JSON (JavaScript Object Notation) is a popular text-based serialization format. In contrast to other text-based formats that store tables, JSON can represent arbitrary objects.

The JSON functionality in the Extreme Optimization Numerical Libraries for .NET is built on the JSON.NET framework. Both stand-alone classes and classes that integrate with JSON.NET serialization are available.

The classes that implement the JSON functionality live in the Extreme.Data.Json namespace. Because of the dependency on JSON.NET, they were given their own assembly, Extreme.Data.Json.dll.

Data Access Library features

The JsonFile object, which is the File-like class for reading and writing JSON, has the same functionality as all other File-like clases. The data source can be provided either as a string or a stream. The string can be the path to a file on the local file system, or it may point to a web resource.

In addition, the Open method can take a NewtonSoft.Json.JsonReader, and the Append method can take a NewtonSoft.Json.JsonWriter as the data source.

All methods take an optional argument of type JsonOptions that gives some additional details on how to read or write the JSON.

In addition to the standard methods, there are also methods that can convert a data frame, matrix, or vector to a JSON string or vice versa. To convert an object to a JSON string, use one of the overloads of the ToJson method. To deserialize an object from a string in JSON format, use the ToVectorT(String, JsonOptions), ToMatrixT(String, JsonOptions), or ToDataFrameR, C(String, JsonOptions) method.

These methods are defined as extension methods, so they can be called on the objects and strings directly. The example below constructs a data frame, serializes it to a string, and then immediately deserializes the string to end up with an identical data frame:

C++

Copy

var data = new Dictionary<string, object>() {
        { "state", new string[] { "Ohio", "Ohio", "Ohio", "Nevada", "Nevada" } },
        { "year", new int[] { 2000, 2001, 2002, 2001, 2002 } },
        { "pop", new double[] { 1.5, 1.7, 3.6, 2.4, 2.9 } }
    };
var df1 = DataFrame.FromColumns(data);
var jsonString = df1.ToJson();
var df2 = jsonString.ToDataFrame();

Dim data = New Dictionary(Of String, Object)() From {
            {"state", {"Ohio", "Ohio", "Ohio", "Nevada", "Nevada"}},
            {"year", {2000, 2001, 2002, 2001, 2002}},
            {"pop", {1.5, 1.7, 3.6, 2.4, 2.9}}}
Dim df1 = DataFrame.FromColumns(data)
Dim jsonString = df1.ToJson()
Dim df2 = jsonString.ToDataFrame()

let columns1 : IVector[] = 
    [|
        Vector.Create(11.0, 14.0, 17.0, 93.0, 55.0) ;
        Vector.Create(22.0, 33.0, 43.0, 51.0, 69.0) 
    |]
let df1 = DataFrame.FromColumns(columns1,
            Index.Create([| "First"; "Second" |]))
let jsonString = df1.ToJson()
let df2 = jsonString.ToDataFrame()

JSON.NET Serialization

The JSON.NET library provides a full framework for serializing and deserializing any kind of .NET object. This is done most commonly through the use of serialization attributes which are applied to classes and their members to specify the exact format of the serialization.

To avoid a direct dependency of our core assembly on the JSON.NET assembly, we use a different mechanism: converters. A converter class is a class that can convert between some types of objects and text in JSON format.

Two converters are available. The ComplexJsonConverter class converts complex numbers of any type. The DataObjectJsonConverter class can convert data frames, matrices, vectors, and indexes. These converters can be used as the argument of a JsonConverter attribute to specify that a field should be serialized and deserialized using this converter.

Below is a complete sample that defines an object, complete with JSON serialization attributes. We then create an instance, serialize it to a string, and then deserialize to get a copy of the original object:

C++

Copy

using System.IO;

using Extreme.Data.Json;
using Extreme.DataAnalysis;
using Extreme.Mathematics;

using Newtonsoft.Json;

[JsonObject]
public class Instrument
{
    public string Symbol { get; set; }
    [JsonConverter(typeof(DataObjectJsonConverter))]
    public Vector<double> History { get; set; }
}

class SerializationExample
{
    public static void Sample()
    {
        var instrument = new Instrument();
        instrument.Symbol = "APPL";
        instrument.History = Vector.Create(1.0, 2.0, 3.0, 4.0, 5.0);
        instrument.History.Index = Index.CreateDateRange(new DateTime(2016, 05, 1), 5, Recurrence.Daily);
        var s = new JsonSerializer();
        var sw = new StringWriter();
        s.Serialize(sw, instrument);
        var json = sw.ToString();
        Console.WriteLine(json);
        // Needed for deserialization:
        s.Converters.Add(new DataObjectJsonConverter());
        var sr = new StringReader(json);
        var instrument2 = s.Deserialize<Instrument>(new JsonTextReader(sr));
        Console.WriteLine(instrument2.Symbol);
        Console.WriteLine(instrument2.History);
    }
}

Imports System.IO

Imports Extreme.Data.Json
Imports Extreme.DataAnalysis
Imports Extreme.Mathematics

Imports Newtonsoft.Json

<JsonObject>
Public Class Instrument
    Public Property Symbol As String
    <JsonConverter(GetType(DataObjectJsonConverter))>
    Public Property History As Vector(Of Double)
End Class

Class SerializationExample

    Public Shared Sub Sample()
        Dim instrument = New Instrument()
        instrument.Symbol = "APPL"
        instrument.History = Vector.Create(1.0, 2.0, 3.0, 4.0, 5.0)
        instrument.History.Index = Index.CreateDateRange(New DateTime(2016, 5, 1), 5, Recurrence.Daily)
        Dim s = New JsonSerializer()
        Dim sw = New StringWriter()
        s.Serialize(sw, instrument)
        Dim json = sw.ToString()
        Console.WriteLine(json)
        ' Needed for deserialization:
        s.Converters.Add(New DataObjectJsonConverter())
        Dim sr = New StringReader(json)
        Dim instrument2 = s.Deserialize(Of Instrument)(New JsonTextReader(sr))
        Console.WriteLine(instrument2.Symbol)
        Console.WriteLine(instrument2.History)
    End Sub
End Class

Class DataAccess
    Public Shared Sub Json()
        ' #Region JSONToFrom
        Dim data = New Dictionary(Of String, Object)() From {
                    {"state", {"Ohio", "Ohio", "Ohio", "Nevada", "Nevada"}},
                    {"year", {2000, 2001, 2002, 2001, 2002}},
                    {"pop", {1.5, 1.7, 3.6, 2.4, 2.9}}}
        Dim df1 = DataFrame.FromColumns(data)
        Dim jsonString = df1.ToJson()
        Dim df2 = jsonString.ToDataFrame()
        ' #endregion
    End Sub


    Public Shared Sub RFile()
        ' #Region RReadDataFrame
        Dim df1 = RdataFile.ReadDataFrame(Of DateTime, String)("c:\data.rda")
        Dim df2 = RdataFile.ReadDataFrame("www.example.com/sample.rda", "frame")
        ' #endregion
        ' #Region RReadVectorMatrix
        Dim vector1 = RdataFile.ReadVector(Of Double)("c:\vector.rda")
        Dim matrix1 = RdataFile.ReadMatrix(Of Double)("www.example.com/matrix.rda", "mat")
        ' #endregion
        ' #Region RReadDataFrames
        Dim dataFrameVars = {"frame1", "frame2"}
        Dim dataFrames = RdataFile.ReadDataFrames("c:\data.rda", dataFrameVars)
        Dim frame1 = dataFrames("frame1")
        Dim frame2 = dataFrames("frame2")
        ' #endregion
        ' #Region RWrite
        RdataFile.Write("c:\data.rda", df1)
        Dim Stream = File.OpenWrite("c:\output.rda")
        RdataFile.Write(Stream, matrix1)
        ' #endregion
    End Sub

    Public Shared Sub RStream()
        ' #Region RStreamReadDataFrame
        Using s1 = RdataFile.Open("www.example.com/sample.rda")
            Dim df1 = s1.ReadDataFrame(Of DateTime, String)()
            Dim df2 = s1.ReadDataFrame("frame")
        End Using
        ' #endregion
        ' #Region RStreamReadVectorMatrix
        Using s2 = RdataFile.Open("c:\vector.rda")
            Dim vector1 = s2.ReadVector(Of Double)()
            Dim matrix1 = s2.ReadMatrix(Of Double)("mat")
        End Using
        ' #endregion
        ' #Region RStreamReadDataFrames
        Using s2 = RdataFile.Open("c:\sample.rda")
            Dim dataFrameVars = {"frame1", "frame2"}
            Dim dataFrames = s2.ReadDataFrames(dataFrameVars)
            Dim frame1 = dataFrames("frame1")
            Dim frame2 = dataFrames("frame2")
        End Using
        ' #endregion
    End Sub
    Public Shared Sub RStream2()
        Dim frame1 As DataFrame(Of Long, String) = Nothing
        Dim frame2 As DataFrame(Of Long, String) = Nothing
        Dim matrix1 As Matrix(Of Double) = Nothing
        Dim vector1 As Vector(Of Double) = Nothing
        ' #region RStreamWrite
        Using stream = RdataFile.Create("c:\sample.rda")
            stream.Write({frame1, frame2}, {"df1", "df2"})
            stream.Write(matrix1, "matrix1")
            stream.Write(vector1, "vector1")
        End Using
        ' #endregion
    End Sub


    Public Shared Sub MatlabFiles()
        ' #region MatlabReadDataFrame
        ' dim df1 = MatlabFile.ReadDataFrame<DateTime, string>("c:\data.mat")
        ' dim df2 = MatlabFile.ReadDataFrame("www.example.com/sample.mat", "frame")
        ' #endregion
        ' #Region MatlabReadVectorMatrix
        Dim vector1 = MatlabFile.ReadVector(Of Double)("c:\vector.mat")
        Dim matrix1 = MatlabFile.ReadMatrix(Of Double)("www.example.com/matrix.mat", "mat")
        ' #endregion
        ' #region MatlabReadDataFrames
        ' string[] dataFrameVars = { "frame1", "frame2" }
        ' dim dataFrames = MatlabFile.ReadDataFrames("c:\data.mat", dataFrameVars)
        ' dim frame1 = dataFrames["frame1"]
        ' dim frame2 = dataFrames["frame2"]
        ' #endregion
        ' #region MatlabWrite
        ' MatlabFile.Write("c:\data.mat", df1)
        ' dim stream = File.OpenWrite("c:\output.mat")
        ' MatlabFile.Write(stream, matrix1)
        ' #endregion
    End Sub

    Public Shared Sub MatlabStreams()
        ' #region MatlabStreamReadDataFrame
        ' using (dim s1 = MatlabFile.Open("www.example.com/sample.mat"))
        ' {
        '     dim df1 = s1.ReadDataFrame<DateTime, string>()
        '     dim df2 = s1.ReadDataFrame("frame")
        ' }
        ' #endregion
        ' #Region MatlabStreamReadVectorMatrix
        Using s2 = MatlabFile.Open("c:\sample.mat")
            Dim vector1 = s2.ReadVector(Of Double)()
            Dim matrix1 = s2.ReadMatrix(Of Double)("a")
        End Using
        ' #endregion
        Using s2 = MatlabFile.Open("c:\sample.mat")
            ' #Region MatlabStreamReadDataFrames
            Dim dataFrameVars = {"frame1", "frame2"}
            Dim dataFrames = s2.ReadDataFrames(dataFrameVars)
            Dim frame1 = dataFrames("frame1")
            Dim frame2 = dataFrames("frame2")
            ' #endregion
        End Using
    End Sub

    Public Shared Sub MatlabStreams2()
        ' DataFrame<long, string> frame1 = null, frame2 = null
        ' Matrix<double> matrix1 = null
        ' Vector<double> vector1 = null
        ' ' #region MatlabStreamWrite
        ' using (dim stream = MatlabFile.Create("c:\data.mat"))
        ' {
        '     stream.Write({ frame1, frame2 }, New string[] { "df1", "df2" })
        '     stream.Write(matrix1, "matrix1")
        '     stream.Write(vector1, "vector1")
        ' }
        ' ' #endregion
    End Sub

    Public Shared Sub StataFiles()
        ' #Region StataReadDataFrame
        Dim df1 = StataFile.ReadDataFrame(Of DateTime, String)("c:\data.dta")
        Dim df2 = StataFile.ReadDataFrame("www.example.com/sample.dta")
        ' #endregion
    End Sub
    Public Shared Sub StataStreams()
        ' #Region StataStreamReadDataFrame
        Using s1 = RdataFile.Open("www.example.com/sample.dta")
            Dim df1 = s1.ReadDataFrame(Of DateTime, String)()
        End Using
        ' #endregion
    End Sub


    '
    ' Fixed-width text
    '

    Public Shared Sub FixedWidthFile()
        '#Region FWReadDataFrame
        Dim columnBreaks = {0, 10, 20, 30}
        Dim options = New FixedWidthTextOptions(columnBreaks)
        Dim df1 = FixedWidthTextFile.ReadDataFrame(Of DateTime, String)(
                "c:\data.txt", options)
        Dim df2 = FixedWidthTextFile.ReadDataFrame(
                "http://www.example.com/sample.txt", columnBreaks)
        '#endregion
        '#Region FWReadVectorMatrix
        Dim vector1 = FixedWidthTextFile.ReadVector(Of Double)(
                "c:\vector.txt", options)
        Dim complexBreaks = {0, 10, 20, 30, 40, 50}
        Dim matrix1 = FixedWidthTextFile.ReadComplexMatrix(Of Double)(
                "http://www.example.com/matrix.txt", complexBreaks)
        '#endregion
    End Sub

    Public Shared Sub FixedWidthStream()
        '#Region FWStreamReadDataFrame
        Dim options = New FixedWidthTextOptions({0, 10, 20, 30})
        Using s1 = FixedWidthTextFile.Open("http://www.example.com/sample.txt", options)
            Dim df1 = s1.ReadDataFrame(Of DateTime, String)()
        End Using
        '#endregion
        '#Region FWStreamReadVectorMatrix
        Dim breaks = {0, 10}
        Using s2 = FixedWidthTextFile.Open("c:\vector.txt", breaks)
            Dim vector1 = s2.ReadComplexVector(Of Double)()
        End Using
        '#endregion
    End Sub

    '
    ' Delimited text
    '


    Public Shared Sub DFile()
        '#Region DReadDataFrame
        Dim df1 = DelimitedTextFile.ReadDataFrame(Of DateTime, String)("c:\data.csv")
        Dim df2 = DelimitedTextFile.ReadDataFrame(
                "http://www.example.com/sample.tsv", DelimitedTextOptions.Tsv)
        '#endregion
        '#Region DReadVectorMatrix
        Dim vector1 = DelimitedTextFile.ReadVector(Of Double)("c:\vector.csv")

        Dim culture = CultureInfo.GetCultureInfo("de-DE")
        Dim options = DelimitedTextOptions.CsvForCulture(culture)
        Dim matrix1 = DelimitedTextFile.ReadMatrix(Of Double)(
                "http://www.example.com/german.csv", options)
        '#endregion
        '#region DWrite
        DelimitedTextFile.Write("c:\data.csv", df1)
        Using stream = File.OpenWrite("c:\output.csv")
            DelimitedTextFile.Write(stream, matrix1)
        End Using
        '#endregion
    End Sub

    Public Shared Sub DStream()
        '#Region DStreamReadDataFrame
        Using s1 = DelimitedTextFile.Open("http://www.example.com/sample.csv")
            Dim df1 = s1.ReadDataFrame(Of DateTime, String)()
        End Using
        '#endregion
        '#Region DStreamReadVectorMatrix
        Using s2 = DelimitedTextFile.Open("c:\vector.csv")
            Dim vector1 = s2.ReadVector(Of Double)()
        End Using
        '#endregion
        Dim matrix1 As Matrix(Of Double) = Nothing
        '#Region DStreamWrite
        Using stream = DelimitedTextFile.Create("c:\data.csv")
            stream.Write(matrix1)
        End Using
        '#endregion
    End Sub


    '
    ' Matrix Market
    '

    Public Shared Sub MMFile()
        ' #Region MMReadVectorMatrix
        Dim vector1 = MatrixMarketFile.ReadVector(Of Double)("c:\vector.mtx")
        Dim matrix1 = MatrixMarketFile.ReadMatrix(Of Double)(
                "http://www.example.com/german.mtx")
        ' #endregion
        ' #Region MMWrite
        MatrixMarketFile.Write("c:\data.mtx", vector1)
        Using stream = File.OpenWrite("c:\output.mtx")
            MatrixMarketFile.Write(stream, matrix1)
        End Using
        ' #endregion
    End Sub

    Public Shared Sub MMStream()
        ' #Region MMStreamReadDataFrame
        Using s1 = MatrixMarketFile.Open("http://www.example.com/sample.mtx")
            Dim df1 = s1.ReadDataFrame(Of DateTime, String)()
        End Using
        ' #endregion
        ' #Region MMStreamReadVectorMatrix
        Using s2 = MatrixMarketFile.Open("c:\vector.mtx")
            Dim vector1 = s2.ReadVector(Of Double)()
        End Using
        ' #endregion
        Dim matrix1 As Matrix(Of Double) = Nothing
        ' #Region MMStreamWrite
        Using stream = MatrixMarketFile.Create("c:\data.mtx")
            stream.Write(matrix1)
        End Using

open System.IO

open Extreme.Data.Json
open Extreme.DataAnalysis
open Extreme.Mathematics

open Newtonsoft.Json

[<JsonObject>]
type Instrument = 
  {
     Symbol : string;
     [<JsonConverter(typeof<DataObjectJsonConverter>)>] History : Vector<float>
  }

module SerializationExample =

    let Sample = 
        let instrument = 
          { 
            Symbol = "APPL";
            History = Vector.Create(1.0, 2.0, 3.0, 4.0, 5.0)
          }
        instrument.History.Index <- 
            Index.CreateDateRange(new DateTime(2016, 05, 1), 5, Recurrence.Daily);
        let s = new JsonSerializer()
        let sw = new StringWriter()
        s.Serialize(sw, instrument)
        let json = sw.ToString()
        printfn "%s" json
        // Needed for deserialization:
        s.Converters.Add(new DataObjectJsonConverter())
        let sr = new StringReader(json)
        let instrument2 = s.Deserialize<Instrument>(new JsonTextReader(sr))
        printfn "%s" (instrument2.Symbol)
        printfn "%O" (instrument2.History)

Send comments on this topic to support@extremeoptimization.com

Copyright © 2004-2023, Extreme Optimization. All rights reserved.
Extreme Optimization, Complexity made simple, M#, and M Sharp are trademarks of ExoAnalytics Inc.
Microsoft, Visual C#, Visual Basic, Visual Studio, Visual Studio.NET, and the Optimized for Visual Studio logo
are registered trademarks of Microsoft Corporation.