perri.to: A mashup of things (by Horacio Duran)

One of the most common stumbling blocks of programming internet facing software in Go is JSON parsing. As a typed language, Go requires a properly defined recipient for deserialized JSON. A common workaround is an infinite nested tree of map[string]interface{} and a lot of recursive type assertions.

While developing atlassian-connect-go I hit the same wall, many of the Atlassian types were only defined as JSON. I ended up fixing it creating LazyApiCoder which can create a full set of go types from either a swagger 3 schema or a set of JSON examples.

My approach:

I developed a big part of the plugin using @mholt6 fantastic tool JSON to Go. I had a mix of types from examples offered in Atlassian documentation at different parts. I decided that I wanted to have all the types possible. I wrote this python script that would fetch me all their examples (at least the visible ones). On top of that I went for LAC’s first iteration, automate the usual workaround.

My first attempt is what now can be invoked as LAC --target schema.go --package apackagename --source "generators/jira/jira/*.json". This crude aproach, unmarshals all the json into a very generi type (map[string]interface{}) and type asserts recursively. If the example is good enough, the resulting struct is reasonable. The naming is a bit questionable and it ends up disambiguating different but equaly named types withsome cheap logic. It will, nevertheless, solve your problem if all you have is a sample.

After I had used the stepping stone to build a big part of the plugin, I hit the second wall. By doing very fine tuning of the plugin, I found some types lacking fields. Looking a bit into it, I found the examples were at fault, there were simply not exhaustive enough or I had not stumbled upon an example of my use case.

Improved Version

Reading my python script, I realized there was a much better place to take this information from. Atlassian (and many other providers) has a swagger 3 specification. A swagger spec contains enough information to craft very decent Go types. The new types could have a very accurate Type, Comments and be sure of being as exhaustive as the API would ever need.

The new version is invoked as LAC --target schema.go --package apackagename --swaggerfile generators/jira/jira/swagger-v3.v3.json and takes only one file at a time, given the nature of swagger descriptions.

Since Swagger is a very docummented spec, all I had to do is read it and, for the cases where it was not so clear, consult known examples. I had coded my first version using an intermediate format, a set of go dictionaries that was used to produce the end code. My only task was to create a converter from Swagger to the intermediate format. The result is a couple of short functions which interpret swagger and build an in memory type representation. The renderer had to learn a few tricks, espeially for embedding types. The biggest challenges were:

• anyOf, allOf and oneOf clauses that were trsanslated to several embedded pointer types in one field/type
• $ref only types which were also embedded types as a result
• recursive referencing, that was made using pointers to self.
• the additionalProperties used in a property of type object with only $ref that ended up needing to be interpreted as map[string]ReferencedType
• the nameless/empty properties that ended up being some flavor of interface{} or map[string]interface{} or []interface which will deserialize ok but present more challenges to the end user.
• making sure descriptions reached comments for all the right types

You can find a fine sample of the results in this JIRA schema

One nice side effect of this is that you can use it as a go generate invocation, if your code contains a swagger spec and you get free types.

Conclusion

There are other solutions that try to work with swagger, some abbandoned, some in development and at least one in use. My problem with all existing libraries for this is that they want to support all of Swagger. I was not interested in integrating endpoints or generating go servers, much less in clients. My goal was simple, create types to work with APIs described in Swagger and I think this is very succesful in it’s goal. I am sure there are bugs and some interpretation errors. I will add more flags as time goes by to modify behavior and to allow for free interpretations of the not very strict parts.

Feel free to send bugs, comments and PRs, they will be well received.