Reflection
Some time ago, I decided that I wanted to learn a little about reflection in Go. Being a typed language, it is not common the need to discover details about a given type.
This said, sometimes it might be useful every now and then (and I am sure there are a lot of other aspects I cannot think of where reflection is the perfect fit) so what I did is get myself a fun exercise to learn some reflection. My learning exercise was goSQLMarshal but that would not be a clean sample to talk about, since it mixes a lot of SQL and, as usual when one is learning, I might have thrown a lot of useless code in the middle to try new things.
Additional information about reflection can be found here
The whole code for this example can be found in its own github repo
Practical sample
To make this tutorial easier to understand, I created a whole new sample project that should have the least amount of clutter possible without making it ugly. The samples will include testing, as this is something that most howtos miss and I feel adds some value.
The problem at hand is: An HTML form from Go structs creator It will include:
- Creation of HTML5 form fields for very basic go types (string, int[64], float[64]).
- Creation of HTML5 form with its fields filled with the values from the passed go struct.
The project
Among other tings, Reflection can help us know more about a struct fields and values. As you know, interfaces help us constraint variables to types that implement certain methods but there is nothing that can be done for Fields and this is where reflection can come in handy.
Getting The fields
The first thing we are going to do is determine the fields on the type we want to convert into a form, to make this example smaller and les convoluted we will only support strings, ints and floats.
// getFields will return a map of the field names and their HTML
// form input type.
// Since this function is only for educational purposes it handles
// only the most basic types and will return error when a more complex
// one is passed.
func getFields(goType POSTFillable) (map[string]string, error) {
// Only valid for structs for the purposes of this example.
t := reflect.TypeOf(goType)
// We require a struct for this sample so if this is a ptr, lets get
// the concrete type pointed to.
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
if t.Kind() != reflect.Struct {
return nil, fmt.Errorf("interface must be a struct is %v", t.Kind())
}
fieldCount := t.NumField()
// we know how many fields there are, we assume there are all valid.
ret := make(map[string]string, fieldCount)
for i := 0; i < fieldCount; i++ {
f := t.Field(i)
// Normally I could very well use the type and a function with
// a switch going over the possible types here
// but it would make this example a bit more complicated.
fieldHTMLType, valid := GOHTML[f.Type.Name()]
if !valid {
return nil, fmt.Errorf("cannot find an HTML equivalent for %q of type ", f.Name)
}
// Adding tags to an attribute you can add extra metadata.
tag := f.Tag.Get("html")
if fieldHTMLType == HTMLFieldText {
switch tag {
case TagSecret:
fieldHTMLType = HTMLFieldPassword
case TagEmail:
fieldHTMLType = HTMLFieldEmail
case TagURL:
fieldHTMLType = HTMLFieldURL
case TagTel:
fieldHTMLType = HTMLFieldTel
}
}
ret[f.Name] = fieldHTMLType
}
return ret, nil
}Lets break this down a bit.
// Only valid for structs for the purposes of this example.
t := reflect.TypeOf(goType)
// We require a struct for this sample so if this is a ptr, lets get
// the concrete type pointed to.
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
if t.Kind() != reflect.Struct {
return nil, fmt.Errorf("interface must be a struct is %v", t.Kind())
}goType is a POSTFillable interface (ill explain this later) but this is mostly because
i dont like to use interface{}.
The first thing we want to get, is information about the type so, even though goType is an
intance of its type, what we care now, its about the type itself.
As you can see in the doc TypeOf returns a
reflect.Type that is basically an struct that tells us a lot about the structure of our type.
This sample will only work with structs or pointer to structs, mostly because it makes no much sense to make a form from other types, for instance, an integer.
When you want to learn something about a given type, having a pointer to the type tells you very little, luckily reflect helps us solve this issue.
The reflect.Type struct knows about the kind of the underlying type, kind can be one of these
if it is reflect.Ptr or reflect.Interface your can request the reflect.Type of the concrete
type by using reflect.Type.Elem(), beware, this will break if the Kind() is not one of the aforementioned.
Once you have the concrete element, we check that its kind is a Struct, which is the scope of our example.
fieldCount := t.NumField()
// we know how many fields there are, we assume there are all valid.
ret := make(map[string]string, fieldCount)After making sure we have the Kind of object we want, we check its fields, reflect.Type.NumFields() returns
the quantity of fields in the object.
Since we are doing this for educational purposes, we assume that all of them are of valid types and just create the result set.
Then, for each of the fields, we will try to get the relevant information.
f := t.Field(i)
// Normally I could very well use the type and a function with
// a switch going over the possible types here
// but it would make this example a bit more complicated.
fieldHTMLType, valid := GOHTML[f.Type.Name()]
if !valid {
return nil, fmt.Errorf("cannot find an HTML equivalent for %q of type ", f.Name)
}reflect.Value this can only be done, once again,
for structs.
Having a handle of the fields, allows us to learn some metadata from its type, which we use to determine
if it is one of the ones we support.
Then we proceed to extract a bit more useful data.
// Adding tags to an attribute you can add extra metadata.
tag := f.Tag.Get("html")
if fieldHTMLType == HTMLFieldText {
switch tag {
case TagSecret:
fieldHTMLType = HTMLFieldPassword
case TagEmail:
fieldHTMLType = HTMLFieldEmail
case TagURL:
fieldHTMLType = HTMLFieldURL
case TagTel:
fieldHTMLType = HTMLFieldTel
}
}Before adding the name/type to our result map, we check if a tag was not used to add a bit more metadata that can be used by our generator.
Citing this doc:
By convention, tag strings are a concatenation of optionally space-separated key:"value" pairs. Each key is a non-empty string consisting of non-control characters other than space (U+0020 ' '), quote (U+0022 '"'), and colon (U+003A ':'). Each value is quoted using U+0022 '"' characters and Go string literal syntax.
Tags are quite useful, if we are creating a package that relies in reflection, the users of the package can tag their types as a way to give extra information, an example of this is gojson that relies on tags to determine the name of a map field in a json and if the field is included in the encoding when it is empty. In this sample we will support a few tags to let the user specify which variant of text field is a string going to be converted to.
The supported Tags are:
const (
TagSecret = "secret"
TagEmail = "email"
TagURL = "url"
TagTel = "telephone"
)They modify if a string is going to be converted into any of:
const (
HTMLFieldText = "text" // the default string
HTMLFieldPassword = "password" // TagSecret
HTMLFieldEmail = "email" // TagEmail
HTMLFieldURL = "url" // TagURL
HTMLFieldTel = "tel" // TagTel This seems to be useful in phone browsers
)The tests
Any code is broken unless there is a test that proves that wrong, so lets assert that what I said above, is true.
The aproach I like to take when testing is to go through all the possible failures that a given logic might have, a function test should have at least as many tests as exit points. Testing the succesful path is never very useful, it is the one you most likely put more effort into crafting and therefore the one less likely to fail so ideally we want to make sure our functionality behaves correctly under un-ideal conditions and perhaps we can even discover a few unexpected behavior and correct them while leaving behind a test as proof that the code is intended to handle that situation.
Another useful rule, if you find a breaking point, write it into a test, then fix it that way you avoid regressions.
type wrongFormValue int
func (w *wrongFormValue) ProcessPOST(map[string]string) error {
return nil
}
func TestGetFieldsFailsWithoutStruct(t *testing.T) {
var testValue wrongFormValue
testValue = 1
_, err := getFields(&testValue)
expectedMessage := "interface must be a struct is int"
errorMessage := err.Error()
if errorMessage != expectedMessage {
t.Logf("error should be %q but is %q", expectedMessage, errorMessage)
t.Fail()
}
}One of the things I said most is that this works for structs and struct pointers only so the code should take care to reject any other kind of type even if it satisfies the requested interface.
In this case I create a type from int and take care to satisfy the interface, but I check that the function fails anyway because it knows the type is actually an int.
type rightForm struct {
}
func (r *rightForm) ProcessPOST(map[string]string) error {
return nil
}
func TestGetFieldsDereferencesPtr(t *testing.T) {
testValue := rightForm{}
_, err := getFields(&testValue)
if err != nil {
t.Logf("error should be nil, is: %v", err)
t.Fail()
}
}Another important thing I said is that our function should be able to handle both structs and pointer to structs, so lets pass a pointer to a valid (yet empty) struct and make sure that the function does not fail.
And finally, the path of least resistence, and ideal case.
type formWithFields struct {
FirstName string
LastName string
Email string `html:"email"`
Website string `html:"url"`
Phone string `html:"telephone"`
Password string `html:"secret"`
Age int
}
func (f formWithFields) ProcessPOST(map[string]string) error {
return nil
}
func TestGetFieldsReturnsFields(t *testing.T) {
testValue := formWithFields{}
result, err := getFields(testValue)
if err != nil {
t.Logf("error should be nil, is: %v", err)
t.Fail()
}
resultLenght := len(result)
if resultLenght != 7 {
t.Logf("result should contain 7 fields, contains %d", resultLenght)
t.Fail()
}
expected := map[string]string{
"Email": "email",
"Website": "url",
"Phone": "tel",
"Password": "password",
"Age": "number",
"FirstName": "text",
"LastName": "text",
}
for fieldName, fieldType := range expected {
value, ok := result[fieldName]
if !ok {
t.Logf("key %q should be present in the map but is not", fieldName)
t.Logf("obtained map is %#v", result)
t.Logf("expected map is %#v", expected)
t.Fail()
}
if value != fieldType {
t.Logf("expected field type for field %q is %q but the obtained values is %q", fieldName, fieldType, value)
t.Fail()
}
}
}The test for the ideal case can be a bit more lax here, we write the ideal input (or even better a series of ideal inputs so you can properly exercise as much permutations as you think wise) and test that all the correct paths have ben triggered, in this case:
- Struct is converted to a map of fields.
- Error output is nil.
- There are no more fields than expected.
- The fields expected are there.
- The fields expected have the expected types.
Bear in mind, that maps are not ordered in go (they might seem so in practice sometimes with older version of go, don’t fall into that trap) so you need to test with that in mind.
Also, when your success test is written, break expectancy to make sure all checks actually work before deeming it finished.
Making the HTML
This part is not related to reflection at all so you might want to skip it but I felt it was nicer to provide the full example.
Once we got the map with all the requierd properties, we need to create the HTML which is, after all, the goal of our toy project.
The creation is a simple template use.
const (
inputTemplate = `<label for="{{.Name}}">{{.Name}}</label><input name="{{.Name}}" type="{{.Type}}" />`
formTemplate = `<form name="goform" method="POST">%s</form>` // This is not really a template.
)
var (
htmlLineTemplate = template.Must(template.New("htmlInputLine").Parse(inputTemplate))
)// htmlInput represents the data that can be used in htmlLoneTemplate.
type htmlInput struct {
Name string
Type string
Value string
}
// doHTML returns a string containing the HTML form for the passed
// fields map.
func doHTML(fields map[string]string) string {
htmlLines := make([]string, len(fields))
fieldNames := make([]string, len(fields))
fieldNo := 0
for fieldName := range fields {
fieldNames[fieldNo] = fieldName
fieldNo++
}
sort.Sort(sort.StringSlice(fieldNames))
for lineNo, fieldName := range fieldNames {
fieldType := fields[fieldName]
field := htmlInput{
Name: fieldName,
Type: fieldType,
}
var htmlLine bytes.Buffer
htmlLineTemplate.Execute(&htmlLine, field)
htmlLines[lineNo] = htmlLine.String()
}
return fmt.Sprintf(formTemplate, strings.Join(htmlLines, "\n"))
}I believe there is not much to explain here, you can find a very complete documentation on
text/template here.
This use of template is very basic, the only things note worthy are htmlInput type that
holds the variables the template will use. The other bit that might be interesting is the
sorting, it is done mainly because maps dont have an order in go (even though older versions
of go might make you think otherwise when tested empirically) and if we sort the fields we
can afterwards obtain a consistent HTML (it is confusing to have a form that is different
each time you see it)
And now to make sure this does what we said:
func TestDoHTMLReturnsExpectedHTML(t *testing.T) {
inputMap := map[string]string{
"Email": "email",
"Website": "url",
"Phone": "tel",
"Password": "password",
"Age": "number",
"FirstName": "text",
"LastName": "text",
}
expectedHTML := `<form name="goform" method="POST"><label for="Age">Age</label><input name="Age" type="number" />
<label for="Email">Email</label><input name="Email" type="email" />
<label for="FirstName">FirstName</label><input name="FirstName" type="text" />
<label for="LastName">LastName</label><input name="LastName" type="text" />
<label for="Password">Password</label><input name="Password" type="password" />
<label for="Phone">Phone</label><input name="Phone" type="tel" />
<label for="Website">Website</label><input name="Website" type="url" /></form>`
outputHTML := doHTML(inputMap)
if outputHTML != expectedHTML {
t.Logf("got: \n%q\nexpected: \n%q", outputHTML, expectedHTML)
t.Fail()
}
}There is not much more to test here other than the fact that all the fields are in there and that the order is the desired one, luckily we can test all those with just using the expected HTML which is ordered and complete. This could be improved by using several different structs and perhaps each one several times to make sure order is respected everytime, but that would be pretty much testing go’s sort given the simplicity of the function.
The resulting HTML and Form:
<form name="goform" method="POST">
<label for="Age">Age</label><input name="Age" type="number" />
<label for="Email">Email</label><input name="Email" type="email" />
<label for="FirstName">FirstName</label><input name="FirstName" type="text" />
<label for="LastName">LastName</label><input name="LastName" type="text" />
<label for="Password">Password</label><input name="Password" type="password" />
<label for="Phone">Phone</label><input name="Phone" type="tel" />
<label for="Website">Website</label><input name="Website" type="url" />
</form>Getting the values
You did not think that it was over, did you? one important part of introspection of a type, is being able to obtain not only the type attributes, but their values. In small tools like marshalers the structure is half of the job, the values to fill it are also very important (or in the case of unmarshaling too)
To complete this part, I decided to join what we learned from the other two functions and make one that gives us a form with the fields filled by the struct attribute values, which makes much more sense.
// valueStringer tries to return a string representing the value
// of the passed reflect.Value and a boolean indicating if it
// was possible.
func valueStringer(value reflect.Value) (string, bool) {
var stringValue string
switch value.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
v := value.Int()
stringValue = fmt.Sprintf("%d", v)
case reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
v := value.Uint()
stringValue = fmt.Sprintf("%d", v)
case reflect.Float32, reflect.Float64:
v := value.Float()
stringValue = fmt.Sprintf("%f", v)
case reflect.String:
stringValue = value.String()
default:
return "", false
}
return stringValue, true
}
// doFilledForm returns a string containing an HTML form with values
// extracted from the passed POSTFillable or error if this is not possible.
func doFilledForm(contents POSTFillable) (string, error) {
fields, err := getFields(contents)
if err != nil {
return "", fmt.Errorf("cannot obtain fields: %v", err)
}
t := reflect.ValueOf(contents)
// We require a struct for this sample so if this is a ptr, lets get
// the concrete type pointed to.
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
htmlLines := make([]string, len(fields))
fieldNames := make([]string, len(fields))
fieldNo := 0
for fieldName := range fields {
fieldNames[fieldNo] = fieldName
fieldNo++
}
for lineNo, fieldName := range fieldNames {
fieldType := fields[fieldName]
value, valid := valueStringer(t.FieldByName(fieldName))
if !valid {
return "", fmt.Errorf("cannot determine the value for %q", fieldName)
}
field := htmlInput{
Name: fieldName,
Type: fieldType,
Value: value,
}
var htmlLine bytes.Buffer
htmlLineWithValueTemplate.Execute(&htmlLine, field)
htmlLines[lineNo] = htmlLine.String()
}
return fmt.Sprintf(formTemplate, strings.Join(htmlLines, "\n")), nil
}Now this works much more like a finished job and provides exactly what one would need. There is a bit of code repetition but I made some concessions in that aspect for the sake of a clearer example and to avoid creating useless abstracions that end up taking a lot of parameters and returning a lot more information than needed.
The relevant differences from its simpler predecesors are:
t := reflect.ValueOf(contents)
// We require a struct for this sample so if this is a ptr, lets get
// the concrete type pointed to.
if t.Kind() == reflect.Ptr {
t = t.Elem()
}We use reflect.ValueOf instead of reflect.TypeOf
value, valid := valueStringer(t.FieldByName(fieldName))
if !valid {
return "", fmt.Errorf("cannot determine the value for %q", fieldName)
}From the reflect.Value item obtained earlier, we can ask FieldByName and
that in time will allow us to obtain the final value.
As we can see, there is a rudimentary type conversion (the call to valueStringer
that will handle the known types:
switch value.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
v := value.Int()
stringValue = fmt.Sprintf("%d", v)
case reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
v := value.Uint()
stringValue = fmt.Sprintf("%d", v)
case reflect.Float32, reflect.Float64:
v := value.Float()
stringValue = fmt.Sprintf("%f", v)
case reflect.String:
stringValue = value.String()
default:
return "", false
}The simple statement takes in the ouput of FieldByName and based on its kind it obtains the
actual type element and converts that to a string (since we need strings to fill in the HTML form)
The remaning code only varies on the intantiated htmlInput
field := htmlInput{
Name: fieldName,
Type: fieldType,
Value: value,
}It contains the extra struct field Value that will be used in the template for this function.
And to make sure this function work, we test that which differs from its predecessors.
func TestDoFillHTMLReturnsExpectedHTML(t *testing.T) {
inputStruct := formWithFields{
FirstName: "Horacio",
LastName: "Duran",
Email: "horacio.duran@gmail.com",
Website: "http://perri.to",
Phone: "+555555555",
Password: "a big secret",
Age: 32,
}
expectedHTML := `<form name="goform" method="POST"><label for="FirstName">FirstName</label><input name="FirstName" type="text" value="Horacio" />
<label for="LastName">LastName</label><input name="LastName" type="text" value="Duran" />
<label for="Email">Email</label><input name="Email" type="email" value="horacio.duran@gmail.com" />
<label for="Website">Website</label><input name="Website" type="url" value="http://perri.to" />
<label for="Phone">Phone</label><input name="Phone" type="tel" value="+555555555" />
<label for="Password">Password</label><input name="Password" type="password" value="a big secret" />
<label for="Age">Age</label><input name="Age" type="number" value="32" /></form>`
outputHTML, err := doFilledForm(inputStruct)
if err != nil {
t.Logf("no error expected, obtained %v", err)
t.Fail()
}
fmt.Println(outputHTML)
if outputHTML != expectedHTML {
t.Logf("got: \n%q\nexpected: \n%q", outputHTML, expectedHTML)
t.Fail()
}
}For this particular example, the test will check that the produced HTML form has the right fields in the right order and populated with the right values, but in real life you should also test the error exit points.
This is the resulting form:
<form name="goform" method="POST"><label for="FirstName">FirstName</label>
<input name="FirstName" type="text" value="Horacio" />
<label for="LastName">LastName</label><input name="LastName" type="text" value="Duran" />
<label for="Email">Email</label><input name="Email" type="email" value="horacio.duran@gmail.com" />
<label for="Website">Website</label><input name="Website" type="url" value="http://perri.to" />
<label for="Phone">Phone</label><input name="Phone" type="tel" value="+555555555" />
<label for="Password">Password</label><input name="Password" type="password" value="a big secret" />
<label for="Age">Age</label><input name="Age" type="number" value="32" />
</form>I hope this was somehow useful to understand the very basic of reflection and that the whole wrapping was not more distracting than useful.
The whole code for this example can be found in its own github repo