Microbus considers errors returned from microservice endpoints as exceptions and emphasizes their detection and surfacing. In addition to being logged as errors, these cross-microservice errors are also counted by a metric, visualized in Grafana dashboards, and noted in distributed traces. Consequently, it is ill-advised to use errors to return state that should not raise an alarm.
The point of view of Microbus is that errors will happen, they will be unpredictable, they must never bring down the system and they should be observable and easily debuggable. With this in mind, the framework is taking an opinionated “throw and log” approach to standardize the capturing and surfacing of errors. Note that “capturing” does not mean “handling”. The latter is left up to the app developer (or user).
The standard http.HandlerFunc signature in Go does not return an error but rather leaves it to the developer to set the status code (often to 500 Internal Server Error) and output an error message to the body of the response. This results in repetitive “log and throw” error handling pattern that app developers may or may not consistently conform to.
func StandardHandler(w http.ResponseWriter, r *http.Request) {
err := doSomething()
if err != nil {
logge.Error("failed to do something", "error", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
err = doSomethingElse()
if err != nil {
logge.Error("failed to do something else", "error", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
Microbus extends the default web handler signature to return an error which is then processed by the framework in a wrapper function that itself conforms to the standard http.HandlerFunc signature. With this pattern, the original code is simpler and more Go-like, and any errors are reported and surfaced in a consistent manner. Specifically, error are printed to the body of the response, logged, and metered so that alerts can be triggered.
func UserCodeHandler(w http.ResponseWriter, r *http.Request) error { // Returning an error
err := doSomething()
if err != nil {
return err
}
err = doSomethingElse()
if err != nil {
return err
}
return nil
}
func FrameworkWrapperOfHandler(w http.ResponseWriter, r *http.Request) {
err := UserCodeHandler(w, r)
if err != nil {
// Standardized error capture
w.WriteHeader(http.StatusInternalServerError)
w.Write(fmt.Sprintf("%+v", err))
logger.Error("Handling request", "error", err)
metrics.IncrementErrorCount(1)
return
}
}
Panics in Go that are not captured terminate the running process and for microservices that are expected to be always on, this is not a desirable outcome. To stop this from happening, the framework captures panics generated by user code and converts them to standard errors, which are then handled as such. The wrapper function introduced earlier comes in handy here.
func UserCodeHandler(w http.ResponseWriter, r *http.Request) error {
panic("omg")
}
func FrameworkWrapperOfHandler(w http.ResponseWriter, r *http.Request) {
err := errors.CatchPanic(func() error { // Convert panics to errors
return UserCodeHandler(w, r)
})
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
w.Write(fmt.Sprintf("%+v", err))
logger.Error("Handling request", "error", err)
metrics.IncrementErrorCount(1)
return
}
}
Errors in Go are a simple string message and do not include a stack trace. In distributed systems that are built by distributed teams, that makes it very difficult to identify the root cause. To address that, the framework replaces the standard errors package with its own implementation of augmented errors that support stack traces. This is not entirely transparent and app developers must use errors.Trace to capture the current stack frame.
import "github.com/microbus-io/errors"
func UserCodeHandler(w http.ResponseWriter, r *http.Request) error {
err := doSomething()
if err != nil {
return errors.Trace(err) // Capture this line into the error's stack trace
}
return nil
}
With this in place, error messages make it clear where the error originated from.
strconv.ParseInt: parsing "nan": invalid syntax
[400]
- calculator.(*Service).Square
/src/github.com/microbus-io/fabric/examples/calculator/service.go:75
- connector.(*Connector).Publish
/src/github.com/microbus-io/fabric/connector/messaging.go:94
- httpingress.(*Service).ServeHTTP
/src/github.com/microbus-io/fabric/coreservices/httpingress/service.go:124
All microservices are ultimately web servers where it is common practice to return an appropriate HTTP status code along with an error. To that end, status codes can be associated with errors using either errors.New or errors.Trace. A 500 Internal Server Error status code is associated with errors by default.
func UserCodeHandler(w http.ResponseWriter, r *http.Request) error {
params, err := parseRequest(r)
if err != nil {
return errors.Trace(err, http.BadRequest) // Wrap existing error and attach a status code
}
if obj, ok := m[params.Key]; !ok {
return errors.New("record not found", http.StatusNotFound) // New error with status code
}
return nil
}
The web handler wrapper is extended to write the errors’ status code to the HTTP response writer.
func FrameworkWrapperOfHandler(w http.ResponseWriter, r *http.Request) {
err := errors.CatchPanic(func() error {
return UserCodeHandler(w, r)
})
if err != nil {
w.WriteHeader(errors.StatusCode(err)) // Return the error's status code
w.Write(fmt.Sprintf("%+v", err))
logger.Error("Handling request", "error", err)
metrics.IncrementErrorCount(1)
return
}
}
In addition to a status code, arbitrary name=value pairs can be associated with an error using either errors.New or errors.Trace. Properties are not part of the error’s message and can be retrieved up the call stack in a structured way.
func UserCodeHandler(w http.ResponseWriter, r *http.Request) error {
id := r.URL.Query().Get("id")
obj, ok err := loadObject(r.Context, id)
if err != nil {
return errors.Trace(err)
}
if !ok {
return errors.Trace(err,
http.StatusNotFound,
"id", id,
)
}
json.NewEncoder(w).Encode(obj)
return nil
}
Microservices run in different processes, often on different hardware or even geographies. When a client microservice calls a remote microservice and the latter returns an error, it would be a lot more developer-friendly if the client experienced the error as if it were local.
func (s *Service) MyEndpoint(w http.ResponseWriter, r *http.Request) error {
response, err := s.Publish(pub.GET("https://another.service/objects")) // Remote call
if err != nil {
return errors.Trace(err)
}
}
The Microbus framework serializes any error generated by the remote microservice and reconstitutes it on the client side. Errors responses are identified by the special header Microbus-Op-Code: Err with the error serialized in the body as JSON, including its stack trace. The status code of the error is reflected in the status code of the HTTP.
HTTP/1.1 500 Internal Server Error
Connection: close
Content-Type: application/json
Microbus-From-Host: beta.error.connector
Microbus-From-Id: k2slru4rof
Microbus-Msg-Id: bHh4yWGo
Microbus-Op-Code: Err
{
"error": "it's really bad",
"statusCode": 500,
"properties": {},
"stack": [
{
"file": "/src/github.com/microbus-io/fabric/connector/messaging_test.go",
"function": "connector.TestConnector_Error.func2",
"line": 343
},
{
"file": "/src/github.com/microbus-io/fabric/connector/messaging.go",
"function": "connector.(*Connector).onRequest",
"line": 225
},
{
"file": "/src/github.com/microbus-io/fabric/connector/messaging.go",
"function": "connector.(*Connector).onRequest",
"line": 226
}
]
}
The web handler wrapper function now looks similar to the following:
func FrameworkWrapperOfHandler(w http.ResponseWriter, r *http.Request) {
err := errors.CatchPanic(func() error {
return UserCodeHandler(w, r)
})
if err != nil {
w.Header().Set("Microbus-Op-Code", "Err") // Mark the response as an error
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(errors.StatusCode(err))
json.NewEncoder(w).Encode(err) // Marshal the error as JSON
logger.Error("Handling request", "error", err)
metrics.IncrementErrorCount(1)
return
}
}
The capturing of errors at the framework level improves system stability, observability and developer experience.